Microsoft Cognitive Services

Link: https://www.microsoft.com/cognitive-services/en-us/apis

Intro: Various AI-related services through online APIs. More importantly, all are free for use.

NAACL 2015 papers

Link: http://naacl.org/naacl-hlt-2015/papers.html

Here is my subjective list of remarkable papers relating to MT research:

 

*** Neural Machine Translation

 

Paul Baltescu and Phil Blunsom. "Pragmatic Neural Language Modelling in Machine Translation"

 

Adrià de Gispert, Gonzalo Iglesias, Bill Byrne. "Fast and Accurate Preordering for SMT using Neural Networks"

 

*** Continuous Models for Statistical Machine Translation

 

Frédéric Blain, Fethi Bougares, Amir Hazem, Loïc Barrault, Holger Schwenk. "Continuous Adaptation to User Feedback for Statistical Machine Translation"

 

Kai Zhao, Hany Hassan, Michael Auli. "Learning Translation Models from Monolingual Continuous Representations"

 

*** Multi-language Translation 

 

Raj Dabre, Fabien Cromieres, Sadao Kurohashi, Pushpak Bhattacharyya. "Leveraging Small Multilingual Corpora for SMT Using Many Pivot Languages"

 

*** Video to Text Translation 

 

Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko. "Translating Videos to Natural Language Using Deep Recurrent Neural Networks" 

 

*** Others

 

Jonathan H. Clark, Chris Dyer, Alon Lavie. "Locally Non-Linear Learning for Statistical Machine Translation via Discretization and Structured Regularization"

 

Graham Neubig, Philip Arthur, Kevin Duh. "Multi-Target Machine Translation with Multi-Synchronous Context-free Grammars"

 

Aurelien Waite and Bill Byrne. "The Geometry of Statistical Machine Translation"

Other papers are also worth reading:

*** News Processing

Areej Alhothali and Jesse Hoey. "Good News or Bad News: Using Affect Control Theory to Analyze Readers' Reaction Towards News Articles"

 

 

Python wrapper for online translators

If you want to use Google Translate and Microsoft Bing Translate for free, you may consider the following Python-based wrappers:

+ Code: Google Translate; Bing Translate

+ Samples:

# Google Translate
import googletrans
gs = googletrans.Googletrans()

languages = gs.get_languages()
print(languages['en'])

print(gs.translate('hello', 'de'))
print(gs.translate('hello', 'zh'))
print(gs.translate('hello', 'vi'))

print(gs.detect('some English words'))

 

#Bing Translate

from mstranslator import Translator

translator =
Translator('cdvhoang', 'HlUUMftdkETWa8E9/jzD4l1CzC8sOhRSJxH+kk0MDBg=')

print(translator.translate('hello', lang_from='en', lang_to='vi')) 

 *** Please note that I don't encourage to use the wrapper for Google Translate because you should respect and pay for using its service (simply it's now commercialized ^_^).

 

ACL 2015 papers

Link: http://acl2015.org/accepted_papers.html

Here is my subjective list of remarkable papers relating to MT research:

*** Conventional Statistical Machine Translation

 A CONTEXT-AWARE TOPIC MODEL FOR STATISTICAL MACHINE TRANSLATION
Jinsong Su, Deyi Xiong, Yang Liu, Xianpei Han, Hongyu Lin and Junfeng Yao

NON-LINEAR LEARNING FOR STATISTICAL MACHINE TRANSLATION
Shujian Huang, Huadong Chen, Xinyu Dai and Jiajun Chen

MULTI-TASK LEARNING FOR MULTIPLE LANGUAGE TRANSLATION
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu and Haifeng Wang

WHAT’S IN A DOMAIN? ANALYZING GENRE AND TOPIC DIFFERENCES IN STATISTICAL MACHINE TRANSLATION
Marlies van der Wees, Arianna Bisazza, Wouter Weerkamp and Christof Monz

*** Neural Machine Translation

ADDRESSING THE RARE WORD PROBLEM IN NEURAL MACHINE TRANSLATION
Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals and Wojciech Zaremba

ENCODING SOURCE LANGUAGE WITH CONVOLUTIONAL NEURAL NETWORK FOR MACHINE TRANSLATION
Fandong Meng, Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang and Qun Liu

IMPROVED NEURAL NETWORK FEATURES, ARCHITECTURE AND LEARNING FOR STATISTICAL MACHINE TRANSLATION
Hendra Setiawan, Zhongqiang Huang, Jacob Devlin, Thomas Lamar and Rabih Zbib

NON-PROJECTIVE DEPENDENCY-BASED PRE-REORDERING WITH RECURRENT NEURAL NETWORK FOR MACHINE TRANSLATION
Antonio Valerio Miceli Barone

ON USING VERY LARGE TARGET VOCABULARY FOR NEURAL MACHINE TRANSLATION
Sebastien Jean, Kyunghyun Cho, Roland Memisevic and Yoshua Bengio

CONTEXT-DEPENDENT TRANSLATION SELECTION USING CONVOLUTIONAL NEURAL NETWORK
Baotian Hu, Zhaopeng Tu, Zhengdong Lu and Hang Li

*** Machine Translation Evaluation and Quality Estimation

ONLINE MULTITASK LEARNING FOR MACHINE TRANSLATION QUALITY ESTIMATION
José G. C. de Souza, Matteo Negri, Marco Turchi and Elisa Ricci

PAIRWISE NEURAL MACHINE TRANSLATION EVALUATION
Francisco Guzmán, Shafiq Joty, Lluís Màrquez and Preslav Nakov

EVALUATING MACHINE TRANSLATION SYSTEMS WITH SECOND LANGUAGE PROFICIENCY TESTS
Takuya Matsuzaki, Akira Fujita, Naoya Todo and Noriko H. Arai

Some notes:

*** According to my observation, there are some research trends depending on data characteristics:
- very big data
- heterogeneous data
- multi-lingual data

*** And of course, deep learning research is still very hot.

Torch vs. Theano vs. Caffe

Link: http://fastml.com/torch-vs-theano/

Here is my summary:

- Torch and Theano are better to be used for research purpose on deep learning (DL) whereas Caffe is more scaled for DL application development.

- Torch and Theano are competitive in terms of speech and performance via different benchmarks. Hence, choosing one of them depends the ease of use from users.

(to be updated)

Andrej Karpathy's blog

Link 1: http://karpathy.github.io/ (Neural Network's basics)
Link 2: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ (Recurrent NN's view)
Intro: A very useful blog from a very good PhD student of Stanford Uni.

Vowpal Wabbit

Link: https://github.com/moses-smt/vowpal_wabbit

Intro: The Vowpal Wabbit (VW) project is a fast out-of-core learning system sponsored by Microsoft Research and (previously) Yahoo! Research. Support is available through the mailing list.

There are two ways to have a fast learning algorithm: (a) start with a slow algorithm and speed it up, or (b) build an intrinsically fast learning algorithm. This project is about approach (b), and it's reached a state where it may be useful to others as a platform for research and experimentation.

How to be a good graduate student

1) From Kevin Murphy: http://www.cs.ubc.ca/~murphyk/Teaching/guideForStudents.html

2) ...

(to be updated)

Multi-Task Learning toolkit

1) MALSAR
Link: http://www.public.asu.edu/~jye02/Software/MALSAR/

Intro: the MALSAR (Multi-tAsk Learning via StructurAl Regularization) package includes the following multi-task learning algorithms:

• Mean-Regularized Multi-Task Learning
• Multi-Task Learning with Joint Feature Selection
• Robust Multi-Task Feature Learning
• Trace-Norm Regularized Multi-Task Learning
• Alternating Structural Optimization
• Incoherent Low-Rank and Sparse Learning
• Robust Low-Rank Multi-Task Learning
• Clustered Multi-Task Learning
• Multi-Task Learning with Graph Structures
• Disease Progression Models
• Incomplete Multi-Source Fusion (iMSF)
• Multi-Stage Multi-Source Fusion
• Multi-Task Clustering

2)
Link: http://klcl.pku.edu.cn/member/sunxu/software/MultiTask.zip
Intro: This is a general purpose software for online multi-task learning. The online multi-task learning is mainly based on Conditional Random Fields (CRF) model and Stochastic Gradient Descent (SGD) training.

I am going to deepen this technique for machine translation and domain adaptation.

MTTK - Machine Translation Toolkit

Link: http://mi.eng.cam.ac.uk/~wjb31/distrib/mttkv1/

Intro: MTTK is a collection of software tools for the alignment of parallel text for use in Statistical Machine Translation. With MTTK you can ...

• Align document translation pairs at the sentence or sub-sentence level, sometimes known as chunking. This is a useful pre-processing step to prepare collections of translations for use in estimating the parameters of complex alignment models. Sub-sentence alignment in particular makes it possible to segment long sentences into shorter aligned segments that otherwise would have to be discarded.
• Train statistical models for parallel text alignment.  The following models are supported : 
• IBM Model-1 and Model-2
• Word-to-Word HMMs  
• Word-to-Phrase HMMs ,  with bigram translation probabilities 
• Parallelize your model training procedures. If you have multiple CPUs available,  you can partition your translation training texts into subsets,  thus speeding up iterative parameter re-estimation procedures and reducing the amount of memory needed in training. This is done under exact EM-based parameter estimation procedures.
• Generate word-to-word and word-to-phrase alignments of parallel text. MTTK can generate Viterbi alignments of parallel text (both training text and other texts) under the supported alignment models.
• Extract word-to-word translation tables from aligned bitext and from the estimated models.
• Extract phrase-to-phrase translation tables (phrase-pair inventories) from aligned parallel text.
• Use the HMM alignment models to induce phrase translations under its statistical models.   Phrase-pair induction can generate richer inventories of phrase translations than can be extracted from Viterbi alignments.
• Edit the C++ source code to implement your own estimation and alignment procedures.

Representation Learning

This research topic is very important in machine learning and a first step for all kinds of machine learning algorithms. Having a robust representation definitely plays a vital role for good performance/accuracy.

Recent progress of representation learning is using deep architecture (normally referring to deep learning). Here I collect some of very good papers worth reading:

1) Representation Learning: A Review and New Perspectives (Yoshua Bengio et al., 2014)

2) Deep Learning of Representations for Unsupervised and Transfer Learning (Yoshua Bengio et al., 2012)

3) ...

Neural Machine Translation

Scientists around the world (especially Google guys) are moving the approaches of Statistical Machine Translation (SMT) (e.g. word-based, statistical with phrase-based or hierarchical, syntax-based) to the next level, namely Neural Machine Translation.

In general, Neural Machine Translation aims to simplify the SMT approaches by taking the source as an input sequence and producing the target as an output sequence via a single, large neural networks.

Here I am trying to catch up the recent progress of Neural Machine Translation.

*** People & Group

1) LISA Lab, University of Montreal 2014 led by Prof. Yoshua Bengio

Latest Demo: http://104.131.78.120/

2) Quoc Viet Le and co. at Google (e.g. Ilya Sutskever, Nal Kalchbrenner)

3) Phil Blunsom's group at Oxford Uni.

4) Dzmitry Bahdanau at Jacobs University Bremen

5) Richard Socher at Stanford Uni.

6) Kyunghyun Cho at NYU?

7) ...

*** Notable Papers

1a) Sequence to Sequence Learning with Neural Networks (Ilya Sutskever et al., NIPS 2014)

Note:

- The core idea behind neural machine translation.

1b) Generating Sequences With Recurrent Neural Networks (Alex Graves et al., ? 2014)
- TBA

2) Neural Machine Translation by Jointly Learning to Align and Translate (Dzmitry Bahdanau et al., EMNLP 2014)

3) On the Properties of Neural Machine Translation: Encoder–Decoder Approaches (Kyunghyun Cho et al., SSST-8 2014)

4) Addressing the Rare Word Problem in Neural Machine Translation (Thang Luong et al., drafted version 2014)

5) On Using Monolingual Corpora in Neural Machine Translation (Caglar Gulcehre et al., arXiv 2015)

6) Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (Richard Socher and co. at MetaMind, arXiv June 2015)
- MT result still not yet released!

7) Effective Approaches to Attention-based Neural Machine Translation (Thang Luong et al., EMNLP'15)

8) (to be updated)

In addition, some other approaches utilized neural processing to enhance the current state-of-the-art SMT framework, for example:

*** For Language Model:

1) Decoding with large-scale neural language models improves translation (Ashish et al., EMNLP 2013)

Note:

- Resulting toolkit: NPLM ver 0.3 (http://nlg.isi.edu/software/nplm/)

Comments

- It is quite hard to choose the optimized parameters (e.g. hidden layer nodes, input and output embedding dimensions) across data-sets and domains.

- In Moses, NPLM feature will slow down the decoder speed.

- It actually improves the translation performance when being used with n-gram LM features. But I am not sure whether it can completely replace n-gram LM features.

2) OxLM: A Neural Language Modelling Framework for Machine Translation (Paul Baltescu et al., The Prague Bulletin of Mathematical Linguistics 2014)

Note:

- Resulting toolkit: OxLM (https://github.com/pauldb89/oxlm)

- Moses already has this feature.

3) rwthlm - A toolkit for training neural network language models (feed-forward, recurrent, and long short-term memory neural networks). The software was written by Martin Sundermeyer.

4) (to be updated)

*** For Translation Model:

1) Fast and Robust Neural Network Joint Models for Statistical Machine Translation (Devlin et al, ACL 2014)

Note: 

- ACL 2014 best paper award. 

- Accoding to the paper, they obtained a very impressive performance for Arabic-English Translation; good performance for Chinese-English Translation (datasets: OpenMT 2012, BOLT; domains: news, web forums).

- Moses already has this feature. Basic implementation of this model is already included in Moses under the name "BilingualLM".

- NPLM can be used to train the models for this.

Comments

- Personally, I tried this model with Moses and evaluated with conversational domains (e.g. SMS, Chat, conversational telephone speech) using OpenMT'15 datasets. I obtained good (but not very impressive, 0.7-1.0 BLEU score) performance compared to basic baseline. Using this model together with other strong features did not give significantly better performance as said in the paper :(.

- Optimizing parameters for this model is an exhausted task.

2) Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (Kyunghyun Cho et al., EMNLP 2014)

3) (to be updated)

*** For Reordering Model:

1) Advancements in Reordering Models for Statistical Machine Translation (Minwei Feng et al., ACL 2013)

2) A Neural Reordering Model for Phrase-based Translation (Peng Li et al., COLING 2014)

3) (to be updated)

Assistant Tools for Scientific Paper Writing

1) PaperRater (online)
Link: http://www.paperrater.com/free_paper_grader

Intro: PaperRater.com is a free resource that utilizes Artificial Intelligence to help students write better. Our technology combines Natural Language Processing, Machine Learning, Information Retrieval, Computational Linguistics, and Data Mining to produce the most powerful automated proofreading tool available on the Internet today. PaperRater.com is used by schools and universities in over 46 countries to help students improve their writing and check for plagiarism.

2) SWAN (offline)
Link: https://cs.joensuu.fi/swan/index.html

Intro: This Swan - Scientific Writing AssistaNt - aims at helping writers with the content, not the grammar or spelling. It guides you towards known good scientific writing practices and helps your readers find your contribution. The tool was designed to help you with your writing, not to merely point out errors. Using the tool should be simple; just enter your text sections into the tool, optionally make some manual elaboration and click the "Evaluate" button. Once you have used the tool with one of your own scientific papers, do let us know how it has helped to you.

3) TBA

Kaggle

Link: http://www.kaggle.com/competitions
Intro: A platform for paid problem solving around the world.

Scientific Writing Assistant

SWAN: http://cs.joensuu.fi/swan/

Intro: This Swan - Scientific Writing AssistaNt - aims at helping writers with the content, not the grammar or spelling. It guides you towards known good scientific writing practices and helps your readers find your contribution. The tool was designed to help you with your writing, not to merely point out errors. Using the tool should be simple; just enter your text sections into the tool, optionally make some manual elaboration and click the "Evaluate" button. Once you have used the tool with one of your own scientific papers, do let us know how it has helped to you.

Google Relation Extraction Corpus

Link: https://code.google.com/p/relation-extraction-corpus/
News: http://googleresearch.blogspot.sg/2013/04/50000-lessons-on-how-to-read-relation.html

SENNA toolkit

Intro: SENNA is a software distributed under a non-commercial license, which outputs a host of Natural Language Processing (NLP) predictions: part-of-speech (POS) tags, chunking (CHK), name entity recognition (NER), semantic role labeling (SRL) and syntactic parsing (PSG).

SENNA is fast because it uses a simple architecture, self-contained because it does not rely on the output of existing NLP system, and accurate because it offers state-of-the-art or near state-of-the-art performance.

Link: http://ronan.collobert.com/senna/

Explicit Semantic Analysis - ESA

Intro: ESA is a vector representation of texts based on Wikippedia as external knowledge base.
Link: http://www.cs.technion.ac.il/~gabr/resources/code/esa/esa.html

Multeval

Link: https://github.com/jhclark/multeval

Intro: MultEval takes machine translation hypotheses from several runs of an optimizer and provides 3 popular metric scores, as well as, standard deviations (via bootstrap resampling) and p-values (via approximate randomization). This allows researchers to mitigate some of the risk of using unstable optimizers such as MERT, MIRA, and MCMC. It is intended to help in evaluating the impact of in-house experimental variations on translation quality; it is currently not setup to do bake-off style comparisons (bake-offs can't require multiple optimizer runs nor a standard tokenization).

Related: http://www.ark.cs.cmu.edu/MT/ (Code for Statistical Significance Testing for MT Evaluation Metrics)

TalkBank

Link: http://talkbank.org/

Intro: The goal of TalkBank is to foster fundamental research in the study of human and animal communication. It will construct sample databases within each of the subfields studying communication. It will use these databases to advance the development of standards and tools for creating, sharing, searching, and commenting upon primary materials via networked computers.