Microsoft Cognitive Services

Link: https://www.microsoft.com/cognitive-services/en-us/apis

Intro: Various AI-related services through online APIs. More importantly, all are free for use.

Deep Learning Frameworks

There are a lot of deep learning frameworks out there, depending on your usage purpose or the familiarity of your programming languages or working tasks, here I only summarize ones that I am familiar with:

1) TensorFlow by Google (released on 10 Nov 2015): http://tensorflow.org/

Comment: 

2) VELES by Samsung (released on 11 Nov 2015): https://velesnet.ml/

Comment:

3) cnn (lightweight and very fast neural network library in C++, also in Python, works both on Windows and Linux machines): https://github.com/kaishengyao/cnn

Comment: cnn has been proven to be much faster than Theano both with and without GPU. Also, it offers the advantage for software production of neural network models since it has been developing in C++ and more importantly, it supports both Windows and Linux platforms. 

4) to be updated

Wiki Parallel Data Extractor

Link: https://github.com/clab/wikipedia-parallel-titles
Intro: Tools for extracting parallel corpora from article titles across languages in Wikipedia

Tay Nung dictionary

Link: https://sites.google.com/site/tndict/home

Intro: Thanks to some guys on facebook of VNese NLP group, I just know about this. One of interesting problems is to preserve and develop the local regional languages (e.g. Tay Nung in the above link) in parallel with official Vietnamese language.

P.S.: I will be back one day about this issue.

Hansard corpus

Link: http://www.hansard-corpus.org/

Intro: This Hansard corpus (or collection of texts) contains nearly every speech given in the British Parliament from 1803-2005, and it allows you to search these speeches (including semantically-based searches) in ways that are not possible with any other resource.

OLAC - Open Language Archives Community

Link: http://www.language-archives.org/

Intro: OLAC, the Open Language Archives Community, is an international partnership of institutions and individuals who are creating a worldwide virtual library of language resources by: (i) developing consensus on best current practice for the digital archiving of language resources, and (ii) developing a network of interoperating repositories and services for housing and accessing such resources.

IR book by Bruce Croft

Title: Search Engines Information Retrieval in Practice by Prof. Bruce Croft
Link: http://ciir.cs.umass.edu/irbook/

Intro: This book provides an overview of the important issues in information retrieval, and how those issues affect the design and implementation of search engines. Not every topic is covered at the same level of detail. The focus is on some of the most important alternatives to implementing search engine components and the information retrieval models underlying them. The target audience for the book is advanced undergraduates in computer science, although it is also a useful introduction for graduate students. (from the link)

 

NAACL 2015 papers

Link: http://naacl.org/naacl-hlt-2015/papers.html

Here is my subjective list of remarkable papers relating to MT research:

 

*** Neural Machine Translation

 

Paul Baltescu and Phil Blunsom. "Pragmatic Neural Language Modelling in Machine Translation"

 

Adrià de Gispert, Gonzalo Iglesias, Bill Byrne. "Fast and Accurate Preordering for SMT using Neural Networks"

 

*** Continuous Models for Statistical Machine Translation

 

Frédéric Blain, Fethi Bougares, Amir Hazem, Loïc Barrault, Holger Schwenk. "Continuous Adaptation to User Feedback for Statistical Machine Translation"

 

Kai Zhao, Hany Hassan, Michael Auli. "Learning Translation Models from Monolingual Continuous Representations"

 

*** Multi-language Translation 

 

Raj Dabre, Fabien Cromieres, Sadao Kurohashi, Pushpak Bhattacharyya. "Leveraging Small Multilingual Corpora for SMT Using Many Pivot Languages"

 

*** Video to Text Translation 

 

Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko. "Translating Videos to Natural Language Using Deep Recurrent Neural Networks" 

 

*** Others

 

Jonathan H. Clark, Chris Dyer, Alon Lavie. "Locally Non-Linear Learning for Statistical Machine Translation via Discretization and Structured Regularization"

 

Graham Neubig, Philip Arthur, Kevin Duh. "Multi-Target Machine Translation with Multi-Synchronous Context-free Grammars"

 

Aurelien Waite and Bill Byrne. "The Geometry of Statistical Machine Translation"

Other papers are also worth reading:

*** News Processing

Areej Alhothali and Jesse Hoey. "Good News or Bad News: Using Affect Control Theory to Analyze Readers' Reaction Towards News Articles"

 

 

ACL 2015 papers

Link: http://acl2015.org/accepted_papers.html

Here is my subjective list of remarkable papers relating to MT research:

*** Conventional Statistical Machine Translation

 A CONTEXT-AWARE TOPIC MODEL FOR STATISTICAL MACHINE TRANSLATION
Jinsong Su, Deyi Xiong, Yang Liu, Xianpei Han, Hongyu Lin and Junfeng Yao

NON-LINEAR LEARNING FOR STATISTICAL MACHINE TRANSLATION
Shujian Huang, Huadong Chen, Xinyu Dai and Jiajun Chen

MULTI-TASK LEARNING FOR MULTIPLE LANGUAGE TRANSLATION
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu and Haifeng Wang

WHAT’S IN A DOMAIN? ANALYZING GENRE AND TOPIC DIFFERENCES IN STATISTICAL MACHINE TRANSLATION
Marlies van der Wees, Arianna Bisazza, Wouter Weerkamp and Christof Monz

*** Neural Machine Translation

ADDRESSING THE RARE WORD PROBLEM IN NEURAL MACHINE TRANSLATION
Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals and Wojciech Zaremba

ENCODING SOURCE LANGUAGE WITH CONVOLUTIONAL NEURAL NETWORK FOR MACHINE TRANSLATION
Fandong Meng, Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang and Qun Liu

IMPROVED NEURAL NETWORK FEATURES, ARCHITECTURE AND LEARNING FOR STATISTICAL MACHINE TRANSLATION
Hendra Setiawan, Zhongqiang Huang, Jacob Devlin, Thomas Lamar and Rabih Zbib

NON-PROJECTIVE DEPENDENCY-BASED PRE-REORDERING WITH RECURRENT NEURAL NETWORK FOR MACHINE TRANSLATION
Antonio Valerio Miceli Barone

ON USING VERY LARGE TARGET VOCABULARY FOR NEURAL MACHINE TRANSLATION
Sebastien Jean, Kyunghyun Cho, Roland Memisevic and Yoshua Bengio

CONTEXT-DEPENDENT TRANSLATION SELECTION USING CONVOLUTIONAL NEURAL NETWORK
Baotian Hu, Zhaopeng Tu, Zhengdong Lu and Hang Li

*** Machine Translation Evaluation and Quality Estimation

ONLINE MULTITASK LEARNING FOR MACHINE TRANSLATION QUALITY ESTIMATION
José G. C. de Souza, Matteo Negri, Marco Turchi and Elisa Ricci

PAIRWISE NEURAL MACHINE TRANSLATION EVALUATION
Francisco Guzmán, Shafiq Joty, Lluís Màrquez and Preslav Nakov

EVALUATING MACHINE TRANSLATION SYSTEMS WITH SECOND LANGUAGE PROFICIENCY TESTS
Takuya Matsuzaki, Akira Fujita, Naoya Todo and Noriko H. Arai

Some notes:

*** According to my observation, there are some research trends depending on data characteristics:
- very big data
- heterogeneous data
- multi-lingual data

*** And of course, deep learning research is still very hot.

Andrej Karpathy's blog

Link 1: http://karpathy.github.io/ (Neural Network's basics)
Link 2: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ (Recurrent NN's view)
Intro: A very useful blog from a very good PhD student of Stanford Uni.

Brat

Link: http://brat.nlplab.org/introduction.html

Intro: brat is a web-based tool for text annotation; that is, for adding notes to existing text documents. brat is designed in particular for structured annotation, where the notes are not free-form text but have a fixed form that can be automatically processed and interpreted by a computer.

Wikipedia and LM

Link: http://trulymadlywordly.blogspot.sg/2011/03/creating-text-corpus-from-wikipedia.html
Intro: How to leverage Wikipedia repository to create huge LM data.

Puck - GPU-based natural language parser

Link: https://github.com/dlwh/puck

Intro: Puck is a high-speed, high-accuracy parser for natural languages. It's (currently) designed for use with grammars trained with the Berkeley Parser and on NVIDIA cards. On recent-ish NVIDIA cards (e.g. a GTX 680), around 400 sentences a second with a full Berkeley grammar for length <= 40 sentences.

Puck is only useful if you plan on parsing a lot of sentences. On the order of a few thousand. Also, it's designed for throughput, not latency.

Multi-Task Learning toolkit

1) MALSAR
Link: http://www.public.asu.edu/~jye02/Software/MALSAR/

Intro: the MALSAR (Multi-tAsk Learning via StructurAl Regularization) package includes the following multi-task learning algorithms:

• Mean-Regularized Multi-Task Learning
• Multi-Task Learning with Joint Feature Selection
• Robust Multi-Task Feature Learning
• Trace-Norm Regularized Multi-Task Learning
• Alternating Structural Optimization
• Incoherent Low-Rank and Sparse Learning
• Robust Low-Rank Multi-Task Learning
• Clustered Multi-Task Learning
• Multi-Task Learning with Graph Structures
• Disease Progression Models
• Incomplete Multi-Source Fusion (iMSF)
• Multi-Stage Multi-Source Fusion
• Multi-Task Clustering

2)
Link: http://klcl.pku.edu.cn/member/sunxu/software/MultiTask.zip
Intro: This is a general purpose software for online multi-task learning. The online multi-task learning is mainly based on Conditional Random Fields (CRF) model and Stochastic Gradient Descent (SGD) training.

I am going to deepen this technique for machine translation and domain adaptation.

MTTK - Machine Translation Toolkit

Link: http://mi.eng.cam.ac.uk/~wjb31/distrib/mttkv1/

Intro: MTTK is a collection of software tools for the alignment of parallel text for use in Statistical Machine Translation. With MTTK you can ...

• Align document translation pairs at the sentence or sub-sentence level, sometimes known as chunking. This is a useful pre-processing step to prepare collections of translations for use in estimating the parameters of complex alignment models. Sub-sentence alignment in particular makes it possible to segment long sentences into shorter aligned segments that otherwise would have to be discarded.
• Train statistical models for parallel text alignment.  The following models are supported : 
• IBM Model-1 and Model-2
• Word-to-Word HMMs  
• Word-to-Phrase HMMs ,  with bigram translation probabilities 
• Parallelize your model training procedures. If you have multiple CPUs available,  you can partition your translation training texts into subsets,  thus speeding up iterative parameter re-estimation procedures and reducing the amount of memory needed in training. This is done under exact EM-based parameter estimation procedures.
• Generate word-to-word and word-to-phrase alignments of parallel text. MTTK can generate Viterbi alignments of parallel text (both training text and other texts) under the supported alignment models.
• Extract word-to-word translation tables from aligned bitext and from the estimated models.
• Extract phrase-to-phrase translation tables (phrase-pair inventories) from aligned parallel text.
• Use the HMM alignment models to induce phrase translations under its statistical models.   Phrase-pair induction can generate richer inventories of phrase translations than can be extracted from Viterbi alignments.
• Edit the C++ source code to implement your own estimation and alignment procedures.

Python

Python is very powerful programming language, especially for text processing (or NLP) in terms of speed, simplicity, and vastly supported libraries. I've just started using it for two years and already love it.

Here I collect my stuffs relating to Python:

Runtime Libraries

1) 

Link: https://www.python.org/downloads/

2) 

Link: https://cloud.google.com/appengine/docs/python/

IDE 

1) Microsoft Visual Studio

OS supported: Windows, Android, iOS

Debugging: fully supported

Auto completion: fully supported

Additional tools for Visual Studio: PTVS

2) Vim editor

OS supported: Linux

Debugging: none

Auto completion: none

3) More? See this.

Tutorials

1) Everything can be searched in the Internet :D.

2) http://stackoverflow.com/

Community-based question answering for everything (including Python).

Deep Learning for NLP

1) CSLM: Continuous Space Language Model toolkit
Link: http://www-lium.univ-lemans.fr/cslm/

Intro: CSLM toolkit is open-source software which implements the so-called continuous space language model.

The basic idea of this approach is to project the word indices onto a continuous space and to use a probability estimator operating on this space. Since the resulting probability functions are smooth functions of the word representation, better generalization to unknown events can be expected. A neural network can be used to simultaneously learn the projection of the words onto the continuous space and to estimate the n-gram probabilities. This is still a n-gram approach, but the LM probabilities are interpolated for any possible context of length n-1 instead of backing-off to shorter contexts. This approach was successfully used in large vocabulary continuous speech recognition and in phrase-based SMT systems.

2) Recurrent Neural Network LM (RNNLM)

Link: http://rnnlm.org/

Intro: Neural network based language models are nowdays among the most successful techniques for statistical language modeling. They can be easily applied in wide range of tasks, including automatic speech recognition and machine translation, and provide significant improvements over classic backoff n-gram models. The 'rnnlm' toolkit can be used to train, evaluate and use such models.

3) word2vec

Link: https://github.com/dav/word2vec or http://deeplearning4j.org/word2vec.html

Intro: This tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words. These representations can be subsequently used in many natural language processing applications and for further research.

4) Long Short Term Memory (LSTM)
Link: http://www.bioinf.jku.at/software/lstm/
Intro: Software for the state of the art recurrent neural network. Long Short-Term Memory Software

5) DL4J Deep Learning for Java
Link: http://deeplearning4j.org/
Intro: Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala. Integrated with Hadoop and Spark, DL4J is designed to be used in business environments, rather than as a research tool. It aims to be cutting-edge plug and play, more convention than configuration, which allows for fast prototyping for non-researchers.

6) CURRENNT
Link: http://sourceforge.net/projects/currennt/
Intro: CUDA-enabled machine learning library for recurrent neural networks which can run both on Windows or Linux machines with CUDA-supported capability. CURRENNT is a machine learning library for Recurrent Neural Networks (RNNs) which uses NVIDIA graphics cards to accelerate the computations. The library implements uni- and bidirectional Long Short-Term Memory (LSTM) architectures and supports deep networks as well as very large data sets that do not fit into main memory.

7) ...

*** Deep learning materials

- For NLP

a) http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial

b) http://cl.naist.jp/~kevinduh/a/deep2014/

c) http://ronan.collobert.com/pub/matos/2009_tutorial_nips.pdf

- For general background

a) http://cl.naist.jp/~kevinduh/notes/duh13deepadvances.pdf

b) http://www.trivedigaurav.com/blog/quoc-les-lectures-on-deep-learning/

c) http://deeplearning.net/reading-list/
d) (new book, 2014, draft  version) http://www.iro.umontreal.ca/~bengioy/dlbook/
e) http://people.idsia.ch/~juergen/deep-learning-overview.html 

- ...

(to be updated)

WaCky - The Web-As-Corpus Kool Yinitiative

Link: http://wacky.sslmit.unibo.it/doku.php?id=tools

Wikipedia Extractor

Link: http://medialab.di.unipi.it/wiki/Wikipedia_Extractor

Intro: WikiExtractor.py is a Python script that extracts and cleans text from a Wikipedia database dump. The output is stored in a number of files of similar size in a given directory. Each file contains several documents in the document format.

HandAlign

Link: http://www.umiacs.umd.edu/~hal/HandAlign/index.html
Intro: a tool to assist manual alignment in MT, summarization, ...