Deep Learning Frameworks

There are a lot of deep learning frameworks out there, depending on your usage purpose or the familiarity of your programming languages or working tasks, here I only summarize ones that I am familiar with:

1) TensorFlow by Google (released on 10 Nov 2015): http://tensorflow.org/

Comment: 

2) VELES by Samsung (released on 11 Nov 2015): https://velesnet.ml/

Comment:

3) cnn (lightweight and very fast neural network library in C++, also in Python, works both on Windows and Linux machines): https://github.com/kaishengyao/cnn

Comment: cnn has been proven to be much faster than Theano both with and without GPU. Also, it offers the advantage for software production of neural network models since it has been developing in C++ and more importantly, it supports both Windows and Linux platforms. 

4) to be updated

Andrej Karpathy's blog

Link 1: http://karpathy.github.io/ (Neural Network's basics)
Link 2: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ (Recurrent NN's view)
Intro: A very useful blog from a very good PhD student of Stanford Uni.

Vowpal Wabbit

Link: https://github.com/moses-smt/vowpal_wabbit

Intro: The Vowpal Wabbit (VW) project is a fast out-of-core learning system sponsored by Microsoft Research and (previously) Yahoo! Research. Support is available through the mailing list.

There are two ways to have a fast learning algorithm: (a) start with a slow algorithm and speed it up, or (b) build an intrinsically fast learning algorithm. This project is about approach (b), and it's reached a state where it may be useful to others as a platform for research and experimentation.

Machine Learning podcast

Link: http://www.thetalkingmachines.com/

Intro: Talking Machines is your window into the world of machine learning. 

Multi-Task Learning toolkit

1) MALSAR
Link: http://www.public.asu.edu/~jye02/Software/MALSAR/

Intro: the MALSAR (Multi-tAsk Learning via StructurAl Regularization) package includes the following multi-task learning algorithms:

• Mean-Regularized Multi-Task Learning
• Multi-Task Learning with Joint Feature Selection
• Robust Multi-Task Feature Learning
• Trace-Norm Regularized Multi-Task Learning
• Alternating Structural Optimization
• Incoherent Low-Rank and Sparse Learning
• Robust Low-Rank Multi-Task Learning
• Clustered Multi-Task Learning
• Multi-Task Learning with Graph Structures
• Disease Progression Models
• Incomplete Multi-Source Fusion (iMSF)
• Multi-Stage Multi-Source Fusion
• Multi-Task Clustering

2)
Link: http://klcl.pku.edu.cn/member/sunxu/software/MultiTask.zip
Intro: This is a general purpose software for online multi-task learning. The online multi-task learning is mainly based on Conditional Random Fields (CRF) model and Stochastic Gradient Descent (SGD) training.

I am going to deepen this technique for machine translation and domain adaptation.

NIPS paper repository

Link: http://papers.nips.cc/

Machine Learning materials

*** Lecture notes or courses
1) http://dk-techlogic.blogspot.in/2012/05/best-machine-learning-resources.html?m=1
2) https://gtnlp.wordpress.com/readinglist/
3) http://cs229.stanford.edu/materials.html
4) http://ciml.info/
5) Machine Learning for NLP: http://www.cs.columbia.edu/~mcollins/courses/6998-2012/lectures.html

*** ML Community
1) http://www.metacademy.org/roadmaps/
2) http://fastml.com/
3) ...

*** Toolkits
1) Liblinear và Liblinear with SBM (C++, Java,...)
Link: http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/…

2) StreamSVM (C++)
Link: http://www.ibis.t.u-tokyo.ac.jp/masin/streamsvm.html

3) Vowpal Wabbit (C++, Python wrapper)
Link: http://hunch.net/~vw/

4) SGD
Link: http://leon.bottou.org/projects/sgd

5) Super-big list of ML softwares
Link: http://mloss.org/software/

6) ...

(to be updated)

Deep Learning for NLP

1) CSLM: Continuous Space Language Model toolkit
Link: http://www-lium.univ-lemans.fr/cslm/

Intro: CSLM toolkit is open-source software which implements the so-called continuous space language model.

The basic idea of this approach is to project the word indices onto a continuous space and to use a probability estimator operating on this space. Since the resulting probability functions are smooth functions of the word representation, better generalization to unknown events can be expected. A neural network can be used to simultaneously learn the projection of the words onto the continuous space and to estimate the n-gram probabilities. This is still a n-gram approach, but the LM probabilities are interpolated for any possible context of length n-1 instead of backing-off to shorter contexts. This approach was successfully used in large vocabulary continuous speech recognition and in phrase-based SMT systems.

2) Recurrent Neural Network LM (RNNLM)

Link: http://rnnlm.org/

Intro: Neural network based language models are nowdays among the most successful techniques for statistical language modeling. They can be easily applied in wide range of tasks, including automatic speech recognition and machine translation, and provide significant improvements over classic backoff n-gram models. The 'rnnlm' toolkit can be used to train, evaluate and use such models.

3) word2vec

Link: https://github.com/dav/word2vec or http://deeplearning4j.org/word2vec.html

Intro: This tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words. These representations can be subsequently used in many natural language processing applications and for further research.

4) Long Short Term Memory (LSTM)
Link: http://www.bioinf.jku.at/software/lstm/
Intro: Software for the state of the art recurrent neural network. Long Short-Term Memory Software

5) DL4J Deep Learning for Java
Link: http://deeplearning4j.org/
Intro: Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala. Integrated with Hadoop and Spark, DL4J is designed to be used in business environments, rather than as a research tool. It aims to be cutting-edge plug and play, more convention than configuration, which allows for fast prototyping for non-researchers.

6) CURRENNT
Link: http://sourceforge.net/projects/currennt/
Intro: CUDA-enabled machine learning library for recurrent neural networks which can run both on Windows or Linux machines with CUDA-supported capability. CURRENNT is a machine learning library for Recurrent Neural Networks (RNNs) which uses NVIDIA graphics cards to accelerate the computations. The library implements uni- and bidirectional Long Short-Term Memory (LSTM) architectures and supports deep networks as well as very large data sets that do not fit into main memory.

7) ...

*** Deep learning materials

- For NLP

a) http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial

b) http://cl.naist.jp/~kevinduh/a/deep2014/

c) http://ronan.collobert.com/pub/matos/2009_tutorial_nips.pdf

- For general background

a) http://cl.naist.jp/~kevinduh/notes/duh13deepadvances.pdf

b) http://www.trivedigaurav.com/blog/quoc-les-lectures-on-deep-learning/

c) http://deeplearning.net/reading-list/
d) (new book, 2014, draft  version) http://www.iro.umontreal.ca/~bengioy/dlbook/
e) http://people.idsia.ch/~juergen/deep-learning-overview.html 

- ...

(to be updated)

Q&A community for machine learning

Link: http://metaoptimize.com/qa/

OpenFst Library

Link: http://www.openfst.org/twiki/bin/view/FST/WebHome

Intro: OpenFst is a library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs). Weighted finite-state transducers are automata where each transition has an input label, an output label, and a weight. The more familiar finite-state acceptor is represented as a transducer with each transition's input and output label equal. Finite-state acceptors are used to represent sets of strings (specifically, regular or rational sets); finite-state transducers are used to represent binary relations between pairs of strings (specifically, rational transductions). The weights can be used to represent the cost of taking a particular transition.

Calais

Link: http://www.opencalais.com/

Intro: The OpenCalais Web Service automatically creates rich semantic metadata for the content you submit – in well under a second. Using natural language processing (NLP), machine learning and other methods, Calais analyzes your document and finds the entities within it. But, Calais goes well beyond classic entity identification and returns the facts and events hidden within your text as well.

Preference Learning

Introduction: http://www.ke.tu-darmstadt.de/publications/papers/PLBook-Introduction.pdf

Applications to NLP???

MLcomp

MLcomp is a free website for objectively comparing machine learning programs across various datasets for multiple problem domains.

http://mlcomp.org/

Machine Learning Open Source Software

http://mloss.org/software/

--
Cheers,
Vu

Read the Web Project at CMU

The project namely "Read the Web" which has been undertaking by researchers at CMU (e.g. Prof. Tom Mitchell): http://rtw.ml.cmu.edu/readtheweb.html

Should be tracking this project regularly.

--
Cheers,
Vu

New book "Probabilistic Graphical Models" (2009)

Probabilistic Graphical Models (more)
Principles and Techniques
Daphne Koller and Nir Friedman, 2009

--
Cheers,
Vu

Bayesian Inference with Tears

Will plan to read this article to understand more about Bayesian inference applied to NLP.
Link: http://www.isi.edu/natural-language/people/bayes-with-tears.pdf
(by Kevin Knight)

--
Cheers,
Vu

Notes in machine learning

http://www.ics.uci.edu/~welling/classnotes/classnotes.html

Think such notes are very useful for me to learn more about topics in machine learning.

Useful datasets for Machine Learning: http://archive.ics.uci.edu/ml/

--
Cheers,
Vu

Markov Logic Networks

Markov Logic Networks, a combination of First Order Logic and Markov Networks, is a new graphical model which will be very important for AI modeling in the future. Prof. Pedro Domingos at Univ. of Washington is a pioneer in this field. There are some major references given by him:

1) New book "Markov Logic - An Interface Layer for AI".
Another editorial book: "Integrating Logic and Statistics: Novel Algorithms in Markov Logic Networks" by Marenglen Biba

2) The course about Markov Logic Networks given by Prof. Pedro Domingos at Univ. of Washington.

3) The article "What's missing in AI - The Interface Layer".

4) Alchemy - Open source AI: http://alchemy.cs.washington.edu/

I wonder whether some NLP problems can benefit from such a new model.

--
Cheers,
Vu

Graphical Models in a Nutshell

The paper by Prof. Daphne Koller :
http://robotics.stanford.edu/~koller/Papers/Koller+al:SRL07.pdf

MUST read this paper to understand the underlying principles behind graphical models before proceeding to investigate more!

--
Cheers,
Vu