Wikipedia and LM

Link: http://trulymadlywordly.blogspot.sg/2011/03/creating-text-corpus-from-wikipedia.html
Intro: How to leverage Wikipedia repository to create huge LM data.

C++11

http://www.stroustrup.com/what-is-2009.pdf

http://stackoverflow.com/questions/8851670/relevant-boost-features-vs-c11

http://blog.smartbear.com/c-plus-plus/the-biggest-changes-in-c11-and-why-you-should-care/

https://marcoarena.wordpress.com/2012/02/05/modern-cpp-cookbooks/

It seems that I really don't need Boost library ^^.

Rapid conversion table

Link: http://www.rapidtables.com/

git basics guide

Link: http://rogerdudler.github.io/git-guide/

Vowpal Wabbit

Link: https://github.com/moses-smt/vowpal_wabbit

Intro: The Vowpal Wabbit (VW) project is a fast out-of-core learning system sponsored by Microsoft Research and (previously) Yahoo! Research. Support is available through the mailing list.

There are two ways to have a fast learning algorithm: (a) start with a slow algorithm and speed it up, or (b) build an intrinsically fast learning algorithm. This project is about approach (b), and it's reached a state where it may be useful to others as a platform for research and experimentation.

Training Google 1T web corpus with IRSTLM

Thanks to http://www44.atwiki.jp/keisks/pages/50.html , here is the way to train enormous LM with Google 1T web corpus using IRSTLM:

build-sublm.pl --size 3 --ngrams "gunzip -c 3gms/*.gz" --sublm LM.000 --witten-bell
merge-sublm.pl --size 3 --sublm LM -lm g_3grams_LM.gz
compile-lm g_3grams_LM.gz g_3grams_LM.blm
(if you get the error: "lt-compile-lm: lmtable.h:247: virtual double lmtable::setlogOOVpenalty(int): Assertion `dub > dict->size()' failed.")
compile-lm -dub=100000000 g_3grams_LM g_3grams_LM.blm (make the -dub option bigger)

I will validate it soon.

Simple batch programming on Windows

Some simple tricks for batch programming on Windows environment:

1) For loop (refer: http://ss64.com/nt/for_l.html)
Syntax   
      FOR /L %%parameter IN (start,step,end) DO command

Key
   start       : The first number
   step        : The amount by which to increment the sequence
   end         : The last number

   command     : The command to carry out, including any
                 command-line parameters.

   %%parameter : A replaceable parameter:
                 in a batch file use %%G (on the command line %G)

Example
FOR /L %%G IN (0,1,195) DO (
echo %%G & ping 1.1.1.1 -n 1 -w 1000 >nul

)

2) ...