Detection of near-duplicate documents

Link: https://github.com/vilda/shash/

Intro: A sample implementation of Charikar's hash for identification of similar documents.

Link: https://code.google.com/p/dkpro-similarity-asl/

Intro: DKPro Similarity is an open source software package for developing text similarity algorithms. The framework is designed to complement DKPro Core, a collection of software components for natural language processing (NLP) based on the Apache UIMA framework. By leveraging the power of the tools available in DKPro Core, it allows for a rich set of similarity computation operations, including the design of full-fledged language processing pipelines and fully customizable processing steps.

The DBpedia Data Set

Link: http://wiki.dbpedia.org/Datasets

Intro: The DBpedia data set uses a large multi-domain ontology which has been derived from Wikipedia. The English version of the DBpedia data set currently describes 3.77 million “things” with 400 million “facts”.

Explicit Semantic Analysis - ESA

Intro: ESA is a vector representation of texts based on Wikippedia as external knowledge base.
Link: http://www.cs.technion.ac.il/~gabr/resources/code/esa/esa.html

Statistical Methods in Language and Linguistic Research

Link: https://www.equinoxpub.com/equinox/books/showbook.asp?bkid=348&keyword=

String Toolkit for Text Processing

Link: http://www.codeproject.com/Articles/23198/C-String-Toolkit-StrTk-Tokenizer

It can be used as lite version of Boost.

Job markets for NLPers

Link: http://www.nlppeople.com/

Multeval

Link: https://github.com/jhclark/multeval

Intro: MultEval takes machine translation hypotheses from several runs of an optimizer and provides 3 popular metric scores, as well as, standard deviations (via bootstrap resampling) and p-values (via approximate randomization). This allows researchers to mitigate some of the risk of using unstable optimizers such as MERT, MIRA, and MCMC. It is intended to help in evaluating the impact of in-house experimental variations on translation quality; it is currently not setup to do bake-off style comparisons (bake-offs can't require multiple optimizer runs nor a standard tokenization).

Related: http://www.ark.cs.cmu.edu/MT/ (Code for Statistical Significance Testing for MT Evaluation Metrics)