Word Alignment Visualization Tool

Link: http://nlg.isi.edu/demos/picaro/

SENNA toolkit

Intro: SENNA is a software distributed under a non-commercial license, which outputs a host of Natural Language Processing (NLP) predictions: part-of-speech (POS) tags, chunking (CHK), name entity recognition (NER), semantic role labeling (SRL) and syntactic parsing (PSG).

SENNA is fast because it uses a simple architecture, self-contained because it does not rely on the output of existing NLP system, and accurate because it offers state-of-the-art or near state-of-the-art performance.

Link: http://ronan.collobert.com/senna/

Detection of near-duplicate documents

Link: https://github.com/vilda/shash/

Intro: A sample implementation of Charikar's hash for identification of similar documents.

Link: https://code.google.com/p/dkpro-similarity-asl/

Intro: DKPro Similarity is an open source software package for developing text similarity algorithms. The framework is designed to complement DKPro Core, a collection of software components for natural language processing (NLP) based on the Apache UIMA framework. By leveraging the power of the tools available in DKPro Core, it allows for a rich set of similarity computation operations, including the design of full-fledged language processing pipelines and fully customizable processing steps.

The DBpedia Data Set

Link: http://wiki.dbpedia.org/Datasets

Intro: The DBpedia data set uses a large multi-domain ontology which has been derived from Wikipedia. The English version of the DBpedia data set currently describes 3.77 million “things” with 400 million “facts”.

Explicit Semantic Analysis - ESA

Intro: ESA is a vector representation of texts based on Wikippedia as external knowledge base.
Link: http://www.cs.technion.ac.il/~gabr/resources/code/esa/esa.html

Statistical Methods in Language and Linguistic Research

Link: https://www.equinoxpub.com/equinox/books/showbook.asp?bkid=348&keyword=

String Toolkit for Text Processing

Link: http://www.codeproject.com/Articles/23198/C-String-Toolkit-StrTk-Tokenizer

It can be used as lite version of Boost.