Showing posts with label monolingual. Show all posts
Showing posts with label monolingual. Show all posts

Thursday, 5 February 2015

Available Data from the CommonCrawl

Link: http://statmt.org/ngrams/
Intro: Multi-language data used for training large-scale LMs crawled from CommonCrawl.

Monday, 1 December 2014

The ClueWeb09 Dataset

Intro: The ClueWeb09 dataset was created to support research on information retrieval and related human language technologies. It consists of about 1 billion web pages in ten languages that were collected in January and February 2009. The dataset is used by several tracks of the TREC conference.
Note: Huge corpus for LM