List of text corpora
Jump to navigation
Jump to search
Text corpora (singular: text corpus) are large and structured sets of texts, which have been systematically collected. Text corpora are used by both AI developers to train large language models and corpus linguists and within other branches of linguistics for statistical analysis, hypothesis testing, finding patterns of language use, investigating language change and variation, and teaching language proficiency.[1]
English language
[edit | edit source]- American National Corpus
- Bank of English
- BookCorpus
- British National Corpus
- Bergen Corpus of London Teenage Language (COLT)
- Brown Corpus, forming part of the "Brown Family" of corpora, together with LOB, Frown and F-LOB
- COCA: see below at English-Corpora.org
- COHA: see below at English-Corpora.org
- Corpus Resource Database (CoRD), more than 80 English language corpora.[2]
- Coruña Corpus, a corpus of late Modern English scientific writing covering the period 1700–1900, developed by the Muste research group at the University of A Coruña
- DBLP Discovery Dataset (D3), a corpus of computer science publications with sentient metadata.[3]
- English-Corpora.org, which contains (among others):
- iWeb, the Intelligent Web-based Corpus: 14 billion words, 6 countries, 2017
- COCA, the Corpus of Contemporary American English: 1.0 billion words, American, 1990-2019
- COHA, the Corpus of Historical American English: 475 million words, American, 1820-2019
- NOW, News on the Web: 23.2 billion+ words, 20 countries, 2010-present
- English Trends, a large English monitor corpus of news articles gathered from RSS feeds, 86+ billion words, 2014–present
- GUM corpus, the open source Georgetown University Multilayer corpus, with very many annotation layers
- Google Books Ngram Corpus[4][5]
- International Corpus of English
- Oxford English Corpus
- RE3D (Relationship and Entity Extraction Evaluation Dataset)
- Santa Barbara Corpus of Spoken American English
- Scottish Corpus of Texts & Speech
- Strathy Corpus of Canadian English
European languages
[edit | edit source]- CETENFolha
- Basque: [6]
- The Corpus of Electronic Texts
- Corpus Inscriptionum Insularum Celticarum (CIIC), covering Primitive Irish inscriptions in Ogham
- Google Books Ngram Corpus
- The Georgian Language Corpus
- Thesaurus Linguae Graecae (Ancient Greek)
- Eastern Armenian National Corpus (EANC) 110 million words. Freely searchable online.
- Spanish text corpus by Molino de Ideas, which contains 660 million words.[7]
- CorALit: the Corpus of Academic Lithuanian Academic texts published in 1999–2009 (approx. 9 million words). Compiled at the University of Vilnius, Lithuania[8]
- Reference Corpus of Contemporary Portuguese (CRPC)
- Turkish National Corpus[9]
- CoRoLa - The Reference Corpus of the Contemporary Romanian Language (Corpus reprezentativ al limbii române contemporane )
- TS Corpus - A large set of Turkish corpora. TS Corpus is a Free&Independent Project that aims to build Turkish corpora, NLP tools and linguistic datasets...
- MacMorpho - an annotated corpus of Brazilian Portuguese text
Slavic
[edit | edit source]East Slavic
[edit | edit source]- Belarusian N-korpus
- Russian National Corpus
- General Internet Corpus of Russian
- General Regionally Annotated Corpus of Ukrainian
- Ukrainian Language Corpus on the Mova.info Linguistic Portal
- Ukrainian Language Corpus
- Araneum Russicum
- Russian Corpus of Biographical Texts[10]
- RuTweetCorp[11]
- RusAge: Corpus for Age-Based Text Classification
South Slavic
[edit | edit source]- Bulgarian National Corpus[12]
- Macedonian Electronic Corpus[13]
- Croatian Language Corpus
- Croatian National Corpus
- Slovenian National Corpus
West Slavic
[edit | edit source]- Czech National Corpus[14]
- National Corpus of Polish
- Slovak National Corpora
German
[edit | edit source]- German Reference Corpus (DeReKo) More than 4 billion words of contemporary written German.
- Free corpus of German mistakes from people with dyslexia
Middle Eastern Languages
[edit | edit source]- Corpus Inscriptionum Semiticarum
- Kanaanäische und Aramäische Inschriften
- Hamshahri Corpus (Persian)
- Persian in MULTEXT-EAST corpus (Persian)[15]
- Amarna letters (for Akkadian, Egyptian, Sumerogram's, etc.)
- TEP: Tehran English-Persian Parallel Corpus[16]
- PTC: Persian Today Corpus: The Most Frequent Words of Today Persian, based on a one-million-word corpus (in Persian: Vāže-hā-ye Porkārbord-e Fārsi-ye Emrūz), Hamid Hassani, Tehran, Iran Language Institute (ILI), 2005, 322 pp. Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- Kurdish-corpus.uok.ac.ir (Kurdish-corpus Sorani dialect) University of Kurdistan, Department of English Language and Linguistics
- Bijankhan Corpus A Contemporary Persian Corpus for NLP researches, University of Tehran, 2012
- Neo-Assyrian Text Corpus Project
- Quranic Arabic Corpus (Classical Arabic)
- Electronic Text Corpus of Sumerian Literature
- Open Richly Annotated Cuneiform Corpus
- Asosoft text corpus[17] – Central Kurdish (Sorani)
- Thesaurus Linguae Aegyptiae (ancient Egyptian, Afro-Asiatic)
Turkic languages
[edit | edit source]- Uzbek national corpus (20 million words)
Devanagari
[edit | edit source]- Nepali Text Corpus (90+ million running words/6.5+ million sentences)
East Asian Languages
[edit | edit source]- Kotonoha Japanese language corpus[18]
- LIVAC Synchronous Corpus (Chinese)
South Asian Languages
[edit | edit source]African languages
[edit | edit source]- Amharic:[21]
- Creole (Gulf of Guinea): [22]
- Hausa:[23]
- Igbo: [24]
- Oromo: [25]
- Yoruba: [26]
- Zulu: [27]
Parallel corpora of diverse languages
[edit | edit source]- Chinese/English Political Interpreting Corpus (CEPIC) [28][29] consists of transcripts of speeches delivered by top political figures from Hong Kong, Beijing, Washington DC and London, as well as their translated/interpreted texts. Developed by Jun Pan and HKBU Library.
- Europarl Corpus - proceedings of the European Parliament from 1996 to 2012
- EUR-Lex corpus - collection of all official languages of the European Union, created from the EUR-Lex database[30]
- OPUS: Open source Parallel Corpus in many many languages[31]
- Tatoeba A parallel corpus which contains over 8.9 million sentences in multiple languages; 107 languages have more than 1,000 sentences each; a further 81 languages have from 100 to 1,000 sentences each.[32]
- NTU-Multilingual Corpus in 7 languages (ara, eng, ind, jpn, kor, mcn, vie)[33] (legacy repo)
- SeedLing corpus - A Seed Corpus for the Human Language Project with 1000+ languages from various sources.[34]
- GRALIS parallel texts for various Slavic languages, compiled by the institute for Slavic languages at Graz University (Branko Tošović et al.)
- The ACTRES Parallel Corpus (P-ACTRES 2.0) is a bidirectional English-Spanish corpus consisting of original texts in one language and their translation into the other. P-ACTRES 2.0 contains over 6 million words considering both directions together.[35]
{{#section-h::Parallel_text|Parallel corpora}}
Comparable Corpora
[edit | edit source]- Corpus of Political Speeches contains four collections of political speeches in English and Chinese from The Corpus of U.S. Presidential Speeches (1789–2015), The Corpus of Policy Address by Hong Kong Governors (1984–1996) and Hong Kong Chief Executives (1997–2014), The Corpus of Speeches given on New Year's days and Double Tenth days by Taiwan Presidents (1978–2014), and The Corpus of Report on the Work of the Government by Premiers of the People's Republic of China (1984–2013). Developed by HKBU Library.
- WaCky - The Web-As-Corpus Kool Yinitiative Web as Corpus (eng, fre, deu, ita)
- Disambiguating Similar Language Corpora Collection (DSLCC)[36] (Bosnian, Croatian, Serbian, Indonesian, Malay, Czech, Slovak, Brazilian Portuguese, European Portuguese, Peninsular Spanish, Argentine Spanish)
- Wikipedia Comparable Corpora(registration required) when (41 million aligned Wikipedia articles for 253 language pairs)
- The TenTen Corpus Family – comparable web corpora of target size 10 billion words. These corpora are available in the corpus management system Sketch Engine, currently, there exist TenTen corpora for more than 30 languages (such as English TenTen corpus,[37] Arabic TenTen corpus,[38] Spanish TenTen corpus,[39] Russian Tenten corpus,[40][41]). The overview of existing TenTen corpora can be found at https://www.sketchengine.co.uk/documentation/tenten-corpora/
- Timestamped JSI web corpora – web corpora of news articles crawled from a list of RSS feeds. Newsfeed corpora are being prepared in the framework of the project implemented by the Jožef Stefan Institute at Slovenian scientific research institute.[42] and published in Sketch Engine. More information about the project is on the project websites.
L2 (English) Corpora
[edit | edit source]- Cambridge Learner Corpus[43]
- Corpus of Academic Written and Spoken English (CAWSE),[44] a collection of Chinese students’ English language samples in academic settings. Freely downloadable online.
- English as a Lingua Franca in Academic Settings (ELFA),[45] an academic ELF corpus.[46][47]
- International Corpus of Learner English (ICLE),[48] a corpus of learner written English.
- Louvain International Database of Spoken English Interlanguage (LINDSEI),[49] a corpus of learner spoken English.
- Trinity Lancaster Corpus, one of the largest corpus of L2 spoken English.[50][51]
- University of Pittsburgh English Language Institute Corpus (PELIC)[52]
- Vienna-Oxford International Corpus of English (VOICE),[53] an ELF corpus.[46]
References
[edit | edit source]- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Professor Mark Davies at BYU created an online tool to search Google's English language corpus, drawn from Google Books, at http://googlebooks.byu.edu/x.asp.
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value). A search engine for the Google Books Ngram Corpus that supports wildcard queries and offers an API.
- ^ [1],Basque corpora
- ^ (in Spanish) Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Hadi Veisi, Mohammad MohammadAmini, Hawre Hosseini; Toward Kurdish language processing: Experiments in collecting and processing the AsoSoft text corpus, Digital Scholarship in the Humanities, fqy074, https://doi.org/10.1093/llc/fqy074
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ D. Upeksha, C. Wijayarathna, M. Siriwardena, L. Lasandun, C. Wimalasuriya, N. de Silva, and G. Dias . 2015. Implementing a Corpus for Sinhala Language. In Symposium on Language Technology for South Asia.
- ^ Glossa (uio.no)
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ https://arxiv.org/pdf/2102.06991.pdf, https://wortschatz.uni-leipzig.de/en/download/Hausa
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ https://www.researchgate.net/publication/336274457_Digital_Yoruba_Corpus, https://www.sketchengine.eu/corpora-and-languages/yoruba-text-corpora/
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Guy Emerson, Liling Tan, Susanne Fertmann, Alexis Palmer and Michaela Regneri . 2014. SeedLing: Building and using a seed corpus for the Human Language Project. In Proceedings of the use of Computational methods in the study of Endangered Languages (ComputEL) Workshop. Baltimore, USA.
- ^ H. Sanjurjo-González and M. Izquierdo. 2019. P-ACTRES 2.0: A parallel corpus for cross-linguistic research. In Parallel Corpora for Contrastive and Translation Studies: New resources and applications (pp. 215-231). John Benjamins Publishing.
- ^ Liling Tan, Marcos Zampieri, Nikola Ljubešic, and Jörg Tiedemann. Merging comparable data sources for the discrimination of similar languages: The DSL corpus collection. In Proceedings of the 7th Workshop on Building and Using Comparable Corpora (BUCC). 2014.
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Belinkov, Y., Habash, N., Kilgarriff, A., Ordan, N., Roth, R., & Suchomel, V. (2013). arTen-Ten: a new, vast corpus for Arabic. Proceedings of WACL.
- ^ Kilgarriff, A., & Renau, I. (2013). esTenTen, a vast web corpus of Peninsular and American Spanish. Procedia - Social and Behavioral Sciences, 95, 12-19.
- ^ Хохлова, М. В. (2016). Обзор больших русскоязычных корпусов текстов. In Материалы научной конференции" Интернет и современное общество" (pp. 74-77).
- ^ Khokhlova, M. (2016). Comparison of High-Frequency Nouns from the Perspective of Large Corpora. RASLAN 2016 Recent Advances in Slavonic Natural Language Processing, 9.
- ^ Trampuš, M., & Novak, B. (2012, October). Internals of an aggregated web news feed. In Proceedings of the Fifteenth International Information Science Conference IS SiKDD 2012 (pp. 431-434)
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ a b Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Juffs, A., Han, N-R., & Naismith, B. (2020). The University of Pittsburgh English Language Corpus (PELIC) [Data set]. Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
- ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).