FineWeb-2 (kor_Hang)
17.5%
Web pages · ODC-By 1.0; Common Crawl terms of use
Web pages · No claims on the preparation; Common Crawl terms of use
Naver news comments (KcBERT corpus), DC Inside comments, NSMC movie reviews
25%
Online comments · CC BY-SA 4.0; Apache-2.0; CC0 1.0
OpenSubtitles v2024 (OPUS)
25%
Film and TV subtitles · OPUS: credit opensubtitles.org and cite Lison & Tiedemann 2016
Encyclopedia · CC BY-SA 4.0
Every text went through Shiberu's Korean analyzer (Kiwi, with the same word grouping and dictionary-form rule the Reader and the dictionary popup use), so each entry is a word as a learner looks it up: 있는 → 있다, 교무실에서 → 교무실, 공부했어요 → 공부하다.
Each source gives a rate per million words for every word. The blend is a weighted median: a word's score is the highest rate that sources holding at least half of the weight reach. The written side (web and Wikipedia) or the informal side (comments and subtitles) can carry a word, but no single source can lift its own oddities, such as Wikipedia's citation words or character names in subtitles. A word must appear in at least two sources.
Left out: numbers and anything with a digit, words written only in Latin letters or hanja, anything with a bare jamo (ㅋㅋ, ㅎㅎ, ㅠㅠ), and particles that punctuation cut off their noun.
Only counts are published: the files contain no text from any source.