Preserving বাংলা in the age of AI

Why this matters

Today's AI treats Bengali as a second-class language

Bengali is spoken by more than 250 million people. Yet the systems shaping the future of knowledge, from ChatGPT, Claude, and Gemini to Indic-focused models, were built for English first. The gap is measurable, and we measured it.

Our approach

Build the language model Bengali deserves, from the data up

The constraint on a great Bengali model is not compute or model size. It is clean, deep Bengali data, and coverage of the dialects. So we build in that order: the data and the tools first, the model as what they make possible.

What we have achieved, measured and transparent

Two Bengali tokenizers, both ahead of every public baseline we tested

Both tokenizers need fewer tokens per Bengali word than every system we tested, including two Bengali-monolingual models and the frontier models, and neither breaks a conjunct while the others break many. This holds on six disjoint held-out sets: Wikipedia, literary and formal, general web, news, romanized Banglish, and FLORES+.

Not just an average

Deity names, a national poet, West Bengal places: one token, every time

A register average can hide how a tokenizer treats specific, culturally load-bearing words. We measured a fixed list of 13: deity names, a national poet, well-known West Bengal places, all conjunct-dense, on every tokenizer we track, 14 external systems plus our own.

Ours tokenizes every one of the 13 words as exactly one token, including the triple-conjunct আকাঙ্ক্ষা and the multi-akshara রবীন্দ্রনাথ. Correction, kept visible rather than quietly edited away: an earlier version of this page said "no exception" as if ours were the only one - BanglaBERT and BanglaT5 also score a perfect 1.00 here, added to this comparison 2026-08-17. What still sets ours apart: we guarantee it by construction, not by whatever a vocabulary happened to cover on these 13 words.

Beyond BPE, now built and measured

BMBT: reading Bengali on its own terms

The tokenizer above achieved that result the way almost every serious tokenizer does today: it retrains BPE, a statistical compressor, on Bengali data. That fixes the vocabulary while keeping the frame, and we thought the frame was the actual problem. Bengali is not a language whose structure needs discovering by counting bytes. It is an abugida, a generative system with explicit, Unicode-standardized rules for how a written syllable (an akshara) is built from a consonant, its conjuncts, and its vowel signs. So we built BMBT, Bornomala's Bengali Tokenizer: it reads the script by its own grammar first, a finite-state machine, and falls back to statistics only for what the grammar cannot explain: loanwords, code-mixing, noise.

Morphology is now built too. BMBT aligns its token boundaries to Bengali's suffix structure. Getting there meant separating two things we had been treating as one guarantee: never severing a conjunct, and never splitting a written syllable. Only the first is what a Bengali tokenizer exists to protect. Splitting the syllable shbe into its consonant cluster and its vowel sign leaves two units any Bengali reader names; severing the cluster itself leaves a fragment that means nothing. Holding the stronger rule was costing us a third of all Bengali morpheme boundaries, because a vowel sign attaches to the consonant before it while belonging to the suffix after it. Relaxing it only at morpheme seams took reachable boundaries from 62% to 100%, with not one conjunct severed, checked across 1.47 million characters.

That 100% figure describes the morphology rule layer itself, checked directly against text before any tokenizer was trained on it. The trained artifact built from it, bmbt-64k-morph, is now benchmarked too, on the same Wikipedia held-out set as every other row on this page: fertility 1.855, against 1.524 without the morphology layer, a real cost of about 22%, in the direction our own formal proof predicted (a grammar-constrained BPE is expected to do a little worse, never better, on raw token count). Most of that gap is the point, not a defect: 8.36% of splittable clusters now split deliberately at a morpheme seam. The rate that actually matters, real conjuncts broken, stays close to zero at 0.033% (6 of 18,396 splittable clusters) on this trained measurement, not the literal zero the rule layer alone guarantees, and we report the difference rather than round it away. Only the Wikipedia register has been run through this benchmark so far; the other three registers are the natural next measurement.

The same work made us correct our own measurement. Our conjunct-fragmentation figure counted any split written unit rather than only broken conjuncts, and divided by every unit including the 61% that consist of a single character and cannot be split by anyone. That made every number, ours and everyone else's, look about twice as good as it was. Corrected, the ranking of the other tokenizers changes: multilingual BERT, not DeepSeek-V3, does the most real damage to Bengali. We kept the old figure alongside the new one so nothing already published becomes uncheckable.

The tokenizer above still outperforms every external baseline tested, by a wide margin, on every held-out register, and BMBT ties it there too. Full numbers: the benchmark doc.

Read the position paper Read the formal spec Read the BMBT architecture

Where Bengali is actually typed

Banglish: a gap we found, measured, and partly closed

Most everyday Bengali on a phone is not written in Bengali script at all. It is Banglish, Bengali typed in Latin letters ("tumi kemon acho"), the register the six-way held-out comparison above already flags as the one where our own tokenizers score worst. Here is what that measurement found, and what we built in response.

Rare, and first of its kind

Things no one had done before

Our values and goals

Open, honest, and for the language

Help build it

This is bigger than one person

Project Bornomala is open and needs many hands. If you care about Bengali, about language technology, or about preserving a culture in the age of AI, there is a place for you.

Answers

Frequently asked