# Project Bornomala

> A first-of-its-kind, non-commercial research effort, based in West Bengal, to
> build a Bengali-first, dialect-aware language model and to preserve the Bengali
> language, its dialects, and its culture. It sits at the intersection of modern
> AI and a rich living language. Everything is open, measured, and transparent.

## The problem it addresses

- Mainstream AI (ChatGPT, Claude, Gemini) is optimised for English and treats
  Bengali as a second-class language: far more tokens per word, and it routinely
  breaks Bengali conjuncts (the written unit a reader recognises).
- Even Indic models (Sarvam, AI4Bharat) optimise across many languages and cover
  Standard Bengali, not the literary register or the West Bengal dialects.
- Bangladesh leads Bengali NLP but centres on Standard and Dhaka-region Bangla.
  The western half of the language is still missing. Bornomala is complementary.
- The deep Bengali literary corpus (1850 to 1950) exists mostly as scanned page
  images. This is a data problem, not a compute problem.

## What has been achieved (measured, reproducible)

- A Bengali tokenizer that outperforms existing public tokenizers on a measured
  comparison on held-out Bengali: fertility 1.52 tokens per word, ahead of
  AI4Bharat IndicBERTv2 (1.65), Sarvam-1 (2.59), GPT-4o (2.61),
  BrahmicTokenizer-131K (2.62), mBERT (2.78), and DeepSeek-V3 (2.99).
- It keeps 72 percent of words whole and almost never splits a Bengali conjunct
  (0.01 percent), while general tokenizers split between about 4 and 29 percent.
- Two tokenizers are shipped, not one. v1 finds Bengali's structure by counting;
  BMBT parses the script's own grammar with a finite-state machine, adds a
  featural decomposition of every syllable, aligns its token boundaries to
  Bengali's suffix structure (100 percent of morpheme seams reachable, no
  conjunct ever severed), and segments at 6.20 M characters per second against
  the C regex's 2.72 M. They tie exactly on compression, which the project's own
  formal proof predicts.
- The fragmentation metric was corrected: it had counted any split written unit
  rather than only broken conjuncts, and divided by units that cannot be split
  at all. Both the legacy and corrected figures are published.
- Method: trained on a literary-weighted corpus including the first 15,000
  Bengali Wikipedia articles; evaluated on 828 held-out lines from later
  articles, plus five further disjoint held-out registers (literary/formal,
  general web, news, romanized Banglish, FLORES+ - the exact corpus an
  external tokenizer-fertility paper's own numbers come from); every other
  tokenizer is its real public tokenizer on the same text. Nothing is
  estimated or fabricated. The closest rival is BanglaBERT (csebuetnlp), not
  IndicBERTv2 - added 2026-08-17, it beats IndicBERTv2 on every core
  register. To our knowledge, this is the first fully reproducible benchmark
  comparing modern Bengali tokenizers across compression, word preservation,
  and conjunct fragmentation using a common evaluation pipeline.
- Hard-words showcase: a fixed set of 13 conjunct-dense, culturally load-bearing
  words (deity names, the national poet Rabindranath Tagore, West Bengal place
  names) measured on 19 tokenizers. Ours (v1 and BMBT) tokenizes every one of
  the 13 as exactly one token - so do BanglaBERT and BanglaT5 (added
  2026-08-17), a correction from an earlier claim that ours was the only one;
  ours is the only one that guarantees it by construction rather than
  empirical vocabulary coverage. IndicBERTv2, the closest tokenizer not
  tied, averages 1.31 tokens/word here and still splits 3 of the 13; every
  other system averages 3.3 to 11.1 tokens/word.
- BMBT's morphology layer, trained and benchmarked on Wikipedia held-out:
  fertility 1.855 with the morphology layer on, against 1.524 without it, a
  real ~22% cost, in the direction the project's own formal proof predicted.
  8.36% of splittable clusters split deliberately at a morpheme seam;
  real conjuncts broken stays close to zero at 0.033%, not the literal zero
  the rule layer alone guarantees. Only the Wikipedia register is benchmarked
  so far.
- Banglish (romanized Bengali chat text): both tokenizers measured dead last,
  17th of 17 loadable tokenizers, on real romanized Bengali. Built a
  tiered fix: tiers 0-2 (a 155,615-entry lookup table plus an n-gram
  classifier, no model, no GPU) resolve 85.6% of real Banglish words and
  reach fertility 1.740 after transliterating to Bengali script first,
  better than every tokenizer on the leaderboard including the previous
  leader SUTRA (1.850). A tier-3 neural model, trained from scratch on a
  real GPU, resolves the harder remainder at 53.9% exact-match accuracy and
  13.5% character error rate on a reserved blind test split, real and
  honestly short of production-ready, blocked on more training compute.
- Track A2 (Gate G3): measured how much of a Bengali corpus survives dedup
  and quality filtering, across four real sources. Word survival ranges from
  79.6% (CC-100, the one genuine raw-web proxy) to 99.3% (AI4Bharat
  IndicCorp v2), comfortably clearing the programme's 5-billion-token floor
  at every ratio measured.
- Data-collection tooling for Track B (OCR) and Track C (dialect) now exists:
  a schema, a validator, and CSV templates, tested against real and
  deliberately-broken data. Infrastructure only; no real dialect or OCR data
  has been collected with it yet.

## Values

Open (Apache 2.0 code, Creative Commons data), transparent (every number states
its data), non-commercial research, cultural preservation, and consent-based
dialect and speech collection.

## Links

- Repository: https://github.com/konkomaji/bornomala
- Founder: Konko Maji (work.konkomaji@gmail.com)
- License: Apache-2.0 (code), CC BY / CC BY-SA (data)