40B · Gemma-4-class · Coding · Agentic · India-First

Applied Data & Tokenizer Decisions for a 40B India-First Model

The model-specific choices and the target numbers — fertility per language and domain, and the vocab size they imply.

Scope. This is the decision layer, not a re-teaching of the fundamentals. The general mechanics from the session — Chinchilla budgeting and repetition economics, verified-origin accounting, the always-on channel, two-stage classifier filtering, dedup scope, synthetic/model-collapse limits, provenance capture, proxy-model ablation, the train/eval firewall, and "tokenizer and corpus are one system" — are taken as established and are referenced only where a number depends on them.

The binding constraint is not the 40B parameter count. It is verified native-origin Indic supply and trustworthy agentic verifiers. Four objectives share one fixed budget, and a capability-blind selector starves the three scarce ones (Indic, India-first, agentic) to feed the abundant one (English web). The one distinction the framework does not stress, and the crux of "views the world from an Indian perspective": India-first is a sourcing property, not fluency. A model fluent in Hindi, Tamil and Telugu can still reason from a Western frame; India-first requires native-origin discourse and excludes translation — a strictly smaller pool than "Indic," tracked on its own line and tied to a benchmark.

01Data by stage — what to collect for these four objectives

Pre-training — target ~10T tokens

A specific mixture for the four objectives (a hypothesis to rank by ablation, not a fixed recipe).

DomainShare≈ TokWhy this share, for this model
General web (Eng + high-resource)50%5.0TThe Gemma-4 general-quality floor.
Curated high-quality10%1.0TBooks, Wikipedia, arXiv, StackExchange, textbooks — highest reasoning density.
Code17%1.7TCoding objective + independent lift to general reasoning; capped so it doesn't dent other domains.
Math / STEM6%0.6TReasoning backbone coding and agentic both lean on.
Indic (native + India-first + synthetic)14%1.4TThe differentiator; delivered via the protected channel.
Agentic / tool-use formatted3%0.3TNot to teach agency — to expose tool-call syntax early so RL doesn't start cold.
The one honest number the mixture hides Unique verified human-origin native Indic text is only ≈ 300–500B tokens. The 1.4T Indic line is reached by per-language repetition (Hindi and an Odia-tier language sit at different scarcity points, so passes are set per language) plus bias-checked synthetic. The India-first subset is smaller still and booked separately. Annealing (final ~10–20%) up-weights code, math, Indic, India-first and agentic formats and concentrates native-origin documents, since late data sets the model's default voice.

Post-training (SFT) — low millions of examples, four tracks

RL / alignment — where a bad verifier teaches the wrong behavior

Solution — Data
≈10T pre-training tokens at 50/10/17/6/14/3; four SFT tracks, execution- and outcome-verified; RLVR with the Indic-agentic track staged after English and preference from native raters. The load-bearing honesty: the 14% Indic line is really ~300–500B unique tokens stretched by per-language repetition + bias-checked synthetic, and India-first is booked as its own sub-line — never hidden inside "Indic."

02Cleaning — the departures the four objectives force

Standard filtering, dedup and extraction are per the session. What changes here: every setting is per-domain and per-language, so the abundant English pool never sets the parameters for the scarce three, which get the most conservative treatment. Only the columns that depart from the general-web default are shown.

StageCodeAgenticIndic / India-first
DedupNear-dup within reposLoose; allow repetitionLoose, within-language; semantic dedup on translations only
HeuristicParse / autogen / vendoredTrajectory structural validityPer-script thresholds, recomputed
QualityCode-native signalsVerifier-defined; track false-positive ratePer-language classifier from native positives
India-first—Scope Indic-agentic separatelyNew stage: classify native vs. translated → native subset feeds the protected channel
SyntheticExecution-verifiedVerifier-confirmedTeacher-bias-checked; cultural verification; "generated in Hindi" ≠ India-first
Decontam.Code firewallAgentic firewallCross-lingual (translated/transliterated) matching

Keep-and-reweight (2026): on scarce domains, quality filtering becomes a continuous per-sample weight, not keep/drop. Hard deletes are reserved for safety/PII, decontamination hits, and unverifiable agentic trajectories.

Solution — Cleaning
One rule decides everything: the abundant English pool never sets the parameters for the scarce three. Every filter, dedup scope and threshold is per-domain and per-language; a dedicated native-vs-translated stage feeds the protected channel; and keep-and-reweight replaces hard-delete except for safety/PII, decontamination, and unverifiable agentic trajectories.

03Fertility targets — the numbers the assignment asks for

Objective is parity, not minimum fertility: if Hindi encodes at 2× English, it gets ~half the gradient signal per unit of meaning under a fixed budget. Measured on parallel text (FLORES-style), per-byte, so Dravidian agglutination isn't penalized. Focus the ~11 highest-leverage languages (speakers × data × strategic weight); the long tail rides shared scripts, so 11 languages cost only ~8–9 script allocations.

Language(s)Script (shared)Target (tok/word)
EnglishLatin1.3–1.4 (floor)
Hindi (+ Marathi, Sanskrit, Nepali, Konkani, Maithili)Devanagari1.5–1.8
Bengali (+ Assamese)Bengali–Assamese1.6–1.9
GujaratiGujarati1.7–2.0
PunjabiGurmukhi1.7–2.0
UrduPerso-Arabic1.7–2.0
OdiaOdia1.8–2.1
TamilTamil1.9–2.3
TeluguTelugu1.9–2.3
KannadaKannada1.9–2.3
MalayalamMalayalam2.0–2.4

Hard constraint: no Indic target above ~2.2–2.4 tok/word; worst-case gap ≤ ~1.6× English. Dravidian sits highest from agglutination only; per-byte it's close to Indo-Aryan.

DomainGoalMove & target
CodeLow fertility is the goalWhitespace-run tokens (2/4/8/16, tabs), top-6-language keywords/ops, FIM sentinels. ~1.1–1.2× English prose (~3.5–4.5 bytes/tok).
MathConsistency, not compressionSplit digits (single, or 3-digit R→L). "1000"=1 tok but "1001"=2 breaks arithmetic. Operators/LaTeX atomic.
ScienceNo notation byte-soupFew-thousand slots for LaTeX macros, Greek, units, chem so \alpha, \nabla = 1 token.
AgenticDeterministic scaffolding~256–1,024 reserved specials (roles, tool-call open/close, JSON, XML), each ≈1 token; must co-tokenize with Indic text.

04Tokenizer vocabulary — derived from those targets

Bucket≈ Vocab share
English / general / Latin90–100K
Code (syntax, whitespace, identifiers, FIM)20–25K
Math + science notation5–8K
Agentic / structural specials1–2K
Indic (11 languages, ~8–9 scripts)120–135K
Byte fallback + headrooma few K
Recommendation~256K

Target 256K (operate 200K–256K). Sarvam's 68K works for a narrower goal (~10 Indic + English); with four competing objectives, 68K forces thinning Indic coverage or letting code/English fertility creep. 256K is the proven point for this profile (Gemma, SUTRA 256K, Granite 262K). Above ~262K, the embedding/softmax cost inference latency and rare Indic tokens under-train — exactly where fresh supply is already scarce. Non-negotiable switches: split-digit (arithmetic) and preserved-whitespace (code), plus the reserved agentic block. At 256K on a ~40B model, embeddings are ≈1.5–2B params — ~4–5% if tied (recommended). Start by adapting a strong Indic tokenizer and co-tune with the corpus, not building from zero.

Solution — Fertility → Tokenizer
Target parity (≤ ~1.6× English, no Indic language above ~2.4 tok/word) across 11 languages on 8–9 scripts, with split-digit math and atomic agentic specials. That allocation sums to ~256K vocab with tied embeddings — deliberately not 68K (too tight for four objectives) and not >262K (inference latency + under-trained rare Indic tokens, where supply is already scarce).

05Testing — borrow three, build the fourth

ObjectiveApproachInstruments
General (Gemma-4 parity)BorrowMMLU-Pro, GPQA-Diamond, IFEval — head-to-head vs Gemma 4, per-benchmark deltas + CIs. Certifies English axis only.
CodingBorrow (high trust)LiveCodeBench (time-gated), SWE-bench Verified/Pro. Eval harness = the RLVR reward.
AgenticBorrow harness, build verifier trustτ/τ²-bench, BFCL, WebArena, OSWorld. Human-confirm sampled "successes"; report false-positive rate.
Agentic × IndicBuildNo adequate benchmark exists — a research build.
Indic fluencyBorrowMILU, IndicGenBench, IndicQA/XTREME, BharatBench. Tests speech, not perspective.
India-first perspectivePartial + buildParamBench, VIRAASAT, DRISHTIKON, DOSA, IndianBhed as coverage + a custom probe.

Build the India-first probe around three failure modes generic sets miss: (1) counterfactual pairs — answers that change with frame (BNS vs. US law, Indian civic/tax procedure, kinship, contested regional history); (2) default probing — silent assumptions (₹ vs $, default sport, date format, "first floor"); (3) native code-mixed comprehension (real Hinglish/Tanglish). Score open-ended items with culturally native human raters.

Three traps beyond the session's firewall: the LLM judge carries the teacher's Western bias (measured translationese bias, worse in low-resource languages) — audit, ensemble, prefer native humans, report κ not raw match; cross-lingual metrics must be tokenizer-fair (bits-per-byte, never per-token loss); contamination spans pre-→mid-→post-training, so decontaminate and run the four-quadrant dashboard at every stage boundary to catch trades (does more code dent Indic?).

Solution — Testing
Borrow trustworthy instruments for general, coding, agentic and Indic-fluency; build the two artifacts that don't exist off the shelf — the India-first probe (counterfactual + default + code-mixed) and the Indic-agentic verifier; validate every verifier and judge before trusting its number; and score the four-quadrant mixture, tokenizer-fair, at each stage boundary against a pre-committed bar.