The model-specific choices and the target numbers — fertility per language and domain, and the vocab size they imply.
The binding constraint is not the 40B parameter count. It is verified native-origin Indic supply and trustworthy agentic verifiers. Four objectives share one fixed budget, and a capability-blind selector starves the three scarce ones (Indic, India-first, agentic) to feed the abundant one (English web). The one distinction the framework does not stress, and the crux of "views the world from an Indian perspective": India-first is a sourcing property, not fluency. A model fluent in Hindi, Tamil and Telugu can still reason from a Western frame; India-first requires native-origin discourse and excludes translation — a strictly smaller pool than "Indic," tracked on its own line and tied to a benchmark.
A specific mixture for the four objectives (a hypothesis to rank by ablation, not a fixed recipe).
| Domain | Share | ≈ Tok | Why this share, for this model |
|---|---|---|---|
| General web (Eng + high-resource) | 50% | 5.0T | The Gemma-4 general-quality floor. |
| Curated high-quality | 10% | 1.0T | Books, Wikipedia, arXiv, StackExchange, textbooks — highest reasoning density. |
| Code | 17% | 1.7T | Coding objective + independent lift to general reasoning; capped so it doesn't dent other domains. |
| Math / STEM | 6% | 0.6T | Reasoning backbone coding and agentic both lean on. |
| Indic (native + India-first + synthetic) | 14% | 1.4T | The differentiator; delivered via the protected channel. |
| Agentic / tool-use formatted | 3% | 0.3T | Not to teach agency — to expose tool-call syntax early so RL doesn't start cold. |
Standard filtering, dedup and extraction are per the session. What changes here: every setting is per-domain and per-language, so the abundant English pool never sets the parameters for the scarce three, which get the most conservative treatment. Only the columns that depart from the general-web default are shown.
| Stage | Code | Agentic | Indic / India-first |
|---|---|---|---|
| Dedup | Near-dup within repos | Loose; allow repetition | Loose, within-language; semantic dedup on translations only |
| Heuristic | Parse / autogen / vendored | Trajectory structural validity | Per-script thresholds, recomputed |
| Quality | Code-native signals | Verifier-defined; track false-positive rate | Per-language classifier from native positives |
| India-first | — | Scope Indic-agentic separately | New stage: classify native vs. translated → native subset feeds the protected channel |
| Synthetic | Execution-verified | Verifier-confirmed | Teacher-bias-checked; cultural verification; "generated in Hindi" ≠ India-first |
| Decontam. | Code firewall | Agentic firewall | Cross-lingual (translated/transliterated) matching |
Keep-and-reweight (2026): on scarce domains, quality filtering becomes a continuous per-sample weight, not keep/drop. Hard deletes are reserved for safety/PII, decontamination hits, and unverifiable agentic trajectories.
Objective is parity, not minimum fertility: if Hindi encodes at 2× English, it gets ~half the gradient signal per unit of meaning under a fixed budget. Measured on parallel text (FLORES-style), per-byte, so Dravidian agglutination isn't penalized. Focus the ~11 highest-leverage languages (speakers × data × strategic weight); the long tail rides shared scripts, so 11 languages cost only ~8–9 script allocations.
| Language(s) | Script (shared) | Target (tok/word) |
|---|---|---|
| English | Latin | 1.3–1.4 (floor) |
| Hindi (+ Marathi, Sanskrit, Nepali, Konkani, Maithili) | Devanagari | 1.5–1.8 |
| Bengali (+ Assamese) | Bengali–Assamese | 1.6–1.9 |
| Gujarati | Gujarati | 1.7–2.0 |
| Punjabi | Gurmukhi | 1.7–2.0 |
| Urdu | Perso-Arabic | 1.7–2.0 |
| Odia | Odia | 1.8–2.1 |
| Tamil | Tamil | 1.9–2.3 |
| Telugu | Telugu | 1.9–2.3 |
| Kannada | Kannada | 1.9–2.3 |
| Malayalam | Malayalam | 2.0–2.4 |
Hard constraint: no Indic target above ~2.2–2.4 tok/word; worst-case gap ≤ ~1.6× English. Dravidian sits highest from agglutination only; per-byte it's close to Indo-Aryan.
| Domain | Goal | Move & target |
|---|---|---|
| Code | Low fertility is the goal | Whitespace-run tokens (2/4/8/16, tabs), top-6-language keywords/ops, FIM sentinels. ~1.1–1.2× English prose (~3.5–4.5 bytes/tok). |
| Math | Consistency, not compression | Split digits (single, or 3-digit R→L). "1000"=1 tok but "1001"=2 breaks arithmetic. Operators/LaTeX atomic. |
| Science | No notation byte-soup | Few-thousand slots for LaTeX macros, Greek, units, chem so \alpha, \nabla = 1 token. |
| Agentic | Deterministic scaffolding | ~256–1,024 reserved specials (roles, tool-call open/close, JSON, XML), each ≈1 token; must co-tokenize with Indic text. |
| Bucket | ≈ Vocab share |
|---|---|
| English / general / Latin | 90–100K |
| Code (syntax, whitespace, identifiers, FIM) | 20–25K |
| Math + science notation | 5–8K |
| Agentic / structural specials | 1–2K |
| Indic (11 languages, ~8–9 scripts) | 120–135K |
| Byte fallback + headroom | a few K |
| Recommendation | ~256K |
Target 256K (operate 200K–256K). Sarvam's 68K works for a narrower goal (~10 Indic + English); with four competing objectives, 68K forces thinning Indic coverage or letting code/English fertility creep. 256K is the proven point for this profile (Gemma, SUTRA 256K, Granite 262K). Above ~262K, the embedding/softmax cost inference latency and rare Indic tokens under-train — exactly where fresh supply is already scarce. Non-negotiable switches: split-digit (arithmetic) and preserved-whitespace (code), plus the reserved agentic block. At 256K on a ~40B model, embeddings are ≈1.5–2B params — ~4–5% if tied (recommended). Start by adapting a strong Indic tokenizer and co-tune with the corpus, not building from zero.
| Objective | Approach | Instruments |
|---|---|---|
| General (Gemma-4 parity) | Borrow | MMLU-Pro, GPQA-Diamond, IFEval — head-to-head vs Gemma 4, per-benchmark deltas + CIs. Certifies English axis only. |
| Coding | Borrow (high trust) | LiveCodeBench (time-gated), SWE-bench Verified/Pro. Eval harness = the RLVR reward. |
| Agentic | Borrow harness, build verifier trust | τ/τ²-bench, BFCL, WebArena, OSWorld. Human-confirm sampled "successes"; report false-positive rate. |
| Agentic × Indic | Build | No adequate benchmark exists — a research build. |
| Indic fluency | Borrow | MILU, IndicGenBench, IndicQA/XTREME, BharatBench. Tests speech, not perspective. |
| India-first perspective | Partial + build | ParamBench, VIRAASAT, DRISHTIKON, DOSA, IndianBhed as coverage + a custom probe. |
Build the India-first probe around three failure modes generic sets miss: (1) counterfactual pairs — answers that change with frame (BNS vs. US law, Indian civic/tax procedure, kinship, contested regional history); (2) default probing — silent assumptions (₹ vs $, default sport, date format, "first floor"); (3) native code-mixed comprehension (real Hinglish/Tanglish). Score open-ended items with culturally native human raters.
Three traps beyond the session's firewall: the LLM judge carries the teacher's Western bias (measured translationese bias, worse in low-resource languages) — audit, ensemble, prefer native humans, report κ not raw match; cross-lingual metrics must be tokenizer-fair (bits-per-byte, never per-token loss); contamination spans pre-→mid-→post-training, so decontaminate and run the four-quadrant dashboard at every stage boundary to catch trades (does more code dent Indic?).