AltSlate Labs

Efficient, honest AI — small models trained from scratch, with the full recipe published.

We build compact language models that punch above their weight, and we release everything needed to trust and reproduce them: code, weights, evaluations, and reports.

🪰 The Jugnu family

Tiny language models pretrained from scratch (jugnu — जुगनू — means "firefly": small, but it glows). All under 150M parameters, all trained on 4× Blackwell GPUs, all Apache-2.0.

Model	Params	BLiMP	ARC-Easy	WikiText-2 (byte-ppl)
JugnuLM-53M	53.5M	78.1	51.4	2.04
JugnuLM-110M (R0)	109.7M	81.3	52.5	1.95
JugnuLM-110M-R1 — + value residuals	109.7M	81.1	54.7	1.94
JugnuLM-110M-R2 — + Muon	109.7M	80.8	56.1	1.93
JugnuLM-110M-R3 — + data blend	109.7M	81.8	53.6	1.91
JugnuLM-110M-R4a — + logit KD (heavy)	109.7M	80.4	57.0	2.18
JugnuLM-110M-R4b — + logit KD (light)	109.7M	79.3	55.5	1.92
🏆 JugnuLM-110M-R2+ — R2 recipe, scaled	109.7M	82.5	55.1	1.87

JugnuLM-110M-R2+ is our flagship — the kept R2 recipe (value residuals + Muon) scaled to 25B tokens under a WSD schedule. It posts the family's best BLiMP and perplexity, beats the baseline on every metric, and ranks #1 on the Tiny-ML Leaderboard efficiency score — a hair ahead of GPT-X2-125M, at a smaller size. (It's a value-residual model, so load it with trust_remote_code=True.)

JugnuLM-110M's 81.3% BLiMP ≈ GPT-X2-125M at ~12% fewer parameters and ~9× fewer training tokens.

The ablation ladder

Starting from an honest 110M baseline (R0), we add one lever at a time and keep only what beats the previous rung on the metric that matters:

R1 — value residuals (ResFormer): ARC-Easy +2.2 → kept.
R2 — Muon optimizer: ARC-Easy +1.4, lower perplexity → kept.
R3 — diverse data blend (+DCLM +FineMath): best BLiMP & perplexity, but ARC-Easy −2.5 → dropped. An honest negative — at this scale, FineWeb-Edu's educational filtering feeds ARC-Easy more than raw diversity does.
R4 — logit distillation (from a 1.7B teacher): brackets a narrow operating point — R4a (heavy KD) lifts ARC-Easy to a family-best 57.0 but hurts perplexity (2.18); R4b (light KD) recovers perplexity but loses the ARC gain → both dropped. Offline KD across a ~15× teacher–student gap isn't a clean win at this scale.
R2+ — the competitive run: take the strongest kept stack (R2) and scale it — 25B tokens + a WSD schedule → best-in-family BLiMP and perplexity, and #1 on the leaderboard efficiency score.

Training code & full recipe: github.com/AltSlate-Labs/jugnu

What we care about

Efficiency per parameter — the sub-150M regime: on-device, low-latency, cheap-to-serve models.
Reproducibility — public code, open weights, and one-command evals.
Honest reporting — clean baselines and measured ablations over unvalidated clever tricks. We publish what didn't work (see R3 and R4), not just what did.

AltSlate Labs

🪰 The Jugnu family

The ablation ladder

What we care about

Links