The 125M base, supervised fine-tuned to answer grounded questions from a passage — the strongest of the 125M variants on our evaluation.
A base completer turned into a question answerer by supervised fine-tuning the 125M base on our QA dataset, with the loss masked to the answer tokens only.
Trained on Modal L4 (loss masked to the answer tokens). On the blind-judge evaluation it scores 0.261 judged correctness [0.228–0.293] — the best 125M variant. At this scale it has learned the QA format and answers the easier questions, but reliable factual QA needs far more capacity.
| Class | LlamaForCausalLM (from scratch) |
| Layers | 12 |
| Hidden size | 768 |
| Attention | 12 heads · dim 64 · full MHA |
| Feed-forward | SwiGLU · inner 3,072 |
| Positional | RoPE · θ 10,000 |
| Norm | RMSNorm · ε 1e-5 |
| Context | 1,024 tokens |
| Vocabulary | 16,384 · byte-level BPE |
| Embeddings | tied input/output |
| Init from | slm-125m-e4 base |
| Method | full fine-tune |
| Trainable params | 125.8M — all |
| Training data | 15,000 QA pairs · 3 epochs |
| Training tokens | 10.7M (3.6M/epoch × 3) |
| Optimizer steps | 2,812 |
the full lineage. Unlike the 500M and Gemma sites, this base was pretrained from scratch here, so its cost is part of the total rather than someone else's.
| Stage | Detail | Cost |
|---|---|---|
| 125M pretraining (4 legs) | v1 + extended + e2 + e4 · 27.5 h on A100-40GB | $70.14 |
| QA supervised fine-tune | 2,812 steps · 9.7 min on L4 | $0.20 |
| Evaluation (shared) | 13 versions on 500 held-out questions, this model's share | $1.48 |
| Total | $71.82 |
Figures are Modal GPU usage (time × rate) across this model's lineage; shared datasets are charged at this model's share. Whether base pretraining is included is stated above — it is for the models pretrained here, and excluded for imported bases. Evaluation-derived metrics come from an independent blind-judge harness on a frozen, decontaminated held-out set. Serving is billed separately and scales to zero.