Fine-tune Doctor (v0.71.27)

Most fine-tunes don't fail loudly. They fail *silently*: the model never stops generating, the assistant turns were never actually trained, or your DPO run quietly learns "longer = better" instead of the preference you meant. v0.71.27 "Fine-tune Doctor" is a pure-CPU, zero-GPU pre-flight that catches these before a single training step.

No competitor — Unsloth, Axolotl, LLaMA-Factory — ships any of these checks. They let you find out after the GPU hour.

soup data doctor — 8 chat-template checks

bash
soup data doctor chats.jsonl --model meta-llama/Llama-3.1-8B

It renders your data through the model's real tokenizer and chat template, then runs eight checks:

#CheckCatches
1chat_template presentTokenizer with no chat template at all
2template_renderA template that throws on your rows
3generation markersMissing {% generation %} spans (needed for response-only masking)
4eos_in_labelsThe #1 "model never stops generating" bug — every assistant turn's trained span must actually contain an EOS/EOT token (checked on *every* turn, not just the last)
5bos_duplicationTemplate *and* tokenizer both prepending BOS
6system_roleMistral-style templates that reject a leading system turn
7unknown_rolesRoles the template doesn't understand
8truncation_riskp95 rendered length vs max_length

The verdict uses the same OK / MINOR / MAJOR taxonomy as soup diagnose: exit 0 on OK/MINOR, exit 2 on MAJOR, so it drops straight into CI.

See the mask, don't guess it

bash
soup data doctor chats.jsonl --model meta-llama/Llama-3.1-8B --show-mask 3

--show-mask N prints N sample rows with every token coloured trained vs masked, rendered through the real collator path (answer-only, per-message train-field, or RAFT span-mask), not a reimplementation. An assistant-mask bug becomes visible instantly. The flags --train-on-responses-only and --train-on-messages-with-train-field pick the exact masking strategy soup train would use, so what you see is what you'll train.

Dataset-derived strings are stripped of C0 control characters before hitting your terminal (Rich's markup escape doesn't neutralise raw ESC bytes); the --output JSON is unaffected.

soup data lint — the preference-data linter

bash
soup data lint prefs.jsonl              # word-count length bias
soup data lint prefs.jsonl --model Qwen/Qwen2.5-7B   # exact token-length bias

For dpo / orpo / simpo / ipo / bco / kto data it flags:

  • `length_bias` — chosen systematically longer than rejected (the #1 silent DPO degradation), reported as a Cohen's d effect size so you know how bad it is, not just that it exists.
  • `label_imbalance` — KTO desirable:undesirable ratio out of balance.
  • `near_duplicates` — MinHash/LSH near-dupes (reuses the soup data dedup kernel).
  • `identical_pairs` — chosen == rejected, i.e. zero preference signal.
  • `prompt_leak` — the prompt echoed verbatim inside the completion.

Numbers

Validated live against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer; the smoke pass found and fixed two genuine bugs. Part of the release line that took the suite to 16,001 tests across 313 files.

See also

  • soup diagnose — the post-training report card whose OK/MINOR/MAJOR taxonomy the doctor mirrors.
  • Pre-flight & tooling — soup plan/apply, soup tunability, hardware-fit, the rest of the "catch it before GPU" surface.
  • MCP server — data_doctor is one of the tools exposed to your coding agent.