Training Methods

Soup supports 23 training tasks via the task config key. That number is the exact cardinality of the task field in the config schema, not a marketing round-up: the full list is in All Training Tasks below. Switching method means changing one line of YAML, not changing tools.

soup edit set (ROME / MEMIT / AlphaEdit) and soup steer train (CAA / ITI / RepE) are not task: values. They are weight-surgery and inference-time commands, documented under Unlearning & knowledge editing and RAG & steering.

If the model does not fit in your card, add training.stream_layers: true and the frozen base never enters VRAM at all. That works for sft and, since v0.72.4, for dpo, orpo, simpo and kto, with DPO's reference model taken from the same stream with its adapters switched off, so it needs no second copy of the model (on an untied checkpoint, dpo and kto still hold one copy of the output head per step, which the pre-flight charges). See Layer streaming and Preference losses over streaming.

Variables that move Soup's own files, and the ones that carry credentials, are listed on Environment variables and state files.

Warning:

Moving a soup.yaml from an older release? Since v0.76.0, dozens of combinations that loaded on v0.75 are refused with a message that names the field and what to change. Since v0.77.0 the 31 staged fields (validated, read by nothing) stop the load when set away from their default, each with a line ending refused as of v0.77. Delete the line.; so do data.remove_unused_columns: true and an explicit training.quantization: 4bit / 8bit (or load_in_8bit: true) on a task that never quantises, and so do fourteen of the fifteen settings upstream lists as accepted by v0.76 and never applied (the fifteenth, a KTO row with a malformed label, is dropped at data load with a reason). Run soup doctor --config soup.yaml before you train: it exits 2 on a refusal and 0 when the config loads, and it stops at the first cross-field refusal, so fix that setting and run it again until it prints no error (soup train exits 1 with Config validation error: on the same config). Since v0.77.2, task: sft also refuses reasoning_effort, train_on_eot, use_flash_attn and use_liger on the paths that never read them (vision and audio, backend: mlx, backend: unsloth, data.format: raft and pre_tokenized; which of the four each path refuses is in the table on that page, #2278). The list, grouped by what each setting touches, with the start of each message, is Settings Soup refuses at load; the runs that may need repeating are on the v0.77.0 and v0.76.0 release pages.

Warning:

Memory after upgrading to v0.77.0. training.gradient_checkpointing now reaches the trainer. With the default false, grpo, ipo, bco, reward_model and online_dpo train with checkpointing off (TRL's own default used to apply), and on backend: unsloth false now disables Unsloth's checkpointing. Set training.gradient_checkpointing: true to keep the memory you had. distill, classifier, reranker, cross_encoder, asr and unlearn now honour true too (#1778, #1981, #2140). See Speed and memory.

Supervised Fine-Tuning (SFT)

The most common method. Train on instruction-response pairs.

base: meta-llama/Llama-3.1-8B-Instruct
task: sft

data:
  train: ./data/train.jsonl
  format: alpaca

training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Two things SFT refuses since v0.74.0 (#535). A row that leaves nothing to learn is refused rather than trained as an all-masked sequence. On the text path with assistant-only masking on (the default, data.train_on_responses_only: true), a row whose assistant reply is missing, or is cut off entirely by data.max_length, stops the run before training with a message like train row 3: no causal-loss target remains after tokenization/truncation at data.max_length=2048; the assistant response is absent or fully truncated. Increase data.max_length or shorten the prompt. The row is named by split and number, and the same check covers pre-tokenized datasets. Long answers under a small max_length are how most people meet it. Second, the final save is refused (refusing to save the final model) when the last logged loss, grad_norm, entropy, train_loss or eval_loss is NaN or infinite, or when any trainable parameter is, so a run that diverged does not write an adapter that merely looks finished. A metric that was non-finite earlier in the run and then recovered does not trigger it.

Packing keeps the assistant-only mask (v0.76.0, #1236). With training.packing: true the loss mask built by data.train_on_responses_only (the default), train_on_messages_with_train_field, mask_history or a pre-tokenized labels column is carried through packing. Through v0.75.2, TRL 0.29.1 dropped the labels column when packing and fell back to training on every input token but the first, prompt and system tokens included. See Re-run this.

data.mask_history: true is read since v0.76.0 (#761). With the default data.train_on_responses_only: true it keeps only the last assistant turn of each conversation in the loss; every earlier assistant turn is masked along with the user and system turns. It matters only for multi-turn chat shapes (chatml, sharegpt and the other message lists): a single-turn row trains the same either way, and a flat format such as alpaca has no turns. With train_on_responses_only: false it is refused at config load, because every token would train, including the history it asks to exclude. The transformers and unsloth text SFT paths and task: distill honour it; backend: mlx and vision or audio SFT do not, and soup doctor --config reports it as ignored there.

SFT also writes a per-row loss record, <output>/rewind.jsonl (training.rewind_log, default true), which soup rewind reads to show which dataset rows sat in a step whose loss spiked: see soup rewind.

Direct Preference Optimization (DPO)

Train with preference pairs (chosen vs rejected).

base: meta-llama/Llama-3.1-8B-Instruct
task: dpo

data:
  train: ./data/preferences.jsonl
  format: dpo

training:
  dpo_beta: 0.1
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

prompt, chosen and rejected can each be a string or a message list. Since v0.76.1 every preference row reaches the trainer in one layout, all strings or all message lists with the prompt turns in prompt only; a row that cannot be written that way is dropped at load with a reason that names the field, and a dataset that mixes text rows with conversational ones loads as conversations, with a note. On v0.76.0, a string prompt beside chosen / rejected message lists that repeat the user turn, the shape of the bundled examples/data/dpo_sample.jsonl, made task: dpo, ipo, bco and preference fail inside trl after the model had loaded; on v0.76.1 they train on that shape. See Data formats for the rule.

On Apple Silicon, backend: mlx runs dpo natively since v0.77.0 (Soup's own MLX loop, TRL's sigmoid loss, reference log-probs precomputed from the base), and also orpo, simpo and kto; since v0.77.2 the unified task: preference with preference_loss dpo, orpo or simpo selects the same native trainers (ipo, bco and blends stay refused); see Preference training on MLX. On the transformers backend task: simpo cuts a prompt only when the row does not fit in data.max_length: the prompt is then cut to at most data.max_length // 2 tokens (a shorter prompt stays whole), keeping its end, and the answers get the room left; a row that fits is unchanged, whatever its prompt length (#1196, by @ousamabenyounes). DPO and ORPO keep their data.max_length // 2 cap. (Since closed in v0.77.2: through v0.77.1 a SimPO prompt over data.max_length // 2 tokens was cut even when the whole row would fit, #1185.) See Preference variety.

Group Relative Policy Optimization (GRPO)

Reasoning training (DeepSeek-R1 style) with reward functions instead of a reward model.

base: meta-llama/Llama-3.1-8B-Instruct
task: grpo

data:
  train: ./data/reasoning_train.jsonl
  format: sharegpt
  max_length: 4096

training:
  grpo_beta: 0.1
  num_generations: 4
  reward_fn: accuracy     # or 'format', 'verifiable', 'judge_rank', or a path to a custom .py
  gradient_checkpointing: true   # since v0.77.0 the default false means off on grpo
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Built-in reward functions:

• accuracy — 1.0 when the completion's final answer matches the gold's, else 0.0 (no partial credit); see How the final answer is read
• format — checks for structured <think>...</think> reasoning blocks
• judge_rank (v0.77.0) — an LLM judge scores each prompt's completions against each other, from 0 to 1; it needs training.grpo_judge and grpo_judge_max_calls, and sends your prompts and completions to the judge host. See GRPO Plus

On Apple Silicon, backend: mlx trains GRPO natively since v0.77.1 on the same objective (TRL 0.29.1's loss_type: dapo defaults) with the built-in accuracy and format rewards only: no custom file, no ensemble, no grpo_variant. Since v0.77.2 the base may also be one pre-quantized at 4 bits in affine mode (mlx-community/...-4bit), with the quantized layers frozen and only the LoRA adapter trained; any other quantized base is refused after the model loads. Upstream covers that path only with its hosted macOS MLX test job. See GRPO on MLX.

Custom reward functions — point to a Python file:

# my_reward.py
def reward_fn(completions, **kwargs):
    # kwargs carries your dataset's own columns, row-aligned with completions.
    golds = kwargs["answer"]
    return [1.0 if g in c[-1]["content"] else 0.0 for c, g in zip(completions, golds)]

What is in kwargs. Source columns are preserved and passed through, and the trainer additionally exposes the target as answer: an alpaca output, or the final assistant turn of a chat row. Required columns are validated before generation starts, so a missing one refuses the run rather than producing a silently useless reward: accuracy and the verifiable math domain need answer, verifiable code needs expected or answer, and json_schema needs schema.

What a reward function must return. Exactly one finite number per completion, in the order of completions. The GRPO trainer wraps every reward function (built-in, .py file or ensemble member) in a check, and stops with a message naming the function when it returns the wrong count, a non-number, nan or infinity, or a boolean: return 1.0 and 0.0, not True and False. A comma-separated reward_fn: "accuracy,format" trains an ensemble and is refused on every task except grpo; reward_fn: verifiable is refused unless verifiable_domain (math, code or json_schema) is set.

How the final answer is read

accuracy and the verifiable math domain read the completion and the gold with the same parser. The final answer is, in this order: the rest of the line after the last ####; the content of the last \boxed{} (nested braces are fine); what follows the last The answer is or Answer:, up to the end of its clause. A completion with none of these is read by its last line and its last number. A numeric gold is compared by value, so #### 1,000, \boxed{1000} and The answer is $1000. all match a gold of 1000; any other gold is compared as text, ignoring case, whitespace and LaTeX wrappers. Units are not stripped (42 apples does not match 42), and nothing is evaluated (\frac{1}{2} does not equal 0.5).

An answer phrase that names more than one distinct value is a hedge and states no answer: The answer is either 41 or 42. scores 0.0 against every gold. A gold that states no final answer, or hedges, is refused before generation, naming its split, row and the number of other rows with the same problem, because it would score every completion 0.0 and give GRPO no signal. math scores a numeric gold 1.0 within 1e-4 and 0.6 within 1e-2.

KL penalty, grpo_beta: 0, and the gradient watchdog

training.grpo_beta is the KL coefficient (default 0.1). Since v0.76.0 every grpo_variant applies it: gspo, dapo, dr_grpo, bnpo, two_sided and rft used to validate grpo_beta and drop it, so a variant run trained with no KL term (#1232). A recipe that relies on being KL-free should now say so with grpo_beta: 0, which is valid, skips the reference-model forward pass, and is refused together with reward_hack_mitigation kl_control or pid_lagrangian (they need a positive beta to steer).

Every task: grpo run also attaches a gradient watchdog (#342): if non-finite gradients (NaN or Inf) appear, the optimizer step does not update the weights, and the skipped-step count is logged at the end of the run (as a console warning when the fraction is 5% or more) and saved in trainer_state.json, where soup adapters audit reads it. Since v0.77.0 the step's warning also gives the count and the first few names of the parameters that went non-finite, and trainer_state.json records them (#2060). That is the transformers path. On backend: mlx (v0.77.1) the loop skips an update on a non-finite loss or gradient itself: the weights and optimizer moments are unchanged, the learning-rate schedule is held (the transformers path still advances its scheduler), and skipped_steps is recorded in adapter_config.json and the run result.

PPO / Full RLHF Pipeline

Three-step pipeline: SFT warmup -> Reward Model -> PPO alignment.

# Step 3: PPO alignment
base: meta-llama/Llama-3.1-8B-Instruct
task: ppo

data:
  train: ./data/prompts.jsonl
  format: chatml

training:
  reward_model: ./output_rm   # From step 2
  epochs: 1                   # passes over the prompt dataset
  ppo_epochs: 4               # optimization passes inside each PPO update
  ppo_clip_ratio: 0.2
  ppo_kl_penalty: 0.05
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

On task: ppo, reward_hack_detector, reward_hack_mitigation and echo_trap_enabled are refused when the trainer is built, before any model loads (#1441, #1514): the trl PPO trainer this build supports takes no reward functions, so the signal those callbacks read is never captured. They are wired on GRPO only. task: ppo itself is not refused and still runs; it is those flags that it refuses. soup train --help still describes --reward-hack-detector, --reward-hack-mitigation and --echo-trap-tokenizer-aware as applying to GRPO and PPO, which no longer matches. soup train --dry-run does not see the refusal. PPO also needs a train set of at least batch_size x gradient_accumulation_steps x processes rows, or soup train refuses before loading anything.

Gradient checkpointing on PPO (v0.77.2). Three changes, all by @ybai08, all read from upstream's changelog and tests (CPU tests; no GPU run). task: ppo now switches gradient checkpointing on for the policy when training.gradient_checkpointing asks for it (#2340): trl's PPO trainer runs its own loop instead of Trainer.train(), so outside quantization: 4bit / 8bit, where prepare_model_for_kbit_training already turned it on, the setting reached PPOConfig and nothing else and the policy trained without checkpointing. A run that asked for it is now checkpointed; upstream publishes no memory or speed figure for the change. With 4bit or 8bit plus gradient_checkpointing: true, task: ppo no longer crashes at the first rollout with AttributeError: 'PolicyAndValueWrapper' object has no attribute 'gradient_checkpointing_disable' (#2319). And task: ppo, grpo and online_dpo with checkpointing on no longer add a forward hook to the policy's input embedding at every step (#2373): the trl loops switch checkpointing off and back on around generation, and each switch registered one more hook that transformers never removed, so every generated token ran one more hook per step already taken. No speed or memory figure is published for any of the three. Upstream lists no re-run for any of the three, and the crash left no finished run.

The two epoch keys are different things. epochs (default 3) is complete passes over the training data; ppo_epochs is the optimization passes within each PPO update. Soup forwards both, plus ppo_kl_penalty, to the names the installed TRL PPOConfig uses.

All Training Tasks

All 23, exactly as the schema accepts them.

On backend: mlx only sft, dpo, orpo, simpo, kto (v0.77.0) and grpo (v0.77.1) run, plus task: preference with preference_loss dpo, orpo or simpo since v0.77.2; every other task is refused at config load. A kto row whose label is not a boolean, one of the documented strings or exactly -1, 0 or 1 is dropped at data load with a reason since v0.77.0 (#2126); through v0.76.1 such rows trained as desirable or undesirable. soup data validate --format kto counts them and prints the reasons for the first three, then ... and N more.

Validation split and evaluation

data.val_split (default 0.1) holds that share of the rows out of training, and since v0.76.0 the run evaluates them. Until then no trainer on the transformers backend scheduled an evaluation: the held-out rows were withheld from training and used for nothing, and val_loss stayed empty (#1223).

data:
  val_split: 0.1    # the default
training:
  eval_steps: 50    # optional; unset = evaluate at the end of every epoch

• Default: every epoch. At the end of each epoch the trainer runs one pass over the validation split and records eval_loss. It appears as the Val loss row on the live panel, as val_loss in the experiment tracker, and as TrainEvent.val_loss in the Web UI. Upstream states that runs take a few percent longer.
• training.eval_steps: N evaluates every N optimizer steps instead, counted after gradient accumulation like save_steps. The last step is evaluated too when it is not a multiple of N.
• Batch size. The evaluation runs at the resolved per-device train batch (after batch_size: auto), not Hugging Face's default of 8. Layer-streamed runs evaluate the same way.
• val_split: 0 trains on every row and evaluates nothing, unless the data brings its own validation split (a Hub dataset's validation split, or a pre-tokenized cache's val or validation split).

Generation-based tasks (grpo, ppo, online_dpo) do not hold rows out. Evaluating them means generating completions, which costs about as much per row as training on it. So data.val_split is ignored on them, every row trains, and the run prints a one-line note saying so. Their default config therefore trains on more rows than it did on v0.75.2. On grpo, set training.eval_steps to hold the split out: the validation number is then the held-out reward, eval_reward, shown as Val reward on the panel and stored as val_reward in the tracker (val_loss stays empty, because TRL's GRPO eval_loss sits near zero and does not measure held-out quality, #1389). ppo and online_dpo refuse eval_steps. On grpo the evaluation batch is the largest multiple of num_generations that fits in the train batch (TRL needs whole groups per evaluation batch), and Soup says so when that differs from the train batch. The evaluation's rewards never reach the reward-hack detector, the mitigation controller or the echo-trap detector, which read training generations only (from upstream's documentation, not run here). On backend: mlx the split is always held out (since v0.77.1): MLX GRPO scores it once after training and reports val_reward, and set data.val_split: 0 to train on every row. See GRPO on MLX.

Refused at config load: training.eval_steps on the MLX preference tasks (dpo, orpo, simpo, kto, which evaluate once per epoch), on task: grpo with backend: mlx (which scores its split once, after training), on task: unlearn, on ppo and online_dpo, and with val_split: 0 on local files or remote URIs, where there would be nothing to evaluate. prm and moe_lora_routing evaluate, but attach no live callback, so their eval_loss reaches the trainer's log history and not the tracker. On backend: mlx, task: sft honours eval_steps since v0.77.0 (#1990, #2074), in the same unit, completed optimizer updates after accumulation; unset, MLX keeps mlx-lm's own cadence. See Evaluation and small splits.

Which tasks quantize the base

training.quantization is 4bit unless you set it, except on eight tasks whose trainers never quantize: distill, classifier, reranker, cross_encoder, prm, moe_lora_routing, unlearn and asr. On them an unset value resolves to none, an explicit 4bit or 8bit (or load_in_8bit: true) is refused at load since v0.77.0, naming the task (training.quantization: 4bit has no effect on task='distill': its trainer never quantises the base, so no run can honour it. Delete the line or set quantization: none.; v0.76 warned and treated it as none), and a Quant Menu value such as gptq is refused too. Details are under Keys a task ignores.

Running Training

# Start training
soup train --config soup.yaml

# Resume from checkpoint
soup train --config soup.yaml --resume auto
soup train --config soup.yaml --resume ./output/checkpoint-500

# With W&B logging
soup train --config soup.yaml --wandb

# With TensorBoard
soup train --config soup.yaml --tensorboard

# With DeepSpeed (multi-GPU)
soup train --config soup.yaml --deepspeed zero2

# With FSDP2
soup train --config soup.yaml --fsdp full_shard

# Skip confirmation (pass --yes in scripts)
soup train --config soup.yaml --yes

# Continual learning: rehearse an old dataset so the new task
# does not erase it (v0.71.36, sft/pretrain only)
soup train --config new_task.yaml --replay old_task.jsonl --replay-ratio 0.1

Pass --yes in scripts. Since v0.77.0 soup train with stdin piped or redirected and no --yes exits 3 with stdin is not a TTY, so the start-training confirmation cannot be answered. before it loads any data, and piping y into it no longer answers the prompt (#2077, #2052). --dry-run and --cloud do not ask.

What a run records about its data (v0.77.0). soup train hashes a single local text-training file (data.train) at startup, before it loads or transforms it, and records the SHA-256 of the whole file as training_data_sha256 in the run record and the experiment tracker (#2024). It is the file registered as the training input, not proof of what the run trained on: it records source bytes at startup, not the transformed examples, and the file is not frozen. A file larger than the capture bound, or one that cannot be read whole, is recorded as unknown, as are mixed, interleaved, replay, remote, streaming, tokenized and directory inputs, TTS, unlearning and non-text modalities; --dry-run captures nothing. On the transformers backend, resident, layer-streamed and multi-GPU runs also record the Hub commit of the base they loaded; Unsloth and MLX loads, and local or cached-path hints, are not bound yet (#1902). See Registry.

--replay interleaves a seeded sample of old_task.jsonl into training; --replay-ratio is the fraction of the final mixed set. Validation stays pure new-task. See Data Moat II for the full mechanism and its measured limits.

The same three settings exist as config keys, for a soup.yaml you commit: data.replay (the path), data.replay_ratio (default 0.1; above 0 and at most 0.5, so 0.6 is refused at load) and data.replay_seed (0 to 2147483647; unset means seed 0). The ratio and the seed are refused without data.replay, data.replay is refused on any task other than sft and pretrain, and it is refused alongside training.packing or training.multipack, whose bin-packing breaks the ratio. A run that uses it prints Replay: +N old rows interleaved (P% of M).

Read the console before trusting --find-lr

soup train --config soup.yaml --find-lr sweeps the learning rate and writes a report (how it works), but the sweep is not a dry run of your training run. It loads the base model without quantization and without LoRA, puts the whole model on the device and trains every parameter with plain AdamW, one row at a time, on at most 256 tokens of each of the first rows of your data. So it needs the full-precision model to fit, and the rate it recommends is one for full fine-tuning, so it does not tell you the rate for the LoRA run you will launch. It reads each row's text, prompt or messages field only; a row in any other shape, an alpaca instruction and output row for instance, is tokenized as a blank string.

If the sweep cannot run at all (the config fails to load, the base or dataset cannot be loaded, the base does not fit unquantized, torch is missing), soup train --find-lr names the cause, exits 1 and writes no report (v0.76.0, #1203). Through v0.75.2 it printed a yellow line ending writing synthetic curve, wrote a placeholder report and exited 0, so a recommended learning rate from such a run described no model and no data.

The sweep takes one training row per step. With fewer rows than --find-lr-steps it runs one step per row over the same --find-lr-start to --find-lr-end range and says so; the recommendation needs at least 4 points, so --find-lr-steps must be at least 4 and a training set of fewer than 4 rows is refused before the model loads. If the loss turns non-finite partway through, the report covers the steps before it; if that leaves fewer than 4, the command says where it diverged instead of writing a report.

One v0.76.1 fix changes what a finished run trained: task: reward_model no longer trains on a text with the question in it twice, which it did on rows with a string prompt beside chosen / rejected message lists that repeat it, such as the examples/data/chat_preferences.jsonl that the bundled examples/configs/rlhf_step2_reward.yaml trains on (#1866). This was verified on v0.76.0, and the code that produced it is unchanged at every earlier tag this site's audit read, back to v0.40.0. Re-train such a reward model if it matters to you; see Re-run this on the v0.76.1 page.

Re-run this (v0.77.0)

Each of these is a finished run that trained something other than its config said, before v0.77.0. The wording of each cause is upstream's; the full list, with served answers, verdicts and data, is on the v0.77.0 release page.

Settings that loaded and were never applied, and are refused since v0.77.0, need no re-run: removing the line changes nothing about what trains. They include training.use_mod: true on a route with no router, a non-default moe_aux_loss_coeff on backend: unsloth or with stream_layers, neftune_alpha on the 14 tasks that never apply it, reasoning_effort and train_on_eot outside sft, tts and distill, lora.rank_pattern / alpha_pattern on MLX or Unsloth, vision or audio task: distill, quantization: 8bit on Unsloth, and vision or audio SFT on Unsloth (the last two cost memory or speed, not results). See Settings Soup refuses at load.

Re-run this (v0.76.0)

Each of these changed what a finished run trained. In every case the run looked normal: a loss curve and exit 0. The wording of each cause is upstream's; the PR numbers are the CHANGELOG's.