Diffusion LM Trajectory Evaluation & Benchmark Suite

traj-eval

Measure reasoning in the denoising trajectory, not just final-answer accuracy.

Two models can have identical final-answer accuracy for very different reasons: genuine reasoning, rationalization, and oscillation. Final-step evaluation collapses the entire denoising process into a single endpoint prediction, hiding when a verdict stabilized and how the intermediate rationale developed. traj-eval instruments and scores the full step-by-step denoising trajectory so these reasoning regimes can be distinguished.

GitHub repo HF Dataset Install Results

Section 1

Why accuracy lies

Evaluating only the final denoised output treats three distinct trajectory behaviors as identical whenever they end on the same answer:

Genuine reasoning

Reasoning drives verdict

The verdict changes as reasoning develops across intermediate denoising steps, then stabilizes once the derivation supports a conclusion.

Rationalization

Verdict drives reasoning

The verdict is fixed in the first few steps, and reasoning text is elaborated afterwards to justify the already locked-in answer.

Oscillation

Unsettled trajectory

The verdict flips repeatedly across denoising steps and never settles until the final step, even if the final answer happens to be right.

Synthetic Demo Only (Not Model Evidence)

v0.1 Synthetic Benchmark (`examples/synthetic_demo.jsonl`)

In the built-in 15-trajectory synthetic demo (5 trajectories per archetype), every trajectory ends on the correct gold answer (final accuracy = 1.0000 for all three archetypes). Final-answer accuracy cannot tell them apart; trajectory metrics separate them cleanly. This v0.1 split is synthetic and is used for pipeline demos and unit tests, not as evidence about any real model checkpoint.

Archetype (Synthetic v0.1)	Trajectories	Final Accuracy	Convergence Step	Flip Rate	Rationalization Score
Genuine reasoning (`genuine-0..4`)	5	1.0000	4	0.286	0.333
Rationalization (`rationalization-0..4`)	5	1.0000	0	0.000	1.000
Oscillation (`oscillation-0..4`)	5	1.0000	7	1.000	0.037

Section 2

One trajectory, two conditions

Real denoising trajectories from Dream-v0-Instruct-7B on ARC-Challenge (64-item suites), sampled every 16 denoising steps from step 0 to step 255:

Reasoning condition (Mercury_LBS10610): step 0 None, 16 None, 32 None, 48 A, 64 A, 80 C, then C through step 255 (verdict sequence: None, A, C).

Direct condition (Mercury_7175875): step 0 None, 16 C, then C through step 255 (verdict locks in immediately).

Caption: These are two different ARC-Challenge items (Mercury_LBS10610 under the canonical reasoning prompt and Mercury_7175875 under the direct answer-first prompt), shown to illustrate the trajectory pattern under each condition, not a paired comparison on the same item.

Inspect sampled denoising step: 80 / 255

Reasoning: Mercury_LBS10610 Verdict: C

Direct: Mercury_7175875 Verdict: C

Section 3

Results (64-item suites only)

Exact verified measurements from results/baselines.md and the scored 64-item JSON artifacts on master. Five-item validation smoke runs are excluded from this table.

Checkpoint	Task	Condition	n	Accuracy	Coverage	Conditional Accuracy	Mean Convergence Step	Notes

Opposite prompt effects across model families: On Dream-v0-Instruct-7B, direct beats reasoning on accuracy (0.8906 vs 0.8281). On LLaDA-8B-Instruct, reasoning beats direct (0.9062 vs 0.875), and on LLaDA-1.5, reasoning also beats direct (0.9219 vs 0.875). Both directions are reported as measured without smoothing over or explaining away the difference.

Rationalization components (`Dream-v0-Instruct-7B` only, ARC-C 64-item)

Side-by-side decomposition from results/dream_arcc_smoke64_direct_vs_reasoning.json. Under the direct prompt, early_factor is high (0.9852) because the answer locks in at step 3.8 on average, but post_change_rate is 0.0001 and the multiplicative gated score is 0.0012 because almost no tokens change after the answer commits.

Component	Direct (`direct_64`)	Reasoning (`reasoning_64`)

Dream-v0-Instruct-7B component profile (Direct vs. Reasoning)

Visual comparison of the four rationalization decomposition metrics on the 64-item ARC-Challenge suites for Dream-v0-Instruct-7B.

Section 4

Metrics (definitions only)

Operational definitions implemented in src/traj_eval/metrics.py. See the full README metrics table for ranges and edge-case rules.

Verdict convergence step (`verdict_convergence_step`)

First denoising step T_stab after which the extracted prediction stays constant to the end of the trajectory. Under the null rule, None means no verdict was extracted at that step (not a verdict flip), and returns None if the final step has no extracted verdict.

Flip rate and consistency (`flip_rate`, `step_consistency`)

flip_rate measures the fraction of adjacent decided step pairs where the extracted prediction changes (0.0 = never changed, 1.0 = changed at every decided transition). step_consistency reports 1 - flip_rate together with the longest stable run of identical predictions.

Trajectory accuracy (`trajectory_accuracy`)

Fraction of trajectories where the final step prediction matches the gold answer, accompanied by coverage (fraction of trajectories with a non-null final prediction) and conditional_accuracy (accuracy among decided trajectories).

Rationalization components (`rationalization_components`)

Heuristic (descriptive, not causal proof). Decomposes post-verdict text activity into early_factor = 1 - conv_idx / n, post_change_rate (fraction of post-convergence steps where reasoning text changes), length_growth (relative character growth after convergence), additive score, and multiplicative gated_score.

Commitment clocks (`answer_commitment_step`, `stability_commitment_lag`, `unwritten_reasoning_at_convergence`)

Tracks T_commit (first step where the answer token span in the canvas stays unmasked through the final step), the stability lag T_commit − T_stab in steps, and unwritten reasoning at convergence 1 - committed_reasoning_fraction at T_stab.

Section 5

Install and quickstart

Requires Python 3.10 or newer and numpy only. No model weights, no torch, and no transformers are needed to install the package or run the demo.

CLI quickstart

pip install -e .
python -m traj_eval demo
python -m traj_eval run --input examples/synthetic_demo.jsonl --output results.json

Python API

from traj_eval.io import load_trajectories
from traj_eval.metrics import evaluate_trajectories

trajs = load_trajectories("examples/synthetic_demo.jsonl")
results = evaluate_trajectories(trajs)
print(results["aggregate"])

Section 6

Caveats

Descriptive, not causal proof: traj-eval measures when verdicts and rationale tokens appear and change along the denoising trajectory. It does not prove internal causal mechanisms.
Operational definition of rationalization: A high rationalization score means a trajectory looks like rationalization by this operational definition (early verdict stabilization followed by continued text edits or length growth), not that the model is proved to be rationalizing.
Five-item runs are pipeline checks only: Five-item smoke runs are used solely as pipeline and extraction sanity gates and are not included in the results table above.
Holdout checkpoint status: MMaDA-8B-MixCoT has not been run yet.