Crystal Relaxation Architecture Shift

From MultiTurnEnv to RLM

The move from a classic multi-turn environment to an RLM harness was not a cosmetic refactor. It changed where the task lives, how the agent acts, what the evaluator can trust, and which failures are real science problems instead of format noise or environment plumbing.

Legacy loop: direct feedback in chat New loop: taskset + harness + ComposableEnv Outcome: harder benchmark, cleaner signal
Read the original crystal relaxation post Back to portfolio

In the crystal relaxation project, the old setup was good at testing whether a model could emit a plausible final CIF under a tight feedback loop. The RLM version asks a more serious question: can the model actually operate as an agent, use tools, inspect intermediate state, and converge on a physically better structure under a realistic workflow?

What changed, exactly?

The architecture change was a shift in responsibility. The old environment owned the loop. The new stack separates what the model should solve from how the model is allowed to solve it.

Before

Classic MultiTurnEnv

The environment stepped the conversation forward, injected feedback, and expected the model to return a final CIF through a chat-facing answer contract. That made iteration fast, but it also tightly coupled task logic, state management, and evaluation behavior.

  • The agent mostly saw score and feedback, not a real tool surface.
  • Answer extraction lived in the chat contract, so format failures could dominate the result.
  • Shared mutable state created a real concurrency bug before the corrected reruns.
  • The benchmark rewarded quick direct repair more than agent workflow quality.
After

RLM + ComposableEnv

The new setup uses a taskset for dataset and metadata, a harness for system prompt and tools, and ComposableEnv as the orchestrator. The model operates through files, checker calls, and sandbox state, with the final answer written to final.cif instead of squeezed through chat formatting.

  • The agent can inspect inputs, run code, write files, and revise against tool results.
  • Task definition and agent policy are cleanly separated.
  • The output contract is file-based, which sharply reduces extraction noise.
  • Failures become more diagnostic: symmetry, formation energy, composition, and workflow quality.

The new mental model

The migration plan boiled the RLM version down to three pieces. That decomposition matters because it gives each layer one job and makes it easier to reason about scaling, tooling, and debugging.

Taskset

Holds the dataset, task metadata, sandbox inputs, and instruction construction. It answers the question: what is the model solving on this rollout?

Harness

Defines the system prompt, available tools, and the working style of the agent. It answers the question: how is the model allowed to work?

ComposableEnv

Owns the rollout loop, state, sandbox orchestration, scoring handoff, and turn budget. It answers the question: how do these pieces run together at scale?

The point of the RLM transition was not to make the task easier. It was to make the benchmark honest.

Classic stayed stronger on short direct-feedback evals, especially for smaller models. But RLM exposed something more valuable: whether the model could actually operate under a tool-mediated scientific workflow instead of merely responding well to tightly curated chat feedback.

That distinction matters for training. If a rollout fails because a parser could not extract the answer, you learn almost nothing. If it fails because the model preserved composition but never found a low-energy structure, you learn where the agent policy is weak.

Experiments

75/75

Format reliability in RLM

Every completed RLM rerun achieved perfect format reward. The file-based final answer contract almost eliminated the answer-extraction failures that distorted classic runs, especially for Claude.

0.783

Best RLM baseline

GPT-5.5 reached a mean reward of 0.783 with 55 of 75 full-credit rollouts under the larger 15-turn, 5-check non-trusted-image configuration.

0.998

Classic still easier

The corrected classic environment let GPT-5.5 nearly saturate the rubric. That remained useful as a solvability reference, but it also confirmed that classic is a friendlier surface than the RLM harness.

0/75

Leak fixed, then moved past

After isolating classic environment state per rollout, the composition leak disappeared. That fix was necessary, but the migration still mattered because architecture and evaluation were too entangled in the old loop.

Full table results

crystal_relaxation

Classic MultiTurnEnv runs

Local eval id Model Run shape Mean reward pass@1 pass@2 Notes
8962a2f4openai/gpt-5.525x30.9981.0001.000Best overall classic run.
16d858abopenai/gpt-5.525x3, 3 turns0.7580.7600.880Strong but had 32% model/API errors.
536b00bagoogle/gemini-2.5-flash25x3, 2 turns0.6870.7470.933Strong classic baseline; 15/75 perfect rollouts.
4427177agoogle/gemini-2.5-flash25x3, 3 turns0.525 finite-only0.4130.600One invalid/NaN rollout made saved mean null.
1da43e7eopenai/gpt-5.4-mini25x3, 2 turns0.6240.5870.787Solid classic baseline.
b7aee56fopenai/gpt-5.4-mini25x3, 3 turns0.5080.4000.547Worse than the prior 2-turn classic run.
f2c5f410qwen/qwen3-8b10x3, 2 turns0.3320.2670.433Weak classic run.
83a9b09canthropic/claude-haiku-4.525x3, 2 turns0.1850.2000.333Weak; 54/75 zero-reward trajectories.
32315f0canthropic/claude-haiku-4.525x3, 3 turns0.2980.2670.413Improved over prior 2-turn classic run.

RLM

crystal-relaxation-rlm and crystal_relaxation_rlm runs

Read across each row to compare how the same model behaved as the RLM budget increased from the shorter 5-turn baseline to the 10-turn and 15-turn variants.

openai/gpt-5.5

5 turns / 1 check
0.618pass@1 0.493 · pass@2 0.587eval 45491238Best short-budget RLM run; 37/75 full-credit rollouts.
10 turns / 3 checks
0.341pass@1 0.240 · pass@2 0.347eval 003b7c4dFull rerun; lower than the 5-turn baseline.
10 turns / 3 checks
Non-trusted image, concurrency 5
0.516pass@1 0.440 · pass@2 0.520eval 34ff5575Cleaner dependency path; 33/75 full-credit rollouts.
15 turns / 5 checks
Non-trusted image, concurrency 5
0.783pass@1 0.733 · pass@2 0.760eval 60cf0943Best RLM run so far; 55/75 full-credit rollouts.

openai/gpt-5.4-mini

5 turns / 1 check
0.348pass@1 0.013 · pass@2 0.027eval 194e350bGood format/composition, weak energy and symmetry.
10 turns / 3 checks
0.314pass@1 0.013 · pass@2 0.027eval 870af5d2More checks improved composition but not final energy.
10 turns / 3 checks
Non-trusted image, concurrency 5
0.376pass@1 0.080 · pass@2 0.120eval 0d02feebReliable, but still weak on formation energy.
15 turns / 5 checks
Non-trusted image, concurrency 5
0.408pass@1 0.107 · pass@2 0.173eval 5dd698c6Modest gain from larger budget; 8/75 full-credit rollouts.

google/gemini-2.5-flash

5 turns / 1 check
0.322pass@1 0.160 · pass@2 0.280eval 2ec0705dViable after lowering concurrency.
10 turns / 3 checks
0.252pass@1 0.093 · pass@2 0.173eval e59f5b1dCompleted with 3/75 model or agent errors.
10 turns / 3 checks
Non-trusted image, concurrency 5
0.297pass@1 0.160 · pass@2 0.280eval 1476c619Similar pass rate to the older baseline, with more low-reward partials.
15 turns / 5 checks
Non-trusted image, concurrency 5
0.259pass@1 0.107 · pass@2 0.173eval 65760884Regressed under larger budget; 4% error rate.

anthropic/claude-haiku-4.5

5 turns / 1 check
0.320pass@1 0.067 · pass@2 0.133eval 23269e2eViable after lowering concurrency; physical checks remain weak.
10 turns / 3 checks
0.104pass@1 0.000 · pass@2 0.000eval d71cef5bDegenerated to format-only behavior.
10 turns / 3 checks
Non-trusted image, concurrency 5
0.303pass@1 0.133 · pass@2 0.200eval 01cf1c10Stable, but often hits the turn limit.
15 turns / 5 checks
Non-trusted image, concurrency 5
0.454pass@1 0.267 · pass@2 0.413eval 786ad9afBest non-GPT-5.5 larger-budget RLM result.

qwen/qwen3-8b

5 turns / 1 check
0.027pass@1 0.000 · pass@2 0.000eval eb656941Only recorded RLM run in this sweep.
10 turns / 3 checksNo run
10 turns / 3 checks
Non-trusted image, concurrency 5No run
15 turns / 5 checks
Non-trusted image, concurrency 5No run

Why the transition matters

A strict 1:1 comparison on short runs does not always favor RLM. Its value is that the harness is easier and cheaper to scale to longer trajectories, larger checker budgets, and broader sweeps, which is where the extra workflow signal starts to pay off.

It separates formatting from competence

In the classic environment, malformed tags or parser-hostile prose could erase otherwise useful structure generation. In RLM, the final artifact lives in a real file, so scoring is tied more tightly to the structure itself.

It turns tools into first-class training context

The model no longer has to simulate a workflow through text alone. It can inspect the unrelaxed CIF, revise files, call a checker, and use the sandbox the way a real agent would.

It makes failure modes legible

The hard problems become visible: composition mismatch, force-proxy failures, spacegroup collapse to P1, and failure to reach low formation energy. Those are the right problems to optimize.

What stayed hard

RLM fixed a lot of interface noise, but it did not solve the underlying scientific challenge. The migration reduced ambiguity. It did not remove difficulty.

Classic exposed quick-answer strength

The corrected multi-turn environment was fast and direct. Models that could infer a prototype and output a strong CIF immediately did well. GPT-5.5 was exceptional there, and the smaller models benefited from the simpler contract.

  • Direct validator feedback in chat encouraged fast local repair.
  • Short runs were useful for solvability checks and cheap baselines.
  • But the task mixed model quality with answer-format fragility.

RLM exposed workflow strength

The RLM harness rewarded models that could use a checker productively, manage files, and spend turns on diagnosis instead of surface repair. GPT-5.5 improved the most under larger budgets, while weaker models often remained stuck in local cleanup.

  • Claude benefited heavily from the file-based output contract.
  • GPT-5.4-mini often preserved chemistry but missed energy and symmetry.
  • Gemini showed upside, but concurrency and early stopping mattered.

Three lessons from the migration

The architecture shift surfaced a few durable principles that apply beyond crystal relaxation.

  1. State isolation is non-negotiable. The classic composition leak showed how quickly concurrent rollouts can invalidate a benchmark when environment state is shared implicitly.
  2. File-based outputs are better than chat-only answer contracts for tool tasks. RLM did not magically improve chemistry, but it dramatically improved validity and extraction reliability.
  3. More turns only help when the agent can spend them well. GPT-5.5 benefited from larger turn and checker budgets; weaker models often used the extra room for conservative, local edits that never crossed the scientific threshold.

The real outcome

The transition from MultiTurnEnv to RLM was a move from a tightly scripted evaluator to a more faithful agent environment. The old loop is still useful as a quick, easy-to-compare baseline. But the RLM harness is the better place to measure what matters next: tool use, revision strategy, and whether the model can turn partial scientific signal into a better structure.

That is why the most important result in the migration report is not just the top-line reward. It is the change in what the reward means. Once format failures and environment leaks stop dominating the outcome, the benchmark starts telling the truth.

Taskset Harness ComposableEnv Sandbox Workflow final.cif Cleaner Eval Signal