Overview
This environment was created as an evaluation-only benchmark for large language models, focusing on their ability to reason about crystalline materials, generate structured outputs, and improve performance across progressively more complex tasks.
Rather than testing surface-level text generation, the environment emphasizes scientific consistency, compositional reasoning, and constraint satisfaction.
Motivation
Many existing LLM benchmarks focus on natural language understanding or general reasoning. However, evaluating progress in scientific domains—especially materials science— requires more specialized tasks.
This environment was designed to answer questions like:
- Can an LLM reason about valid crystal compositions?
- Can it follow structural and chemical constraints?
- Does performance improve as models scale or are fine-tuned?
- Can errors be detected and penalized in a granular way?
Task Design
The environment contains a collection of evaluation tasks that test different aspects of crystal generation and reasoning. Tasks are intentionally diverse to avoid overfitting to a single pattern.
Key characteristics
- Progressive difficulty from basic composition checks to structured outputs
- Clear input/output contracts
- Deterministic evaluation whenever possible
- Explicit penalties for invalid or inconsistent generations
Supported Evaluation Tasks
The environment evaluates models across five distinct tasks. All tasks utilize the Materials Project (mp-20) dataset as the ground truth source.
1. Composition Constrained Generation
Goal: Generate a valid, stable crystal structure that strictly adheres to a given chemical formula
(e.g., SrTiO3). The model must produce valid lattice parameters and atomic coordinates while maintaining exact stoichiometry.
2. Symmetry-Composition Constrained Generation
Goal: A dual-constraint task requiring the generated structure to satisfy both a specific chemical composition and a target crystallographic space group (e.g., "Generate LiFePO4 in space group Pnma"). This tests the model's geometric reasoning capabilities.
3. Space Group Classification
Goal: Analyze a provided crystal structure (lattice vectors and atomic sites) and accurately identify its International Union of Crystallography (IUCr) space group number (1-230) or symbol.
4. Band Gap Prediction
Goal: Predict the electronic band gap (in eV) given a crystal structure. This tests the model's predictive capabilities.
5. Formation Energy Prediction
Goal: Estimate the formation energy per atom for a given crystal structure, a fundamental metric for determining thermodynamic stability and synthesizability.
Environment Usage
Quickstart
To run evaluations, you'll need to set up the server and then run the client evaluation script.
Step 2: Run Evaluation
Run the evaluation using vf-eval. Select the appropriate task configuration:
Crystal Generation Task
uv run vf-eval crystal_generation \
--num-examples 5 \
--rollouts-per-example 3 \
-m gpt-5 \
--env-args='{"task":"symmetry-composition"}'Classification Task
uv run vf-eval crystal_generation \
--num-examples 5 \
--rollouts-per-example 3 \
-m gpt-5 \
--env-args='{"task":"band_gap"}'Note: Use -a or --env-args to pass specific configuration as a JSON object.
Environment Arguments
The following arguments can be passed via --env-args to customize the environment:
| Arg | Type | Default | Description |
|---|---|---|---|
max_turns |
int | 1 | Maximum number of refinement turns (1 for single-shot) |
formation_energy_threshold |
float | 0.1 | Maximum acceptable formation energy (eV/atom) |
| Metric | Meaning |
|---|---|
reward |
Main scalar reward (weighted sum of criteria) |
Use Cases
- Comparing different LLMs on structured scientific reasoning
- Tracking progress during fine-tuning or alignment experiments
- Identifying failure modes in crystal or materials-related tasks
- Serving as a lightweight benchmark for research iteration
Why an Environment (Not Just a Dataset)?
By packaging these tasks as an environment, evaluation becomes:
- Reproducible
- Configurable
- Comparable across models and runs
This makes it easier to integrate into automated evaluation pipelines and to track longitudinal progress as models evolve.