FlashRSI documentation

Project page · Website quickstart · All integration notes · Preview and GitHub Pages hosting

To browse these pages as a documentation website, run npm ci and npm run docs:dev from the repository root (Node.js 22+), then open the printed URL.

Back to the project README · 简体中文 usage guide · English usage guide

FlashRSI lets you apply a resource-level self-improvement method to an existing agent, run the resulting candidate through its native harness, and measure it on a benchmark. Start with the guide if you want to run an experiment; use the integration and development sections when you want to add a method, benchmark, or harness.

Guide

Install

Use Python 3.11 or newer. Python 3.12 is recommended for the full benchmark matrix.

git clone https://github.com/RUC-NLPIR/FlashRSI.git
cd FlashRSI
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[all]"

Harbor is the only separate optional extra:

python -m pip install -e ".[all,harbor]"

Docker is required by Harbor and several task environments. Install Docker Desktop or Docker Engine, start the daemon, and check it with:

docker run --rm hello-world

The benchmark configuration index lists the service, Docker, data, and Python-version requirements for each benchmark.

Run an experiment

The bundled quickstart runs SkillOpt with MiniAgent on a 12-question SearchQA sample. Target and Worker models use independent settings:

export TARGET_API_KEY="..."
export WORKER_API_KEY="..."

flashrsi run \
  --config examples/quickstart.yaml \
  --algorithm skillopt \
  --benchmark searchqa \
  --agent-harness miniagent \
  --model gpt-5-mini \
  --worker-model gpt-5-mini \
  --set agent.llm.api_key="$TARGET_API_KEY" \
  --set algorithm.worker.llm.api_key="$WORKER_API_KEY"

For a longer run, start from a method-specific configuration in examples/. Use the experiment guide to choose an algorithm, harness, benchmark, Worker, and split.

Useful checks:

flashrsi run --config examples/quickstart.yaml --check-only
flashrsi run --config examples/quickstart.yaml --prepare-only

--check-only prepares dependencies and data without model calls or a result directory. --prepare-only only writes the redacted configuration directory.

Evaluate a Snapshot

A Snapshot is a saved, executable version of an agent's declared resources. Evaluate the selected or an intermediate candidate without continuing evolution:

flashrsi evaluate \
  --config examples/evaluate_hotpotqa.yaml \
  --snapshot /path/to/run-A/snapshots/SELECTED_ID.json \
  --check-only

flashrsi evaluate \
  --config examples/evaluate_hotpotqa.yaml \
  --snapshot /path/to/run-A/snapshots/SELECTED_ID.json

The current CLI uses the subcommand evaluate for the operation often called eval. It runs only the destination benchmark's test split; it does not create a Worker or continue search. See Snapshot evaluation for compatibility rules, split selection, and output files.

Configure an experiment

Configuration is YAML with three independent roles:

Role	Configuration	Responsibility
Target	agent	Executes benchmark tasks with the selected Snapshot.
Worker	algorithm.worker	Proposes edits or edits candidate files.
Benchmark	benchmark	Provides tasks, environments, and grading.

The most common fields are:

Area	Fields	Why it matters
Run	run.output_dir, run.mode, run.snapshot	Select output location or Snapshot evaluation mode.
Target	agent.name, agent.llm.*, agent.max_turns	Select a harness and its model/runtime settings.
Algorithm	algorithm.name, algorithm-specific fields	Select the search policy and budget.
Worker	algorithm.worker.type, algorithm.worker.llm.*	Choose an LLM Worker or Agent Worker.
Benchmark	benchmark.name, split, split_counts, data_path	Select data, task counts, and source paths.

Every nested field can be overridden from the command line with a dotted option or --set PATH=VALUE. Keep API keys in environment variables or command-line expansion; they are redacted from saved run records. See Usage guide for the complete configuration examples and Python API.

Integrations

Algorithms

FlashRSI currently registers twelve method adaptations. The shared runner supplies Snapshot checkout, rollout, scoring, and records; each algorithm keeps its own proposal, feedback, selection, and adoption policy.

Method	Scope	Main control	Detailed guide
OPRO	Instructions	Scored history	Configuration
GEPA	Instructions	Candidate pool and reflection	Configuration
EvoPrompt	Instructions	GA / DE population	Configuration
ACE	Instructions / playbook	Playbook accumulation	ACE
SkillOpt	Skills	Fast / slow updates	Quickstart
Evo-Harness	Skills	Task-stream accumulation	Evo-Harness
AHE	Harness resources	Successive edits and audit	AHE
ADAS	Harness resources	Design archive	ADAS
DarwinX	Harness resources	Branches and recombination	DarwinX
Meta-Harness	Harness resources	File-based history	Meta-Harness
ModularRSI	Modules and harness resources	Branch integration	ModularRSI
RRSI	Harness resources	Reliability and cost admission	RRSI

Implementations are adaptations to FlashRSI's resource and evaluation interfaces. A registered method is available through the runner, but registration does not mean every method, harness, and benchmark combination has a completed model run or reproduces an upstream paper exactly.

Benchmarks and datasets

Benchmark adapters own task loading and grading. They may read pinned Hugging Face data, a local data_path, a native task package, or a simulator. The Agent Snapshot remains separate from the benchmark environment: task files, Docker images, databases, service credentials, and verifiers are supplied by the benchmark integration.

Domain	Covered tasks	Details
Question answering	SearchQA, HotpotQA, MuSiQue, TriviaQA, Natural Questions, 2WikiMultiHopQA	QA setup notes
Reasoning and mathematics	BBEH, SuperGPQA, HLE, MATH, AIME, LiveMathBench	Lightweight benchmarks · Math
Coding	SWE-bench Verified, SWE-bench Pro, DeepSWE, Terminal-Bench	Coding setups · Terminal-Bench
Web research	BrowseComp, DeepSearchQA, WideSearch	Web services
Documents and tool use	OfficeQA-Pro, Harvey LAB, JobBench, SpreadsheetBench 2, AppWorld, BFCL, WorkBench, TravelPlanner	Configuration index
Interactive environments	τ²-bench, τ³ Banking, ALFWorld	τ² · ALFWorld

Use benchmark.split or split_counts to define an experimental train/dev/test size. Native test-only collections keep their source test role; a custom split should be reported as an experiment split rather than as an official leaderboard evaluation. The dataset cards record source, revision, license, and preparation details for released bundles.

Harnesses

A harness adapter declares the resources that may be versioned and calls the harness's native entry point. The same adapter can serve as a Target that solves tasks or an Agent Worker that edits a candidate.

Harness	Editable resources	Integration status	Details
MiniAgent	Prompts, skills, tools, configuration, runtime code	Available	Usage
Claude Code	Instructions, skills, agents, hooks, MCP/LSP, settings	Available	Agent integration
Codex	Python core, instructions, skills, plugins, memory, rules, configuration	Available	Codex guide
Pi	Instructions, skills, tools, settings	Doing	Integration contract
Hermes	Instructions, tools, skills, runtime configuration	Doing	Integration contract
OpenCode	Instructions, tools, plugins, settings	Doing	Integration contract

For task environments and Harbor boundaries, see Harness and benchmark responsibilities.

Developments

Framework and Snapshot

FlashRSI maps each harness into a tagged resource tree. A Snapshot stores the complete declared resource version used for execution; an editing scope controls what an algorithm can read or change. The surrounding resources remain in the executable candidate, so a skill or prompt is evaluated together with the tools and runtime that give it meaning.

Tagged Snapshot files
        │
        ├── scoped view ──► algorithm edits and commits a candidate
        │
        └── complete candidate ──► native harness ──► benchmark trajectory and score

Credentials, task data, temporary logs, and runtime state stay outside the Snapshot. Candidate lineage, changed paths, trajectories, scores, and usage are stored in the run directory. Read How it works and Snapshot evaluation before changing the resource boundary.

Add a benchmark

There are two common paths:

1. Harbor or another environment runtime. Use this when tasks need Docker, a verifier, a database, a simulator, or isolated dependencies. Keep the task package and environment setup in the benchmark integration; bind the environment runner to the Agent and return a normal Trajectory.
2. Raw files or a local/native runner. Use this for text, JSONL, documents, or a local simulator. Load tasks into Task objects, keep the source and revision in metadata, and implement a deterministic grader.

Register the benchmark, add a YAML example, document its metric and split policy, and add an offline contract test. Start with the benchmark interface, the configuration index, and Harbor integration notes.

Add a harness

Choose the smallest mapping that preserves the harness behavior:

• Closed or interface-first: map stable CLI, SDK, or configuration interfaces into tagged files. Keep proprietary internals in the native product.
• Clean open source: map the source tree directly, tag each file by role, and execute the selected Snapshot through the native entry point. MiniAgent follows this pattern.
• Complex open source: reduce or port the execution surface against a pinned upstream baseline, document supported behavior and migration limits, then align that implementation with the Snapshot. Codex uses a Python semantic port of supported paths from a fixed Rust baseline.

Add components.yaml, implement an Agent subclass and _run, return a Trajectory, and keep credentials and task state outside versioned resources. The Agent integration guide contains the adapter contract and the optional container path.

Add an algorithm

Subclass Algorithm, declare the observation and editing scope, and keep proposal, reflection, selection, and memory inside evolve. Use the shared Snapshot checkout, target rollout, benchmark grading, and record facilities. The runner should assemble collaborators; it should not decide the method's search policy.

Add a configuration example, a focused offline test for the algorithm's state transitions, and a short guide that states its scope, feedback, adoption rule, budget, and known differences from the upstream method. The usage guide's extension section and the existing algorithm modules show the expected boundaries.

FAQ

• What is the difference between a Snapshot and a run? A Snapshot is an executable resource version; a run is the record of an evolution or evaluation process that may contain many Snapshots.
• Can I reuse a Snapshot on another benchmark? Yes, with a compatible Agent resource root and component map. Use flashrsi evaluate and follow the cross-benchmark evaluation rules.
• Does a registered benchmark imply a completed result? No. Check the benchmark guide and run records for the supported track, environment, and validation status.
• Where should I start? Run the bundled quickstart, then read Choose and run an experiment.

Reference map

Need	Start here
First run	Quickstart
CLI and YAML settings	Usage guide
Algorithm and benchmark choice	Experiment guide
Snapshot transfer evaluation	Snapshot evaluation
Harness adapter contract	Agent integration
Benchmark environments	Benchmark index
Development history	Implementation notes