FlashRSI · Field guide

Run a first self-improvement experiment.

Give an RSI method a defined editing scope. Run each candidate through a native harness and keep the evidence behind the score.

Start with the run →Understand the designOpen the project page

This quickstart runs SkillOpt with MiniAgent on 12 bundled SearchQA questions. It uses six questions for train, three for dev, and three for test. You need Python and two model configurations; Docker is not required.

What FlashRSI is

FlashRSI gives research teams one experiment interface for changing agent resources, running the complete candidate, and comparing the result across tasks. The method, harness, worker, and benchmark stay explicit in one configuration.

Defined scope

Algorithms declare which prompts, skills, tools, and runtime files they may edit.

Native execution

The full candidate runs through MiniAgent, Claude Code, or Codex.

Shared evidence

Snapshots, trajectories, usage, and scores stay under one run directory.

Broad coverage

12 methods and 30 benchmark suites use the same runner.

Install FlashRSI

From a terminal:

git clone https://github.com/RUC-NLPIR/FlashRSI.git
cd FlashRSI
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all]"

Python 3.11+ is supported. Python 3.12 covers the widest set of benchmark integrations. See Installation for Harbor, τ², Windows activation, and Docker.

Run the experiment

The Target answers benchmark questions. The Worker proposes changes to its skills. Set their credentials separately before running from the repository root:

export TARGET_API_KEY="your-target-key"
export WORKER_API_KEY="your-worker-key"

flashrsi run \
  --config examples/quickstart.yaml \
  --algorithm skillopt \
  --benchmark searchqa \
  --agent-harness miniagent \
  --model gpt-5-mini \
  --worker-model gpt-5-mini \
  --set agent.llm.api_key="$TARGET_API_KEY" \
  --set algorithm.worker.llm.api_key="$WORKER_API_KEY"

Use model names supported by your provider. For a custom endpoint, set agent.llm.base_url and algorithm.worker.llm.base_url. The complete parser reference is in Configuration & CLI.

Check setup first

Run flashrsi run --config examples/quickstart.yaml --check-only to check startup without model calls. It may prepare data and dependencies; it does not test remote model authentication.

Choose the parts of a run

Change one part while keeping the others visible. That is what makes a comparison useful.

AlgorithmHow to search

OPRO, GEPA, ACE, SkillOpt, RRSI, and other methods propose and select edits.

HarnessWhere to execute

A native harness provides the prompts, tools, runtime, and execution boundary.

BenchmarkWhat to measure

Tasks, data, grading rules, and splits turn a candidate into a comparable result.

See the supported algorithms, benchmarks, and harnesses before choosing a combination.

How a run works

Every experiment follows the same evidence path. The method can change; the boundary stays legible.

1. Capture the baseline

   Build the initial AgentSnapshot from the harness components and record its identity.

2. Propose a candidate

   The algorithm uses Target feedback or Worker edits inside its declared resource scope.

3. Run the complete candidate

   The native harness executes the candidate and records the trajectory.

4. Score and select

   The benchmark grades the trajectory; the algorithm keeps or rejects the Snapshot.

5. Inspect the evidence

   Read the config, resource lineage, calls, artifacts, and final test result together.

Inspect the output

The command prints the experiment directory. The sample configuration writes it under runs/, relative to the repository root.

File	What to look for
end-summary.md	Completion status, final test score, and recorded usage.
result.json	The selected snapshot_id, test_score, and test_tasks.
snapshots/	Initial and candidate resource versions.
trajectories/	Task answers, Worker calls, and selection decisions.
config.yaml	The resolved configuration with credentials redacted.

Check that the tasks ran, then inspect failed trajectories. This small sample checks the setup; its score is not a research result. To measure an improvement, evaluate the initial and selected Snapshots on the same held-out tasks.

Evaluate a saved Snapshot

Use any saved compatible candidate, including an intermediate Snapshot:

flashrsi evaluate \
  --config examples/evaluate_hotpotqa.yaml \
  --snapshot /path/to/run/snapshots/SELECTED_ID.json \
  --model gpt-5-mini \
  --set agent.llm.api_key="$TARGET_API_KEY"

This evaluates the destination benchmark's test split without starting a Worker or continuing the search. Read Snapshot evaluation before transferring resources between experiments.

From here

Run experimentsChoose a full runCompare methods, harnesses, splits, and output settings.Build on FlashRSIUnderstand the SnapshotSee how tagged resources create a portable candidate.Project pageBrowse the resultsOpen the benchmark surface and the evidence model.