Open-source model improvement lab

Change a model. Prove it. Keep the original.

RuVLLM trains and edits small adapters for open models, then measures each one against the unchanged base on the same tasks, under a rule written before the results. You get a keep or reject decision backed by every raw answer, and the original weights restore byte for byte.

Start the tutorial Scroll the story ↓ npm i @ruvector/ruvllm-gpu-loop

+50tool passes, 0 regressions (Qwen3.5 4B)

p = 1.8e-15exact paired McNemar test

$0.27GPU cost of that 256-task paired run

256/256base outputs identical across reruns

The story

One adapter, one rented GPU, one honest verdict

Follow a real run from frozen rules to confirmed cleanup. Every number below comes from committed result files in the repository.

01 · FREEZE

Write the rules before the results

RuVLLM freezes the tasks, expected answers, generation settings and the keep rule into a bundle pinned by SHA-256. 192 tool-use tasks and 64 arithmetic tasks, checked against every training prompt for overlap. Nothing can be tuned after the answers come in.

02 · RENT

Rent a GPU inside four spending limits

The GPU loop checks an admission budget, the offer's worst-case price, accrued cost, and arms an independent destroy timer before the job starts. Credentials stay on the control host; the rental only ever sees the pinned job.

03 · RUN

Run both arms on identical tasks

The unchanged base and the adapter answer the same 256 tasks with the same settings on the same physical GPU. The worker checks model files, adapter tensors and GPU identity first, and keeps every raw response, including failures and truncations.

04 · VERIFY

Re-score locally before believing anything

The raw outputs come home and the frozen scorer runs again on your machine. A job only counts as complete when the local report matches the remote one byte for byte. Generated tool calls and programs are never executed.

05 · VERDICT

123 → 173 tool passes, zero regressions

On the confirmation panel the coding-4b adapter for Qwen3.5 4B gained 50 tasks and lost none. Exact McNemar p = 1.8 × 10⁻¹⁵. Arithmetic held at 63/64 in both arms. Decision under the frozen rule: keep.

06 · HELD OUT

Fresh, untouched tasks: 149 → 176

A brand-new panel, never seen during development, confirms it: 40 gains, 13 regressions, p = 2.7 × 10⁻⁴, arithmetic 62 → 64. The regressions are all one kind (Write arguments), and RuVLLM shows them case by case instead of averaging them away.

07 · REJECT

A better-looking adapter, rejected

A second training round scored a perfect 192/192 on tools. It also dropped arithmetic from 62 to 46/64. The rule fixed before training said capability anchors must not regress, so it was rejected automatically. That is the point: the scoreboard can't be argued with after the fact.

08 · CLEAN UP

Destroyed, confirmed, accounted for

Each rental is destroyed and confirmed absent from the account before the job is marked done. One training run and three 256-task paired evaluations cost $2.04 of recorded GPU time in total.

Results

Measured on GPUs, decided by rule

Three arms on the same 256 untouched tasks. The base model's two independent runs produced identical text on all 256 tasks, so differences come from the adapter, not noise.

Held-out results: tool passes 149, 176 and 192 of 192 for base, coding-4b and coding-4b-r2; arithmetic 62, 64 and 46 of 64. — Held-out panel, RTX 4090, BF16.

Tool passes per family out of 24: grep 0, 24, 24; test 8, 24, 24; write 21, 8, 24; all other families 24 for every arm. — Where the adapter changes behaviour, per tool family.

Run	Tool passes	Gains / losses	Arithmetic	Paired p	Decision	GPU cost
Confirmation · coding-4b	123 → 173 / 192	50 / 0	63 → 63	1.8e-15	Keep	$0.27
Held out · coding-4b	149 → 176 / 192	40 / 13	62 → 64	2.7e-4	Keep	$0.91
Held out · coding-4b-r2	149 → 192 / 192	43 / 0	62 → 46	—	Reject	$0.76

Why RuVLLM

Model work you can defend

Rules locked first

Tasks, settings and the keep rule are SHA-pinned before generation, so a result can't be reshaped after you see it.

Paired, per-task statistics

Base and candidate answer identical tasks. Exact McNemar tests on gains and losses, not a single averaged score.

Byte-exact restore

Edits go to a copy inside policy limits; the Rust engine restores the original and proves it with matching hashes.

Cheap, capped GPU runs

Four spending limits and an independent destroy timer. A full 256-task paired evaluation costs under a dollar.

Every answer kept

Raw responses, receipts, logs and failure taxonomy stay with the result, ready for anyone to re-score.

Works where you work

npm CLI, Claude Code plugins over MCP, a Ruflo-compatible server, and an OpenAI-compatible local API on Apple Silicon.

Compare

Versus the usual fine-tune loop

What changes when evaluation is built into the workflow rather than bolted on afterwards.

Question	Typical ad-hoc workflow	RuVLLM
When is the success rule set?	Often after looking at scores	Before generation, SHA-pinned
How are models compared?	Aggregate accuracy, separate runs	Same tasks, same GPU, per-task exact McNemar
Are regressions visible?	Averaged into one number	Per family and per case, with a failure taxonomy
Held-out tasks?	Sometimes reused from development	Fresh panel, checked disjoint from training data
Can a third party re-check?	Usually only summary numbers	Raw outputs + frozen scorer, re-score locally
What happens to the original?	Overwritten or untracked	Untouched; byte-exact restore verified
GPU spend control?	Manual, easy to leave running	Budget, price, accrual caps + destroy timer + confirmed cleanup
Can a shiny result be rejected?	Rarely in practice	Yes: 192/192 tools rejected for an arithmetic loss

Tutorial

From install to your own verdict

Steps 1–5 run on a laptop with no GPU. Step 6 is where rented GPUs come in.

Install the CLI

The npm package ships the experiment controller, the GPU loop and the Vast.ai host adapter.

Needs Node 22+

npm install @ruvector/ruvllm-gpu-loop
npx ruvllm-gpu --help
npx ruvllm-gpu-loop --help

Verify the published evidence

Clone the repository and check the adapter files, frozen protocols, diagrams and saved benchmark generations against their recorded fingerprints.

Needs Python 3.12+ · no model download

git clone https://github.com/ruvnet/ruvllm.git
cd ruvllm
python3 scripts/check_repository.py
python3 scripts/verify_benchmarks.py

Look for "verified": true in both reports.

Make a real weight edit, then restore it

The Rust engine writes a tiny weight file, applies one allowed change, rejects forbidden ones and restores the original bytes.

Needs Rust 1.97.1 (pinned in native/rust-toolchain.toml)

cd native
cargo build --locked --release --bin weight_evolution
python3 smoke.py
cd ..

Expect "passed": true with equal source_sha256 and restored_sha256.

Run a bounded experiment

MetaHarness runs several candidate edits against one unchanged checkpoint, packages the evidence, restores every patch and ranks the qualified candidates. This fixture uses synthetic scores to show the workflow end to end.

(cd integrations/metaharness && npm ci --ignore-scripts)
mkdir -p runtime
node integrations/metaharness/src/cli.mjs fixture \
  --dir "$PWD/runtime/metaharness-fixture" \
  --binary "$PWD/native/target/release/weight_evolution"

Re-score a real GPU run yourself

The held-out run's raw answers are in the repository. Rerun the frozen scorer and get the regression breakdown by tool and failure type.

R=experiments/model-improvement
python3 $R/regression-dashboard.py \
  --bundle $R/frozen/v4 \
  --base $R/results/qwen35-4b-heldout-v4-coding-4b/base \
  --adapter $R/results/qwen35-4b-heldout-v4-coding-4b/adapter \
  --out runtime/v4-dashboard.md

Expect Keep for research, tool 149/192 → 176/192, and 13 Write wrong-argument regressions.

Schedule your own GPU run

Copy the example config to a private owner-only directory, set a spending ceiling, a model pin and a Vast offer, then tick the loop with the pinned host adapter. With budgetUsd: 0 nothing is ever submitted.

Needs a Vast.ai account key on the control host

cp integrations/metaharness/examples/gpu-loop.example.json /abs/private/gpu-loop.json
chmod 600 /abs/private/gpu-loop.json
A=integrations/metaharness/src/vast-host-adapter.mjs
npx ruvllm-gpu-loop status --config /abs/private/gpu-loop.json
npx ruvllm-gpu-loop tick --config /abs/private/gpu-loop.json \
  --adapter "$PWD/$A" --adapter-sha256 "$(sha256sum $A | cut -d' ' -f1)"

Full field reference in GPU-LOOP.md; Claude Code users can drive the same loop with /loop.

What's in the box

Components

Component	What you get	Built with
Weight editor	Inspect, patch, validate and restore Safetensors weights within a policy; RVF evidence packages	Rust
coding-4b adapter	1.5M-parameter LoRA for Qwen3.5 4B, PEFT and MLX formats, checksummed	Safetensors
Paired measurement	Frozen bundles, CUDA worker, exact McNemar scorer, regression dashboard	Python, Rust reports
GPU loop + Vast adapter	Restart-safe queue, reservations, deadlines, destroy timer, confirmed cleanup	Node (npm)
eval-report tools	Failure taxonomy, three-way comparisons, SFT pool building and panel freezing	Rust
Claude Code plugins	MCP tools and a guided weight-experiment command; Ruflo-compatible	Node, MCP
Local serving	OpenAI-compatible API for the adapter on Apple Silicon	MLX