The Gym System: Self-Improving AI Through Measured Practice

How structured generate → evaluate → learn loops close the quality gap — turning AI output from "good enough" to measurably excellent, one iteration at a time.

1. The Problem: Same Mistakes, No Measurement

AI systems make the same mistakes repeatedly. An LLM asked to draw a diagram will produce misaligned boxes. Asked again tomorrow, it will produce the same misaligned boxes. An extraction pipeline will let newsletter CTAs bleed into clean text. Run it next week — same CTAs survive.

The root cause is not capability. Modern LLMs can draw aligned diagrams and can clean text well. The problem is threefold:

The result: human experts spend their time correcting the same classes of errors, session after session, instead of teaching the system to avoid them permanently.

2. The Core Idea: Gyms as Steering Modules

A gym is a steering module that scores a system's output, surfaces failures, and closes the quality loop. It is not a training framework — it does not change model weights. It is not a prompt optimizer — it does not search prompt space. It is a quality feedback system that produces reusable artifacts: skills, conventions, validation tools.

One cycle of the loop:

The key difference from "just iterate on the prompt": every cycle produces durable artifacts that benefit all future work, not just the next run. A skill file created in the ASCII diagram gym improves every diagram the system draws, forever.

3. How It Works

Each gym is defined as a vario recipeA YAML configuration that defines a data pipeline: source items, processing steps, and evaluation functions. Vario is the pipeline engine that executes recipes. — a YAML file that wires together source items, task functions, and scoring functions. The gym CLI orchestrates execution:

gym run ascii_diagrams    # Run the full gen-eval-learn cycle
gym run extraction        # Run extraction quality evaluation
gym list                  # Show all registered gyms and their status

Evaluation is hybrid by design. Programmatic validators catch structural defects (misaligned borders, missing content anchors) with 100% reliability and zero cost. LLM judges assess qualitative dimensions (visual balance, readability, semantic coherence) that no validator can capture. The combination covers more ground than either approach alone.

4. Gym A: ASCII Diagrams

The ASCII diagram gym is the most complete demonstration of the gen-eval-learn loop. It took diagrams from a quality score of 0.42 to 0.86 — a 105% improvement — by identifying five systematic failure patterns and encoding fixes as a permanent skill.

Round 1: Naive Generation

Ten diagram challenges were generated without any skill or convention guidance. The prompt was simply: "Draw an ASCII diagram of: {description}." The results revealed five systematic failure patterns:

+----------+     +----------+     +--------+
| produce  | --> |  score   | --> | reduce |
+----------+     +----------+     +--------+

╔═══════════════════════════════════════╗
║            Pipeline                   ║
╠═══════════════════════════════════════╣
║                                       ║
║  ╭──────────╮   ╭──────────╮          ║
║  │ produce  │━━▶│  score   │          ║
║  ╰──────────╯   ╰─────┬────╯          ║
║                        │              ║
║                        ▼              ║
║                 ╭──────────╮          ║
║                 │  reduce  │          ║
║                 ╰──────────╯          ║
╚═══════════════════════════════════════╝

Cloud
|
+-- Region A
|   +-- Service 1
|   +-- Service 2
|
+-- Region B
    +-- Service 3
    +-- Service 4

╔════════════════════════════════════════════════╗
║                     Cloud                      ║
╠════════════════════════════════════════════════╣
║                                                ║
║  ┌────────────────────┐ ┌───────────────────┐  ║
║  │   Region A         │ │   Region B        │  ║
║  │ ╭──────╮ ╭──────╮  │ │ ╭─────╮ ╭─────╮   │  ║
║  │ │Svc 1 │ │Svc 2 │  │ │ │Svc 3│ │Svc 4│   │  ║
║  │ ╰──────╯ ╰──────╯  │ │ ╰─────╯ ╰─────╯   │  ║
║  └────────────────────┘ └───────────────────┘  ║
║                                                ║
╚════════════════════════════════════════════════╝

The Five Failure Patterns

The Skill: From Patterns to Permanent Fix

All five patterns were encoded into a 187-line skill file (~/.claude/skills/ascii-diagram/SKILL.md) that is loaded into every future session. The skill includes:

• Box Borders by Level: A lookup table mapping four hierarchy levels to specific Unicode character sets (double for containers, single for components, rounded for leaves, heavy for emphasis)
• Verification Procedure: Explicit instruction to count display widths after drawing, with a display_width() Python function as a concrete example
• Inside-out Process: Six steps — identify hierarchy, sketch layout, draw innermost boxes first, add arrows, align, verify
• Common Mistakes: Seven anti-patterns with "why it's wrong" and specific fixes

Round 2: Measured Improvement

5. Gym B: Content Extraction

The extraction gym measures how well the system converts raw HTML pages — full of navigation bars, newsletter CTAs, social share buttons, and ad scaffolding — into clean article text. The challenge: strip the 90–99% boilerplate while preserving every word of actual content.

The Pipeline

Extraction runs a three-stage pipeline: fetch (HTTP request) → extract (readability + markdownify, strips HTML structure) → clean (optional LLM pass to remove surviving UI chrome). Each stage is independently measurable.

Programmatic Scoring

The scoring function uses three deterministic checks, weighted by importance:

The Motivating Example: Substack Newsletters

Baseline vs. LLM-Cleaned Results

What the Gym Revealed

Paywall detection: The substack_mauboussin site extracts only 76 characters from 89KB of HTML — a paywall blocks content access. The gym's scoring correctly identifies this as a failure (0.35), and the LLM cleaning step cannot fix a content-access problem. This surfaced the need for paywall-aware fetch strategies.

Compression ratios as diagnostics: The gym tracks raw HTML → extracted → cleaned character counts. A 178KB → 11KB → 10.6KB progression (94% compression) is healthy. A 89KB → 76 byte extraction signals a structural problem before any scoring runs.

6. Gym C: Judge Calibration

This is the meta-gym — the system that checks its own checkers. If your LLM judge gives a score of 0.85, what does that mean? Is it well-calibrated? Would it give a lower score to objectively worse output? The judge calibration system answers these questions through two key mechanisms.

Monotonicity Testing

The core idea is elegant: if a judge is well-calibrated, deliberately degraded text should score lower than the original. The system applies seven perturbation types, scores both versions, and measures whether the judge consistently detects the degradation:

Calibration Metrics

For each perturbation type, three metrics determine whether the judge passes:

• Mean drop: average(original_score − perturbed_score). Want > 0. A judge that scores degraded text equally is useless.
• Effect size (Cohen's d): standardized difference. Want > 0.5 for meaningful discrimination. This accounts for score variance — a mean drop of 2 points is meaningless if scores swing by 20 points randomly.
• Percent correct: fraction where original > perturbed (strict). Want > 70%. A judge should get the direction right at least 7 times in 10.

A judge passes a perturbation type if mean_drop > 0 AND effect_size > 0.5. The overall calibration score is the fraction of perturbation types passed.

Discrimination Analysis

The screening criteria reject judges that:

• Cluster scores — more than 60% of scores in a 20-point band indicates the judge is not using the full scale
• Fail monotonicity — if deliberately worse text scores the same, the judge cannot discriminate quality levels
• Add zero information as repeats — same-model repeats with the same output add no statistical power

7. Prior Work & How Gyms Differ

8. Metrics

ASCII Diagram Gym

Extraction Gym

Quality Ratings (where applicable)

Using the parallelogram scale for overall gym maturity:

9. Demo

The gym system runs from the command line. Here is what a typical session looks like:

Running a Gym

# Run the ASCII diagram gym
$ gym run ascii_diagrams
[INFO] Loading recipe: learning/gyms/ascii_diagrams/gym.yaml
[INFO] Source: 10 challenges loaded
[INFO] Generating: pipeline... architecture... nested_containers... rating...
[INFO] Evaluating: alignment=0.85, features=1.00, conventions=0.83, judge=0.78
[INFO] Overall: 0.86 (was 0.42 on last naive run)

# Run the extraction gym
$ gym run extraction
[INFO] Loading recipe: learning/gyms/extraction/gym.yaml
[INFO] Source: 6 sites loaded
[INFO] Fetching: substack_waxman... reuters_article... paulgraham...
[INFO] Scoring: intrusion=0.92, anchor=0.83, must_not=1.00
[INFO] Overall: 0.89

# Check all gym statuses
$ gym list
NAME              STATUS    LAST SCORE    LAST RUN
ascii_diagrams    Done      0.86          2026-03-21
extraction        Active    0.89          2026-03-21
badge             Done      —             2026-02-28
fetchability      Done      —             2026-03-01
kb                Done      —             2026-03-02
assessor          Done      —             2026-03-05

Running the Validator Directly

# Check a specific file for ASCII diagram alignment issues
$ python tools/check_ascii.py present/gyms/report.html
Checking present/gyms/report.html...
  Block at line 142: 19 lines, all width 64. ALIGNED.
  236 code blocks scanned across repo.
  585 alignment issues found in 236 blocks.

What to Watch For

In a live demo, the key moments are:

1. The naive generation — watch the model produce a diagram with plain +---+ borders and ragged alignment. This is the starting point.
2. The scoring output — watch the validator catch specific issues: "line 5: content width 40 != box width 41 (off by 1)." These are issues invisible to the human eye in a proportional font.
3. The re-generation — with the skill loaded, watch the model produce a properly hierarchical diagram with aligned borders. Same model, same prompt, dramatically better output.
4. The delta calculation — the system reports exactly how much each axis improved, making the improvement concrete and falsifiable.

10. Vision: Where This Goes

Near-term: All Gyms Active

The system currently has 14 registered gyms across extraction, badge quality, fetchability, knowledge extraction, code cleanup, and more. The near-term goal is each gym actively running, each scoring its parent system's output, each producing measurable improvement deltas. Every gym produces reusable artifacts — skills, conventions, validators — that compound across the entire system.

Medium-term: Gyms as Jobs with Dashboard Integration

Each gym becomes a scheduled job that runs automatically — nightly, after code changes, or on-demand. Results flow into the system dashboard, creating a quality scoreboard where you can see at a glance which capabilities are improving and which are regressing. Regressions trigger alerts; improvements trigger celebrations.

The job integration also enables continuous calibration: the judge calibration gym runs whenever a new model is added to the evaluation pipeline, automatically screening it for discriminative power before it can influence scores.

North Star: The System That Improves While You Sleep

The north star is a system where you wake up and it has gotten measurably better overnight. Not because someone pushed a model update or tuned a prompt — but because the gym loop ran, found failures, encoded fixes, verified them, and deployed. The improvement is automatic, measured, and auditable.

Each gym is a small instance of this vision. The ASCII diagram gym took quality from 0.42 to 0.86 in a single iteration cycle. Scale that to 14 gyms, each running continuously, each producing durable artifacts that benefit the entire system — and the compounding begins. A skill created by one gym improves the input quality for another. Better extraction feeds better knowledge. Better knowledge feeds better principles. Better principles feed better everything.

Report generated March 2026. Data from the rivus gym system. ASCII diagram gym data from 10 challenges with programmatic + LLM hybrid scoring. Extraction gym data from 6 corpus sites. Judge calibration framework from lib/eval/calibration/. Source: static.localhost/present/gyms/report.html.