Accumulating Wisdom: A Self-Improving Knowledge System for AI Coding Agents

How an AI assistant extracts principles from its own work — about code quality, UX design, data processing, and architecture — and uses them to produce better solutions over time.

1. Context: Why Domain Expertise Is the Frontier

In the agentic future, AI agents will be remarkably competent and intelligent out of the box. But there will always be frontiers of specialized capability — from protein folding to reasoning about the impact of news on semiconductor supply chains — that require two things generic models lack:

1. Extensive domain knowledge — not just facts, but structured understanding of how a domain works: which variables matter, what the normal ranges are, how entities relate, what the failure modes look like.
2. Advanced reasoning expertise specific to that domain — the judgment calls, heuristics, and pattern recognition that domain experts have internalized over years. "This earnings call language pattern usually precedes a guidance revision." "This patent citation structure suggests defensive filing, not innovation."

Generic LLMsLarge Language Models — AI models trained on vast text data that can generate, analyze, and reason about text. Examples: Claude, GPT, Gemini. will be commoditized — everyone has access to the same frontier models. What remains scarce and valuable: curated, verified domain expertise that agents can leverage for specialized reasoning.

The agent that can analyze a 10-K like a forensic accountant, evaluate a founder like a seasoned VC, or assess supply chain risk like a procurement expert will outperform a generic agent with the same base model. The differentiator isn't reasoning capability — it's having better, fresher, more structured domain knowledge to reason about.

This report describes the learning system — one half of how we build that domain expertise layer. It handles the internal loop: extracting principles from the agent's own work, testing them, and feeding them back. Its companion, the skill acquisition system, handles the external loop: extracting domain knowledge from the web, verifying it through execution, and building a growing library of specialized capabilities. Together, they compound: the agent gets better at learning, and what it learns makes it better at its work.

2. The Problem: Capable But Not Getting Wiser

AI coding assistants like Claude Code are remarkably capable. But each session starts with limited memory from prior sessions. The agent doesn't accumulate wisdom: hard-won insights about what makes code clean, what makes layouts intuitive, what makes data pipelines robust, what architectural choices pay off.

Analysis of a single user's (Tim's) Claude Code sessions over ~7 weeks found:

The issue isn't raw capability — it's the absence of accumulated judgment. A senior engineer doesn't just know more facts; they've internalized principles about when complexity is warranted, when to verify assumptions, when a workaround signals a wrong abstraction. The agent lacks equally excellent mechanisms to build this kind of wisdom.

3. The Core Idea

What if the AI agent could review its own work — not just what went wrong, but what went right and why — abstract those insights into general principles about code quality, UX design, data handling, and architecture, test them, and carry them forward into every future session?

That's the learning system. It extracts wisdom across multiple dimensions:

The implementation is a closed loop:

The key insight is that principles, not fixes, are the unit of learning. A specific fix ("add --force flag") helps once. A principle ("verify assumptions at system boundaries") prevents entire classes of errors across all future sessions.

4. How It Works

3.1 Mining Failure→Success Pairs

Every Claude Code session produces a JSONLJSON Lines — a file where each line is one JSON event. The raw transcript of everything the agent did. transcript — every tool call, every result, every error. The mining step parses these transcripts looking for sequences where a tool call fails and a subsequent call succeeds.

Why 8 candidates instead of just the next success? After an error, AI agents typically investigate (Read, Grep, search) before fixing. Naively pairing "error → next success" was wrong 62% of the time — it paired the error with a diagnostic step, not the actual fix.

3.2 Multi-Model Repair Scoring

Given 8 candidate repairs, which one actually fixed the problem? This is a judgment call that requires understanding code context. The system uses multiple LLMLarge Language Model — an AI that can read and reason about text. Here, different LLMs serve as independent judges evaluating repair quality. judges:

Majority vote across 2+ models sets the verdict. Default pair: Gemini Flash + Grok Thinking (~$2 total) achieves 90%+ agreement with the expensive flagship. Across 997 pairs, 66% had unanimous agreement across all 6 tested models.

3.3 Principle Proposal: From Specific Failures to General Wisdom

This is where the magic happens. Given hundreds of verified failure→fix pairs, a three-stage LLM pipeline abstracts them into general principles:

3.4 Testing Principles in Sandbox

A principle that sounds wise might actually hurt in practice. How do you know? You test it. The sandbox replayRunning the agent in an isolated environment to test a principle. Compare results with vs. without the principle on the same tasks. system runs Claude Code inside DockerSoftware that creates isolated, disposable environments. Each test runs in a clean container so tests can't affect real work. containers at specific git commits, with and without each principle in its instructions:

# Run eval campaign: 20 prompts × 7 principles + baseline
python -m learning.session_review.sandbox_replay --eval default --parallel 6

The idea is simple but powerful: take a curated set of 20 coding tasks, run each task twice — once with the principle injected into the agent's instructions, once without (baseline). Measure everything: wall-clock time, tool calls, turns, and result quality (scored by LLM judges on a 0–100 scale).

This creates a controlled experiment for each principle. A principle that speeds things up but reduces quality is caught. A principle that helps on some tasks but hurts others gets a "guard clause"A qualification like "EXCEPT when..." that limits a principle to contexts where it actually helps, based on regression testing. — "apply EXCEPT when..."

The campaign runs 20 prompts × 7 principles + baseline = 160 runs, each in an isolated Docker container. Multiple LLM judges score quality independently. Only effects larger than 25% are considered statistically meaningful (wall-clock variance is ~20%).

3.5 The Knowledge Store

Everything lives in a SQLiteA lightweight database in a single file — no server needed. Fast, reliable, and perfect for a single-user knowledge system. database (learning.db) with a clear data model:

Principles are the primary output. They're materializedPre-computed and stored in a different format. The database is the truth; the markdown files are regenerated from it so the agent can read simple text. as markdown files that get loaded into every future Claude Code session:

## No Silent Failures

If something fails, make it visible. Never swallow exceptions,
return empty data on error, or let a process silently produce
no output. The cost of a noisy failure is one interruption.
The cost of a silent failure is hours of debugging the wrong thing.

<!-- evidence: 16 instances, 0 applications, as of 2026-02-14 -->

3.6 Hierarchy of Knowledge Types

Not all learnings are equal. The system distinguishes five levels:

The system's goal is to promote observations upward: raw data becomes patterns, patterns become principles. Each level up means wider applicability and more leverage.

5. Motivating Examples

Example 1: "No Silent Failures" (16 instances)

This is the system's most evidence-backed principle. It emerged from mining sessions where:

• A function returned an empty list on error instead of raising an exception. Downstream code processed the empty list silently, producing a report with zero data. The user didn't notice for hours.
• A web fetch returned a cached error page. The cache layer didn't check content validity. The analysis proceeded on garbage data.
• A batch job skipped failed items without logging. 40% of items had failed. The "successful" run produced 60% of expected output, and nobody noticed until the data was used in a report.

From 16 such instances, the system abstracted: "If something fails, make it visible." This principle now prevents the agent from writing code with silent exception handling, empty fallbacks, or unchecked return values.

Example 2: "Validate Before Bulk, Not After" (6 instances)

Several sessions ran batch processing on hundreds of items, only to discover a systematic problem late in the run:

The abstracted principle: "Before running 300+ items or 2+ hours of processing, run a representative sample (5–10 items) and verify the output has what you need."

Example 3: "Workarounds Piling Up = Wrong Abstraction" (7 instances)

When building UIs with Gradio, the agent kept adding CSS hacks to fix layout issues — a MutationObserver here, a custom JavaScript override there, another !important rule. Seven separate sessions exhibited this pattern before the system identified the meta-lesson:

When you're adding CSS hacks, MutationObservers, or custom JS to fix layout issues — STOP. Multiple workarounds signal you're fighting the framework instead of using it correctly.

This principle now triggers early in sessions, before the agent starts down the workaround path, saving hours of accumulated hack-upon-hack debugging.

Example 4: UX Principles — "Information Prominence" (4 instances)

This principle didn't emerge from a bug — it emerged from observing what makes interfaces good. Across several UI-building sessions, the system noticed a recurring pattern: labels, headers, and scaffolding were dominating the visual hierarchy, while the actual content — the answers the user came for — receded into the background.

The abstracted principle: "Content outweighs chrome. The answers, not the scaffolding, should stand out." This now guides every UI the agent builds — not avoiding a mistake, but making a better design choice from the start.

Example 5: Architecture — "Extend, Don't Invent" (4 instances)

Four separate sessions built new tools from scratch when a small extension to an existing system would have sufficed. The system abstracted:

When you need a new capability, first ask: can an existing system already do this with a small extension? A new extension point on a working system beats a new app every time.

This principle produces less code and better architecture — fewer moving parts, less duplication, a more cohesive codebase.

Example 6: Knowledge-Accumulating Search

A novel pattern that emerged from the system's own learning process: when searching for information across many similar entities (e.g., finding investor relations pages for 500 companies), structure the work so each search enriches a shared knowledge base. After discovering that 60% of S&P 500 companies use the same webcast provider, you can predict the pattern for new companies without searching.

This principle didn't come from a failure — it came from observing that stateless search (running the same strategy for every entity) was wasteful. The system generalized the pattern into a reusable principle with concrete implementation guidance.

6. Measuring Usefulness

How do you know if a learning system is actually making the AI agent better? This is the hardest question, and the system addresses it at multiple levels:

5.1 Direct Metrics

5.2 Process Metrics

5.3 The ROI Framework

The system includes a rigorous cost-benefit analysis framework for fixes (documented in HOWTO.md):

prevention_cost = cost_per_call × total_calls
failure_cost    = cost_per_failure × failure_count
ROI             = failure_cost - prevention_cost

This prevents the common trap of "fix everything" — some errors are so rare or cheap that preventing them costs more than tolerating them. The system categorizes fixes into three tiers:

1. Zero-cost behavioral changes (always worth it): Reorder operations, use correct tools
2. Per-failure hooks (worth it when fail rate > ~20%): Pre-tool hooks that reject bad calls
3. Per-call preflight checks (almost never worth it): Adding validation to every call to catch the ~9% that fail

5.4 What "Working" Looks Like (from the Vision Doc)

The system defines intelligence levels for measuring progress:

Current system: L0–L1 achieved, working toward L3. L4–L5 represent the long-term vision.

7. Prior Work & How This Differs

Retrieval-Augmented Generation (RAG)A technique where an AI retrieves relevant documents before answering, grounding its response in specific sources rather than just its training data.

RAG systems retrieve relevant documents to augment LLM prompts. The learning system shares this retrieval mechanism (FTS5, embeddingsText converted to numbers (vectors) where similar meanings produce similar numbers. "Verify before acting" and "check assumptions first" have similar embeddings despite different words.) but differs fundamentally: RAG retrieves existing documents; the learning system creates new knowledge by abstracting patterns from operational data. The knowledge doesn't exist in any document — it's synthesized from hundreds of error→fix sequences.

Reinforcement Learning from Human Feedback (RLHF)A training method that adjusts a model's internal parameters using human preferences. Expensive and opaque. The learning system changes instructions instead — cheap and transparent.

RLHF trains model weights using human preference signals. The learning system operates outside the model — it doesn't change model weights. Instead, it changes the context (instructions, principles) that the model receives at the start of each session. This is faster to iterate (no retraining), more interpretable (principles are human-readable text), and cheaper (SQLite + LLM calls vs. GPU training runs).

Experience Replay in RL

Reinforcement learning agents store and replay experiences from a buffer. The learning system's failure mining is analogous to experience replay, but with a crucial difference: it abstracts before replaying. Instead of replaying raw (state, action, reward) tuples, it extracts principles that generalize across episodes. One principle prevents dozens of future failures, whereas experience replay helps with the specific situations in the buffer.

Post-Incident Review / Blameless Postmortems

The SRE tradition of learning from incidents is the closest analog. The learning system automates this process: instead of a human writing "we should have validated the API response," the system mines transcripts, identifies the pattern, proposes the principle, tests it, and injects it into future sessions. It's a continuous, automated postmortem running after every session.

Self-Play and Curriculum Learning

The sandbox replay system (testing principles in Docker) is a form of curriculum learning — the agent is evaluated on a curated set of tasks that increase in difficulty. The "gyms" (badge quality, fetchability, knowledge extraction) extend this to continuous self-improvement loops on specific capabilities.

What's Genuinely Novel

Several aspects of this system don't have close precedents:

1. Multi-model consensus for repair identification — using 2+ LLM judges to determine which of 8 candidate actions was the actual fix, with measured agreement rates and cost optimization.
2. The "5-case test" for principle quality — a principle must apply to 5+ distinct situations or it's demoted to a convention/howto. This prevents overfitting to specific errors.
3. Guard clause generation from regression testing — principles that hurt performance on some tasks get automatically qualified with "EXCEPT when..." clauses.
4. The evidence chain — every principle traces back to specific session transcripts, through specific failure pairs, with specific LLM judge scores. Full provenance from raw data to deployed instruction.

8. Current Results

Top Principles by Evidence

Source Distribution

Of 1,126 instances: 95% (1,068) come from automated session mining, 5% (58) from manual input. The system is overwhelmingly self-feeding — humans provide seed knowledge, the mining pipeline does the rest.

Multi-Model Repair Scoring

997 failure pairs scored across 6 models. Key findings:

• 66% unanimous agreement across all models
• Gemini Flash ($1.72/1K pairs) matches flagship accuracy at 92%
• Grok Thinking ($0.25/1K pairs) achieves 88% — cheapest viable judge
• The "investigation vs. repair" distinction is the primary source of model disagreement

9. Vision: Where This Goes

Near-term: Closing the Loop

The biggest gap is application tracking. The system knows what it learned but doesn't yet systematically measure whether those learnings are being used and helping. Closing this loop — tracking when each principle fires, whether the outcome was better — is the difference between a knowledge base and a learning system.

Medium-term: Cross-Domain Pattern Recognition

The highest-value learning isn't domain-specific ("use word boundaries for ticker matching"). It's recognizing that a problem in one domain has already been solved in another.

Long-term: The North Star

A new piece of information on a well-covered topic is boring to the system — low surprise, high explainability, rich connections. That's understanding.

The vision is a system that doesn't just store facts but understands its domain: it can explain why things work, predict what will go wrong, and synthesize novel solutions by combining knowledge from different areas. The intelligence levels (L0–L5) provide a roadmap:

Self-Improving Gyms

Beyond session mining, the system includes "gyms" — structured generate→evaluate→learn loops for specific capabilities:

Each gym is a miniature learning system for a specific capability, using the same Raw + Active architecture: raw episodes are immutable, the active model is swappable, and improvement is measured by replaying the corpus through new versions.

Appendix: System Architecture

Glossary

LLM (Large Language Model)

An AI model trained on vast text data that can generate, analyze, and reason about text. Examples: Claude, GPT, Gemini. In this system, LLMs serve as both the agent being improved and the judges evaluating quality.

JSONL (JSON Lines)

A file format where each line is a complete JSON object. Claude Code logs every session as a JSONL file — each line records one event (a tool call, a response, an error). This is the raw data the learning system mines.

FTS5 (Full-Text Search 5)

SQLite's built-in full-text search engine. Instead of scanning every row looking for a word, FTS5 builds an inverted index (like a book's index) that maps words to the rows containing them. Searches that would take seconds on raw text take milliseconds with FTS5. The learning system uses it to quickly find relevant principles and instances by keyword.

Embeddings

A way to represent text as a list of numbers (a "vector") such that similar meanings produce similar numbers. "Verify before acting" and "Check assumptions first" have very different words but similar embeddings, because they mean similar things. The learning system uses embeddings for semantic search — finding principles that are conceptually relevant, not just keyword matches.

Cosine Similarity

A mathematical measure of how similar two embeddings are, scored from -1 (opposite) to 1 (identical). When you search for "error handling," the system computes the cosine similarity between your query's embedding and every stored principle's embedding, returning the closest matches. It measures the angle between two vectors — small angle means similar meaning.

SQLite

A lightweight database that lives in a single file (no server needed). The learning system stores all its data in learning.db, a SQLite file you can query directly. It's fast, reliable, and perfect for a single-user knowledge system.

Materialized View

A pre-computed copy of data stored in a different format for easy access. The principle .md files are materialized views of the database — the database is the source of truth, and the markdown files are regenerated from it whenever principles change. This way, the agent can read simple text files instead of querying a database.

Sandbox Replay

Running the AI agent in an isolated environment (Docker container) to test whether a principle helps or hurts. The agent gets a coding task, runs it with and without a specific principle in its instructions, and the results are compared. "Sandbox" means the test can't affect real files or systems.

Docker

Software that creates isolated, disposable environments ("containers") for running programs. Each sandbox replay runs in its own Docker container with a clean copy of the codebase, so tests don't interfere with each other or with real work.

RAG (Retrieval-Augmented Generation)

A technique where an LLM retrieves relevant documents before generating a response, grounding its answer in specific sources. The learning system uses RAG-like retrieval (finding relevant principles) but goes further: it creates the knowledge it retrieves, rather than pulling from pre-existing documents.

RLHF (Reinforcement Learning from Human Feedback)

A training technique where human preferences are used to adjust an AI model's internal parameters (weights). The learning system differs: instead of changing the model's weights (expensive, opaque), it changes the text instructions the model receives (cheap, interpretable, instantly reversible).

Ablation Study

Testing what happens when you remove a component. "Does the system work without this principle?" If removing it makes things worse, the principle is earning its keep. If nothing changes, it's dead weight.

Two-Tower Model

A neural network architecture with two separate networks (towers) that each process one type of input and project it into a shared space. Used by Google Search (query + document), Spotify (user + song), and planned for principle retrieval (situation + principle). The advantage: add a new principle and it's immediately retrievable without retraining.

Guard Clause

A qualification added to a principle after testing reveals edge cases where it hurts. Example: "Always use parallel tool calls — except for leaf tasks under 5 seconds." The principle is still valid in general, but the guard clause prevents it from being applied where it does more harm than good.