Assessor Calibration Gym

First run — 2026-03-09 · Model: gemini-flash · 15 corpus entries (8 real, 7 synthetic)

1. The Problem

The person_intel job scores people on 4 pre-chosen dimensions using LLM prompts. The gym doesn't question whether these are the right dimensions — it checks whether the scoring prompts produce well-calibrated numbers for the dimensions we've committed to:

These prompts were written once and never tested against known examples. Without calibration data, we had no idea whether "Jensen Huang = 3 on academic" or "Jensen Huang = 7 on academic." The scoring rubrics existed but the LLM's interpretation of them was unchecked.

2. What Was Built

Ground Truth Corpus

15 people — 8 real, 7 synthetic — spanning the full 0–10 range. Real people (Hinton, LeCun, Karpathy, Jensen Huang, etc.) have accurate enrichment data drawn from public sources. Synthetic people (fictional names, fabricated-but-plausible profiles) fill the lower tiers where no famous person would naturally land. Each entry has hand-built enrichmentData gathered from 12+ APIs: Apollo, Semantic Scholar, USPTO Patents, SEC EDGAR, Wikipedia, arXiv, OpenAlex, DBLP, PubMed, etc. and researchLLM web-search-grounded evidence: career history, exits, network signals, technical contributions, education. data, plus expected score ranges per dimension:

Real people have accurate data (Hinton's h-index of 168, LeCun's 25 patents, Koller's Coursera IPO at $5.9B). Synthetic people have fabricated but plausible profiles.

Gym Architecture

The gym directly calls the assessor functions from jobs/handlers/person_intel.py with corpus data, bypassing the jobs framework, web search, and API enrichment. This means each iteration exercises only the LLM scoring prompt — nothing else — at ~$0.001 per call.

# Run all assessors + core scorer on the corpus
python -m intel.people.gym run

# Run just the academic assessor
python -m intel.people.gym run --assessor academic

# Show cached calibration report
python -m intel.people.gym report

# Test with multiple models
python -m intel.people.gym cross-model --models gemini-flash,haiku,sonnet

# Capture real pipeline output as a new corpus entry
python -m intel.people.gym harvest yann-lecun --expected-academic 9-10

Calibration Metrics

Four metrics per dimension, computed from corpus results:

3. Run 1: The Hallucination Bug

First run of the academic assessor (before any prompt changes):

5 violations out of 15 entries. Two were catastrophic:

The prompt said "Score this person's academic prowess based on these signals" but never said "only" based on these signals. The LLM treated the signals as hints and supplemented with its own knowledge — which happened to match a different person with the same name.

4. The Fix

Added two constraints to the academic assessor prompt in jobs/handlers/person_intel.py:463:

# Before:
Score this person's academic prowess from 0-10 based on these signals:

# After:
Score this person's academic prowess from 0-10 based ONLY on the signals below.

IMPORTANT: Use ONLY the data provided. Do NOT use your own knowledge
about this person or anyone with a similar name. If the signals show
limited academic evidence, the score should be low regardless of what
you may know about people with this name.

Also tightened the bottom of the rubric from "0-2: No academic background" to "0-2: No academic background (no PhD, no publications, no patents, just bachelor's degree)" to anchor the low end more concretely.

5. Run 2: After the Fix

Per-person detail after the fix:

3 remaining violations are minor (1–2 points above expected max). The catastrophic name-hallucination errors are gone.

6. Core Scorer Results

Also ran the core 3-dimension scorer (_stage_score) on the same corpus. This scores prior_success, network_quality, and technical_depth from research data:

The pattern is consistent: ordering is excellent across all dimensions (τ ≥ 0.925 everywhere). The LLM gets relative ranking right. The problem is absolute calibration — it's too generous, especially on technical_depth where 73% of scores cluster in a 2-point band near the top of the scale.

7. Cost & Iteration Speed

This matches the research→analyze split philosophy: the corpus provides cached "research" data, and the gym only exercises the cheap scoring step. Prompt iteration becomes a tight loop: edit prompt → run gym → check report → repeat. No web search, no API calls, no job framework overhead.

8. What's Next

Glossary

Enrichment

Structured data gathered from 12+ free APIs: Apollo (professional profile), Semantic Scholar (publications, h-index), USPTO (patents), SEC EDGAR (filings), Wikipedia, arXiv, OpenAlex, DBLP, PubMed, Wikidata. Saved as enrichment.json per person.

Research

LLM web-search-grounded evidence gathering. An LLM searches the web for a person and returns structured JSON: career history, exits, network signals, technical contributions, education. Costs ~$0.05/person due to web search. Cached in the jobs results table.

Range Accuracy

Percentage of corpus entries where the LLM's actual score falls within the expected [min, max] range set by ground truth. Example: if expected is [9, 10] and actual is 10, that's in-range. If actual is 8, that's a violation.

Kendall τ (tau)

Rank correlation coefficient from -1 to +1. Measures whether the LLM ranks people in the same relative order as ground truth, regardless of absolute scores. τ = 1.0 means perfect agreement on ordering. More important than range accuracy — if the ordering is right, absolute calibration can be fixed by adjusting the rubric.