Vario: Turning One LLM Call Into a Rigorous Process

A streaming pipeline engine that orchestrates multiple AI models — generating, scoring, debating, and refining — so that complex questions get the systematic treatment they deserve.

1. The Problem: One Roll of the Dice

You paste a complex question into an AI chat. Maybe it's "Should we acquire this company?" or "What are the failure modes of this architecture?" or "Evaluate this founder's track record." You get back a single answer. It sounds confident. It might be excellent. It might be mediocre. You have no way to tell.

This is the fundamental problem with single-shot LLM usage: you're rolling the dice once and hoping for the best.

Human experts don't work this way. A good analyst writes a draft, gets a second opinion, argues with a colleague, revises their reasoning, and only then shares their conclusion. A good hiring committee doesn't rely on one interviewer. A good investment memo incorporates multiple perspectives and a devil's advocate review.

Yet the standard AI workflow is: prompt → single response → hope for the best.

What if we could give AI the same systematic rigor that makes human expert processes reliable?

2. The Core Idea

Vario turns a single LLM call into a streaming pipeline of generation, scoring, debate, and refinement — so you get the best answer multiple models can produce, not just whatever one model happened to say first.

The analogy: Vario is to a single LLM call what a committee is to a single opinion. It doesn't just ask one model once — it asks several models, scores their answers, has them debate the hard parts, iteratively refines the best candidates, and gives you a ranked result with provenance for every judgment.

Three things make this non-obvious:

1. Streaming, not batching. Traditional multi-model approaches wait for all responses before doing anything. Vario uses async generatorsPython's async generator pattern: functions that yield values one at a time as they become available, enabling pipeline processing without waiting for all data. — fast models yield results immediately while slow models are still thinking. The scoring phase starts the instant the first candidate arrives.
2. One universal data type. Every piece of data flowing through the pipeline is an ItemVario's universal datum: content + accumulated properties + provenance history. Items accumulate metadata as they flow through pipeline stages. — content plus accumulated properties. An Item that enters produce empty gains a model and cost. When it passes through score, it gains a score and reason. Through revise, its content improves. No schema enforcement, no type hierarchies — just props accumulating as data flows.
3. Recipes compose. Pipelines are defined as YAML recipesYAML-defined pipeline configurations specifying which ops to run, in what order, with what parameters. Can be named, or auto-designed from natural language. — reusable, shareable, and composable. But you can also describe what you want in natural language ("debate across 3 models") and Vario will design the recipe for you.

3. How It Works

The Pipeline Model

Vario pipelines are chains of opsStreaming operations that transform Items. Signature: async generator in, async generator out. Nine built-in ops cover generation, scoring, revision, reduction, and more. — streaming operations that transform Items. Each op is an async generator: it consumes Items from upstream and yields Items downstream.

The key insight: ops don't wait. produce fires 5 parallel LLM calls and yields each result the instant it completes. score starts judging the first candidate while produce is still generating the rest. The pipeline is pull-driven — the collector at the end drives execution through the chain.

The Nine Ops

Execution Trace

Every pipeline run produces a structured RunLogA structured narrative of what happened during execution: stages, timing, cost, token counts, and outcome summary. Persisted to SQLite for analysis. — a record of what happened at each stage: how many items entered and exited, how many tokens were used, how much it cost, and how long it took. This makes every run auditable and every cost predictable.

Recipe: best_of_n | Problem: "Evaluate this acquisition target"
  Stage 0: produce  → 0 in, 5 out  | 2,845 tokens | $0.021 | 3.2s
  Stage 1: score    → 5 in, 5 out  | 1,230 tokens | $0.004 | 1.1s
  Stage 2: reduce   → 5 in, 1 out  | 0 tokens     | $0.000 | 0.0s
Outcome: 1 item, best score 91 | Total: $0.025 | 4.3s

Item Provenance

Every Item carries a history — a list of what happened to it at each stage:

item.history = [
  {"stage_id": "stage_0.produce", "added": {"model": "sonnet", "cost": 0.003}},
  {"stage_id": "stage_1.score",   "added": {"score": 91, "reason": "thorough risk analysis"}},
]

You can always trace why an Item has a particular score, which model generated it, and how much each stage cost. No black boxes.

4. Motivating Examples

Example 1: Best-of-N Selection

The simplest pattern: ask multiple times, score each answer, keep the best.

vario run "What are the risks of acquiring Acme Corp?" -r best_of_n

Cost: ~$0.03. Time: ~5 seconds. Quality lift: The winning answer consistently outperforms any single call.

Example 2: Multi-Model Debate

For nuanced questions where different models have different strengths.

vario run "Is this founder's pivot history a red flag?" \
  -i founder_timeline.md -r model_debate

Example 3: Iterative Refinement

For high-stakes output where good enough isn't good enough.

Vario's repeat op handles the loop automatically with multiple stop conditions:

• Score threshold — stop when quality is high enough
• Minimum improvement — stop when revisions aren't helping
• Drift window — stop when scores oscillate instead of improving
• Budget cap — stop when cost limit is reached

vario run "Write an investment memo for Acme Corp" \
  -i acme_data.md -r refine_until_converged --budget 0.50

Key insight: Human experts naturally do this — write a draft, get feedback, revise, repeat. Vario automates the same loop with explicit quality criteria instead of gut feeling.

Example 4: Corpus Processing

When you need to apply the same analysis to many documents.

steps:
  - op: source          # Load 50 documents from directory
    params:
      directory: corpus/
  - op: fan_out         # Each doc × 3 models = 150 items
    params:
      models: [sonnet, haiku, grok-fast]
  - op: task            # Extract claims from each
    params:
      handler: lib.extract.extract_claims
  - op: evaluate        # Compare against reference claims
    params:
      metrics: [precision, recall, f1]

Example 5: Rapid Multi-Perspective

When you just need diverse viewpoints, fast.

vario run "What's the most overlooked risk in semiconductor supply chains?" -r ask

Fans out to 4+ top-tier models. No scoring, no reduction — just raw perspectives. Takes ~5 seconds, costs ~$0.04. Useful as input to your own thinking, not as a final answer.

5. Comparison with Prior Work

6. Measuring Usefulness

Direct Quality Metrics

Score distribution: In a best-of-5 run, the winning candidate typically scores 15-25 points higher (on a 0-100 rubric) than the median candidate. This is consistent across problem types — the spread exists because models make different tradeoffs on each generation.

Convergence in refinement: The refine_until_converged recipe typically converges in 3-4 rounds. Round 1 catches the biggest gaps (10-15 point improvement). Round 2 addresses secondary issues (5-8 points). Rounds 3+ yield diminishing returns (<2 points), which triggers the min_improvement stop condition.

Process Metrics

Ablation: What Happens Without Vario?

Honest Limitations

• Not free — best_of_n costs 5× a single call. Debate and refinement cost more. The question is whether the quality improvement justifies the cost for your use case.
• LLM-as-judge is imperfect — The scoring model has its own biases. Vario makes these biases auditable (you can read every scoring reason) but doesn't eliminate them.
• Diminishing returns — For simple factual questions, best-of-5 may not beat best-of-1. The process shines on complex, judgment-heavy problems where there's genuine variation in answer quality.
• Latency — Streaming helps, but multi-round refinement takes time. For latency-sensitive applications, the fast recipes (ask_fast, best_of_n) are better suited than iterative refinement.

7. Demo

CLI Walkthrough

# Fan out to top models, see all perspectives
$ vario run "What's the biggest risk in AI agent frameworks?" -r ask

# Items stream in as models respond:
# [sonnet]  → "Security boundaries between agent and tools..."
# [gemini]  → "State management across long-running tasks..."
# [grok]    → "Catastrophic action authorization..."
# [opus]    → "Eval-production gap in agent behavior..."

# Score 5 candidates against a rubric, get the best
$ vario run "Summarize the key risks" -i quarterly_report.pdf -r best_of_n

# Result:
# Score: 88 | Model: sonnet | Cost: $0.03
# "Three primary risk categories emerge from Q4..."

# Describe what you want, Vario designs the pipeline
$ vario design "generate 3 drafts, have them critique each other, \
  revise based on critiques, pick the best"

# Output: YAML recipe with produce → fan_out → score → revise → reduce

# Refine until quality converges or budget exhausted
$ vario run "Write an investment memo for Acme Corp" \
  -i acme_10k.pdf -r refine_until_converged --budget 0.50

# Round 1: score 72 → "misses competitive landscape"
# Round 2: score 83 → "stronger, but regulatory risk understated"
# Round 3: score 89 → "comprehensive" (converged, Δ < 1.0)

from vario.run.runner import execute, load_recipe

recipe = load_recipe("best_of_n")
result = await execute(recipe, "Evaluate this startup", limits={"usd": 0.10})

best = result.items[0]
print(f"Score: {best.score} | Model: {best.model}")
print(f"Cost: ${result.total_cost:.3f} | Time: {result.duration_ms:.0f}ms")
print(best.content)

8. Vision: Where This Goes

Near-Term (3–6 months)

• Python DSL maturity — Recipe-as-code: produce("...") | score(rubric) | top(1). Enables IDE autocomplete, type checking, and Python-native composition.
• Named prompt ops — Library of pre-built (op, prompt) pairs: brainstorm_angles(), steelman_counterargs(), extract_risks(). The prompt engineering gets done once and shared.
• Recipe-as-op composability — Nest recipes inside recipes. A "deep analysis" recipe could call "best_of_n" for each section, then "summarize" to combine them.

Medium-Term (6–18 months)

• Steer op — Observe pipeline state mid-run and dynamically choose what to do next. If all candidates scored >90, skip refinement. If scores are bimodal, split into two debate tracks. The pipeline becomes adaptive, not just sequential.
• LLM-driven continuation — After a pipeline finishes, an LLM reviews the result and decides whether more work is needed and what kind. The system reflects on its own output.
• Cross-pipeline learning — Track which recipes work best for which problem types. Over time, auto-select the optimal recipe based on the question's characteristics.

North Star

Why this compounds: Every recipe that works well becomes a reusable pattern. Every scored run generates data about which models and processes work best for which problems. The system gets better at designing processes as it accumulates execution history. The human's job shifts from crafting prompts to specifying quality criteria — a much more natural way to direct AI work.

Appendix: Architecture Reference

System Diagram

Recipe Library

File Map

Glossary

Item

Vario's universal datum. Content (the actual text) plus accumulated properties (model, score, cost, etc.) plus a provenance history tracking what was added at each pipeline stage.

Op (Operation)

A streaming function that transforms Items. Type signature: AsyncIterator[Item] → AsyncIterator[Item]. Ops compose via pipe() or YAML recipes.

Recipe

A YAML-defined pipeline configuration specifying which ops to run, in what order, with what parameters. Can be hand-authored, loaded from the built-in library, or auto-designed from natural language.

RunLog

A structured execution narrative recording what happened during a pipeline run: stages, timing, token counts, costs, and outcome summary. Persisted to ~/.vario/runs.db.

Async Generator

Python's async def f() -> AsyncIterator pattern. Functions that yield values one at a time as they become available, enabling pipeline processing without waiting for all data to be ready.

Heap (Anytime Priority Queue)

A sorted collection of Items where the best item is always accessible, even while the pipeline is still running. Enables real-time UI updates and early termination.

Convergence

The point where additional refinement rounds stop improving quality. Detected via minimum improvement thresholds, drift windows, or score plateaus.

Provenance

The complete audit trail of an Item: which model generated it, what score it received and why, how many rounds of revision it went through, and what each stage cost.