# LLM Latency Benchmark Report

**Generated**: 2026-01-27 05:42:54

## Summary

| Model | TTFT (median) | Total (median) | Tokens/sec | Success |
|-------|---------------|----------------|------------|---------|
| anthropic/claude-sonnet-4-5-20250929 | 1232ms | 8267ms | 42.5 | 100% |
| anthropic/claude-opus-4-5-20251101 | 1592ms | 7588ms | 50.0 | 100% |
| gemini-pro | N/A | N/A | N/A | 0% |

## Detailed Results

### anthropic/claude-sonnet-4-5-20250929

**TTFT (Time to First Token)**
- Min: 1124ms
- Max: 1355ms
- Mean: 1241ms
- Median: 1232ms
- Stdev: 87ms

**Total Response Time**
- Min: 8120ms
- Max: 8932ms
- Mean: 8415ms
- Median: 8267ms
- Stdev: 331ms

**Individual Runs**

- Run 1: TTFT=1203ms, Total=8267ms, Tokens=300, 42.5 tok/s
- Run 2: TTFT=1290ms, Total=8932ms, Tokens=300, 39.3 tok/s
- Run 3: TTFT=1355ms, Total=8205ms, Tokens=300, 43.8 tok/s
- Run 4: TTFT=1124ms, Total=8120ms, Tokens=300, 42.9 tok/s
- Run 5: TTFT=1232ms, Total=8550ms, Tokens=300, 41.0 tok/s

### anthropic/claude-opus-4-5-20251101

**TTFT (Time to First Token)**
- Min: 1480ms
- Max: 1824ms
- Mean: 1634ms
- Median: 1592ms
- Stdev: 128ms

**Total Response Time**
- Min: 7137ms
- Max: 7851ms
- Mean: 7539ms
- Median: 7588ms
- Stdev: 292ms

**Individual Runs**

- Run 1: TTFT=1683ms, Total=7757ms, Tokens=300, 49.4 tok/s
- Run 2: TTFT=1824ms, Total=7851ms, Tokens=300, 49.8 tok/s
- Run 3: TTFT=1592ms, Total=7588ms, Tokens=300, 50.0 tok/s
- Run 4: TTFT=1480ms, Total=7137ms, Tokens=300, 53.0 tok/s
- Run 5: TTFT=1591ms, Total=7360ms, Tokens=300, 52.0 tok/s

### gemini-pro

**All runs failed**

**Individual Runs**

- Run 1: FAILED - litellm.BadRequestError: litellm.BadRequestError: vertex_ai_betaException - b'{\n  "error": {\n    "code": 403,\n    "message": "Vertex AI API has not been used in project quantjoy before or it is dis
- Run 2: FAILED - litellm.BadRequestError: litellm.BadRequestError: vertex_ai_betaException - b'{\n  "error": {\n    "code": 403,\n    "message": "Vertex AI API has not been used in project quantjoy before or it is dis
- Run 3: FAILED - litellm.BadRequestError: litellm.BadRequestError: vertex_ai_betaException - b'{\n  "error": {\n    "code": 403,\n    "message": "Vertex AI API has not been used in project quantjoy before or it is dis
- Run 4: FAILED - litellm.BadRequestError: litellm.BadRequestError: vertex_ai_betaException - b'{\n  "error": {\n    "code": 403,\n    "message": "Vertex AI API has not been used in project quantjoy before or it is dis
- Run 5: FAILED - litellm.BadRequestError: litellm.BadRequestError: vertex_ai_betaException - b'{\n  "error": {\n    "code": 403,\n    "message": "Vertex AI API has not been used in project quantjoy before or it is dis

## Configuration

- **Prompt**: "Explain how a CPU cache works in 3 paragraphs."
- **Max Tokens**: 300
- **Timeout**: 60s