# Kodit Benchmark: SWE-bench Implementation
## Overview
This document describes the SWE-bench benchmark implementation for evaluating Kodit's code retrieval capabilities. The benchmark uses real-world GitHub issues from the [SWE-bench](https://www.swebench.com/) dataset to measure how much Kodit's retrieval improves LLM patch generation.
---
## 1. Why SWE-bench?
SWE-bench tests repository-level issue resolution—a task where retrieval provides significant value:
| Kodit Capability | SWE-bench Requirement | Alignment |
|------------------|----------------------|-----------|
| Index Git repositories | Real GitHub repos at specific commits | ✅ Perfect |
| Hybrid search (BM25 + semantic) | Find relevant code for bug fixing | ✅ Perfect |
| AST-based snippet extraction | Locate functions/classes to modify | ✅ Perfect |
| Filter by repository | Each task targets a specific repo | ✅ Perfect |
**Why SWE-bench over RepoEval?**
| Feature | SWE-bench | RepoEval |
|---------|-----------|----------|
| Exact commit hashes | ✅ `base_commit` field | ❌ Snapshots only |
| Evaluation method | ✅ Real test execution | ⚠️ Token similarity |
| Task complexity | Real bug fixes | Function completion |
| Retrieval impact | High (large repos) | Medium |
From the [SWE-bench leaderboard](https://www.swebench.com/):
- RAG-based approaches (BM25 retrieval + LLM) achieve **4-7% on Lite**
- Agentless-Lite with embedding retrieval achieves **32% on Lite**
- This demonstrates significant headroom for better retrieval
---
## 2. Dataset
### 2.1 Data Source
The benchmark uses the official SWE-bench datasets from Hugging Face:
| Dataset | Size | Use Case |
|---------|------|----------|
| `princeton-nlp/SWE-bench_Lite` | 300 instances | Primary benchmark |
| `princeton-nlp/SWE-bench_Verified` | 500 instances | Extended benchmark |
### 2.2 Repositories
SWE-bench Lite covers 12 popular Python repositories:
| Repository | Instances | Description |
|------------|-----------|-------------|
| django/django | 114 | Web framework |
| sympy/sympy | 77 | Symbolic mathematics |
| matplotlib/matplotlib | 23 | Plotting library |
| scikit-learn/scikit-learn | 23 | Machine learning |
| pytest-dev/pytest | 17 | Testing framework |
| sphinx-doc/sphinx | 16 | Documentation generator |
| astropy/astropy | 6 | Astronomy library |
| psf/requests | 6 | HTTP library |
| pylint-dev/pylint | 6 | Code linter |
| pydata/xarray | 5 | N-D arrays |
| mwaskom/seaborn | 4 | Statistical visualization |
| pallets/flask | 3 | Web microframework |
### 2.3 Instance Format
Each instance contains:
```python
{
"instance_id": "django__django-11049", # Unique identifier
"repo": "django/django", # GitHub repository
"base_commit": "17455e924e24...", # Exact commit to checkout
"problem_statement": "...", # Issue description (natural language)
"hints_text": "...", # Optional hints
"patch": "diff --git a/...", # Ground truth fix
"test_patch": "diff --git a/...", # Test additions
"FAIL_TO_PASS": ["test_invalid_string..."], # Tests that should pass after fix
"PASS_TO_PASS": ["test_other..."], # Tests that should remain passing
"version": "3.0", # Library version
"environment_setup_commit": "...", # Commit for environment setup
}
```
### 2.4 Example Task
```
Instance: django__django-11049
Commit: 17455e924e243e7a55e8a38f45966d8cbb27c273
Problem Statement:
Correct expected format in invalid DurationField error message.
The current error message says "[DD] [HH:[MM:]]ss[.uuuuuu]" but should
be "[DD] [[HH:]MM:]ss[.uuuuuu]" because seconds are mandatory.
Expected Patch:
diff --git a/django/db/models/fields/__init__.py
- "[DD] [HH:[MM:]]ss[.uuuuuu] format.")
+ "[DD] [[HH:]MM:]ss[.uuuuuu] format.")
Tests to Fix:
["test_invalid_string (model_fields.test_durationfield.TestValidation)"]
```
---
## 3. Experimental Design
### 3.1 Conditions
| Condition | Description |
|-----------|-------------|
| **Baseline** | LLM generates patch with only the problem statement |
| **BM25** | LLM generates patch with BM25-retrieved context (SWE-bench baseline) |
| **Kodit** | LLM generates patch with Kodit-retrieved context |
| **Oracle** | LLM generates patch with gold file context (upper bound) |
### 3.2 Metrics
**Primary Metric**:
- **Resolve Rate**: Percentage of instances where generated patch makes `FAIL_TO_PASS` tests pass
- **Resolve Rate Delta**: `Resolve(Kodit) - Resolve(BM25)` — the improvement over baseline RAG
**Secondary Metrics**:
- **Retrieval Recall@k**: Fraction of modified files found in top-k results
- **Context Utilization**: How often retrieved context appears in generated patches
### 3.3 Evaluation
Evaluation uses the official SWE-bench harness with Docker containers:
1. Apply generated patch to repository at `base_commit`
2. Run `FAIL_TO_PASS` tests in isolated environment
3. Verify `PASS_TO_PASS` tests still pass (no regressions)
4. Instance is "resolved" only if all conditions met
#### Running the SWE-bench Harness
Install and run the official evaluation harness:
```bash
# Install SWE-bench
pip install swebench
# Run evaluation (requires Docker with ~100GB disk space)
python -m swebench.harness.run_evaluation \
--dataset_name princeton-nlp/SWE-bench_Lite \
--predictions_path results/predictions.jsonl \
--max_workers 8 \
--run_id kodit_eval
# For Mac M-series (ARM), build images locally:
python -m swebench.harness.run_evaluation \
--dataset_name princeton-nlp/SWE-bench_Lite \
--predictions_path results/predictions.jsonl \
--max_workers 8 \
--namespace '' \
--run_id kodit_eval
```
**Performance**: ~30 mins for Lite (300 instances) on 16 cores with `cache_level=env`.
**Cloud option**: Run on Modal to avoid local Docker setup:
```bash
pip install modal swebench[modal]
modal setup
python -m swebench.harness.run_evaluation \
--dataset_name princeton-nlp/SWE-bench_Lite \
--predictions_path results/predictions.jsonl \
--modal true
```
---
## 4. Running the Benchmark
### 4.1 Quick Start
```bash
# Step 1: Setup - clone repos at specific commits, index with Kodit
uv run kodit benchmark setup --dataset swebench-lite
# Step 2: Run a single instance (for testing)
uv run kodit benchmark run-one django__django-11049
# Step 3: Run full benchmark
uv run kodit benchmark run --dataset swebench-lite --condition kodit
# Step 4: Evaluate predictions
uv run kodit benchmark evaluate results/predictions.jsonl
```
### 4.2 CLI Commands
```bash
# Show available instances
uv run kodit benchmark list --dataset swebench-lite --repo django/django
# Run specific instances
uv run kodit benchmark run \
--instances django__django-11049 django__django-13447 \
--model claude-3-5-sonnet-20241022 \
--condition kodit
# Compare conditions
uv run kodit benchmark compare \
results/baseline.jsonl \
results/kodit.jsonl
```
### 4.3 Configuration Options
| Option | Default | Description |
|--------|---------|-------------|
| `--dataset` | `swebench-lite` | Dataset variant (lite, verified, full) |
| `--model` | `claude-3-5-sonnet-20241022` | LiteLLM model identifier |
| `--condition` | `kodit` | Retrieval condition (baseline, bm25, kodit, oracle) |
| `--top-k` | `5` | Number of files/snippets to retrieve |
| `--instances` | all | Specific instance IDs to run |
| `--repo` | all | Filter to specific repository |
---
## 5. Architecture
### 5.1 Directory Structure
```
benchmarks/
├── __init__.py
├── cli.py # CLI commands (setup, run, evaluate)
├── swebench/
│ ├── __init__.py
│ ├── instance.py # SWEBenchInstance dataclass
│ ├── loader.py # HuggingFace dataset loader
│ ├── repository.py # Git clone/checkout management
│ ├── retriever.py # Kodit retrieval wrapper
│ ├── prompt.py # Prompt templates
│ ├── generator.py # LLM patch generation
│ └── evaluator.py # SWE-bench harness wrapper
├── repos/ # Cloned repositories (gitignored)
│ └── django__django-11049/ # Instance-specific checkout
├── results/ # Benchmark outputs
│ └── predictions.jsonl
└── cache/ # Indexed repository cache
```
### 5.2 Setup Process
The `setup` command prepares repositories for benchmarking:
1. **Load dataset**: Fetch from `princeton-nlp/SWE-bench_Lite`
2. **Clone repositories**: For each unique `(repo, base_commit)` pair:
```bash
git clone https://github.com/{repo} repos/{instance_id}
cd repos/{instance_id}
git checkout {base_commit}
```
3. **Index with Kodit**: For each cloned repository:
- POST to `/api/v1/repositories` with `file://` URI
- Wait for indexing to complete
- Store mapping: `instance_id → repository_id`
4. **Cache index**: Save Kodit database for reuse
### 5.3 Benchmark Pipeline
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Load Instance│───▶│ Retrieve │───▶│ Build Prompt │
│ from HF │ │ from Kodit │ │ (issue+ctx) │
└──────────────┘ └──────────────┘ └──────┬───────┘
│
┌──────────────┐ ┌──────────────┐ │
│ Evaluate │◀───│ Generate │◀──────────┘
│ with Docker │ │ Patch │
└──────────────┘ └──────────────┘
```
---
## 6. Implementation Details
### 6.1 Retrieval Strategy
The `KoditRetriever` queries Kodit with the problem statement:
```python
class KoditRetriever:
async def retrieve(self, instance: SWEBenchInstance, k: int = 5) -> list[RetrievedFile]:
# Extract key terms from problem statement
keywords = self._extract_keywords(instance.problem_statement)
results = await self._search_service.search(
user_intent=instance.problem_statement,
keywords=keywords,
source_repo=f"github.com/{instance.repo}",
)
# Group snippets by file, return top-k files
files = self._group_by_file(results)
return files[:k]
```
### 6.2 Prompt Template
Following the SWE-bench BM25 baseline format:
```
You will be provided with a partial code base and an issue statement explaining
a problem to resolve.
<issue>
{problem_statement}
</issue>
<code>
[start of {file_path_1}]
{file_content_1}
[end of {file_path_1}]
[start of {file_path_2}]
{file_content_2}
[end of {file_path_2}]
</code>
Generate a patch in unified diff format that resolves the issue.
Only output the patch, no explanations.
```
### 6.3 Prediction Format (Target Output)
Our benchmark must produce a JSONL file compatible with the SWE-bench evaluation harness:
```jsonl
{"instance_id": "django__django-11049", "model_name_or_path": "kodit-claude", "model_patch": "diff --git a/django/db/models/fields/__init__.py b/django/db/models/fields/__init__.py\nindex abc123..def456 100644\n--- a/django/db/models/fields/__init__.py\n+++ b/django/db/models/fields/__init__.py\n@@ -1000,7 +1000,7 @@\n- \"[DD] [HH:[MM:]]ss[.uuuuuu] format.\")\n+ \"[DD] [[HH:]MM:]ss[.uuuuuu] format.\")"}
{"instance_id": "django__django-13447", "model_name_or_path": "kodit-claude", "model_patch": "diff --git a/..."}
```
**Required fields:**
- `instance_id`: Must match exactly (e.g., `django__django-11049`)
- `model_name_or_path`: Identifier for tracking (e.g., `kodit-claude-sonnet`)
- `model_patch`: Unified diff format with `diff --git` header
This file is then passed to the SWE-bench harness:
```bash
python -m swebench.harness.run_evaluation \
--predictions_path results/predictions.jsonl \
...
```
---
## 7. Results Format
### 7.1 Per-Instance Results
```json
{
"instance_id": "django__django-11049",
"condition": "kodit",
"retrieved_files": ["django/db/models/fields/__init__.py"],
"retrieval_recall": 1.0,
"generated_patch": "diff --git a/...",
"resolved": true,
"fail_to_pass_results": {"test_invalid_string": "PASSED"},
"latency_ms": 2340
}
```
### 7.2 Aggregate Results
```json
{
"benchmark": "swebench-lite",
"model": "claude-3-5-sonnet-20241022",
"timestamp": "2024-01-15T10:30:00Z",
"conditions": {
"baseline": {
"resolve_rate": 0.15,
"retrieval_recall_5": 0.0,
"instances_run": 300
},
"bm25": {
"resolve_rate": 0.22,
"retrieval_recall_5": 0.45,
"instances_run": 300
},
"kodit": {
"resolve_rate": 0.28,
"retrieval_recall_5": 0.62,
"instances_run": 300
}
},
"kodit_delta_vs_baseline": "+13%",
"kodit_delta_vs_bm25": "+6%"
}
```
---
## 8. Expected Results
Based on SWE-bench leaderboard data and CodeRAG-Bench findings:
| Condition | Expected Resolve Rate |
|-----------|----------------------|
| Baseline (no retrieval) | ~15% |
| BM25 retrieval | ~22% |
| Kodit retrieval | ~28% |
| Oracle (gold files) | ~45% |
**Key Insight**: The gap between BM25 (22%) and Oracle (45%) represents the potential improvement from better retrieval. Kodit's hybrid search should capture more of this potential than pure BM25.
---
## 9. Troubleshooting
### Repository cloning fails
- Ensure network access to GitHub
- Some repos may require authentication for private forks
- Use `--skip-clone` if repos already exist locally
### Indexing takes too long
- Large repos (django, sympy) can take 10-30 minutes
- Use `--repo` flag to test with smaller repos first (flask, requests)
- Pre-indexed caches can be shared across runs
### Docker evaluation fails
- Ensure Docker daemon is running
- SWE-bench requires significant disk space for containers
- Use `--dry-run` to test pipeline without evaluation
### API key issues
- Set appropriate API keys for your LLM provider
- `ANTHROPIC_API_KEY` for Claude models
- `OPENAI_API_KEY` for OpenAI models
---
## 10. Implementation Checklist
| # | Task | Priority | Status |
|---|------|----------|--------|
| 1 | Create `SWEBenchInstance` dataclass | High | ✅ DONE |
| 2 | Implement HuggingFace dataset loader (`download` command) | High | ✅ DONE |
| 3 | Implement `prepare-instance` command (clone repo, index with Kodit) | High | TODO |
| 4 | Implement Kodit retrieval wrapper | High | TODO |
| 5 | Implement prompt builder | High | TODO |
| 6 | Implement patch generator | High | TODO |
| 7 | Output predictions in SWE-bench JSONL format | High | TODO |
| 8 | Add result aggregation and reporting | Medium | TODO |
| 9 | Add BM25 baseline comparison | Medium | TODO |
### Completed
- **`SWEBenchInstance`** (`src/benchmark/swebench/instance.py`): Immutable dataclass with all SWE-bench fields
- **`DatasetLoader`** (`src/benchmark/swebench/loader.py`): Downloads from HuggingFace, saves/loads JSON
- **`download` command**: `uv run kodit-benchmark download --dataset lite`
- **Dataset stored at**: `benchmarks/data/swebench-lite.json` (300 instances)
### Next Step
**`prepare-instance` command**: Takes a single instance ID, clones the repo at the exact commit, starts a fresh Kodit server, and indexes the repository.
```bash
uv run kodit-benchmark prepare-instance django__django-11049
```
This will:
1. Look up the instance from the downloaded dataset
2. Clone `django/django` to `benchmarks/repos/django__django-11049/`
3. Checkout the exact `base_commit`
4. Start a fresh Kodit server (using existing `start-kodit` infrastructure)
5. Index the repository via Kodit API
6. Wait for indexing to complete
---
## 11. References
- [SWE-bench Website](https://www.swebench.com/)
- [SWE-bench GitHub](https://github.com/SWE-bench/SWE-bench)
- [SWE-bench Evaluation Guide](https://www.swebench.com/SWE-bench/guides/evaluation/)
- [SWE-bench Docker Setup](https://www.swebench.com/SWE-bench/guides/docker_setup/)
- [SWE-bench Paper](https://arxiv.org/abs/2310.06770)
- [SWE-bench Lite Dataset](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite)
- [Agentless Paper](https://arxiv.org/abs/2407.01489) - RAG-based approach achieving 32%
- [CodeRAG-Bench Paper](https://arxiv.org/abs/2406.14497) - Analysis of retrieval impact
