Hittbar

Can an AI agent find the right API in If Insurance's public catalogue, and what would actually make it better at it?

Hittbar is Swedish for findable.

If's developer portal lists 44 product pages and 68 API products. Its CMS already has a field called aiDescription on every product page, and on the day of this snapshot it was empty on all 44. The obvious move is to fill it. This repo measures whether that would help, before anyone spends the effort.

Unofficial. Built from the portal's public pages only; the portal's llms.txt allows summarisation, indexing, generation and Q&A. No If API is ever called. Snapshot: 2026-09-24.

The short answer

1. When the agent reads the whole catalogue, today's descriptions already work. Claude Opus 5 picked the right product for 66 of 66 tasks, in every condition and every run, including the traps. Adding aiDescription made each call 77% larger, 64% more expensive and slower, for no gain.

2. When the agent has to search, language is the gap, not description quality. The catalogue is English only. Given a search tool with no hint about language, Opus wrote Swedish or Norwegian queries for 14 of 24 Nordic-language tasks, and Swedish tasks reached the top 5 only 27–40% of the time. One sentence in the tool description, "The catalogue is written in English", took that to 73–93% and made every query English.

3. For search, more relevant words beat words written for agents. The control condition (the page's own prose) retrieved better than the generated aiDescription.

What this would mean for the catalogue: state the language in whatever tool or MCP server exposes it; don't prioritise hand-written agent guides at this size; when the catalogue outgrows a context window, index the page text that already exists.

And then I built the thing the numbers point to: an MCP server over the catalogue (hittbar/server.py) whose search indexes the page text and whose tool description states the language. Below.

Related MCP server: internal-swagger-mcp

The setup

Catalogue. hittbar/scrape.py reads the portal's public /llms-full.txt, then each product page's embedded Next.js data: page, API products, their APIs and descriptions → data/catalog.json.

Tasks. 66 developer needs in data/tasks.jsonl, 55 English, 7 Swedish, 4 Norwegian, in four kinds:

The tasks were committed (9e32d60) before any aiDescription was generated, so they could not be tuned to the result.

Three views of the catalogue, identical except for one addition:

If ai beat published only because it adds words, pagetext would beat it by as much.

Grading is deterministic. Each task lists the product ids that satisfy it; an empty list means only "none" is right. The agent's answer is constrained by a JSON schema whose product field is an enum of real ids, so a wrong answer is always a real, wrong product. No LLM judge anywhere.

Results

1. Full catalogue in context: hittbar/evaluate.py

Claude Opus 5, effort low, three runs per condition, 198 calls per condition.

A ceiling. The eval cannot separate the conditions, and I checked that the 100% is real by reading the reasons on the hardest traps: a used-car warranty page that has no product (the right answer lives on another page), managing a subscription to roadside events (not the events themselves), and Finnish car insurance, which can be priced but not bought.

2. Search with the developer's own words: hittbar/search.py

BM25 over one document per product, no model. A deliberately naive baseline.

3. Search with a query the agent writes: hittbar/query.py

The objection to table 2 is that no agent searches with the user's words verbatim. So here Opus sees only the task and a search tool's description, never the catalogue, and writes the query. Three runs, 58 answerable tasks each. Recall@5:

Per-run recall@5 with the hint: published 0.83–0.88, ai 0.85–0.91, pagetext 0.91–0.93. The ai range overlaps published; pagetext does not.

The mechanism, from the queries themselves (runs/query-*.json):

Task (sv):  Sälj hemförsäkring till privatpersoner i Sverige, med tre skyddsnivåer, direkt i vår app.
No hint:    hemförsäkring privat Sverige försäljning API offert tre skyddsnivåer köp i app
Hint:       home insurance sales API for private customers Sweden, quote and purchase, three coverage levels

What I could not claim

• That aiDescription is useless. It is useless here: 44 pages fit in one context window, and a frontier model reads them all. A smaller model, a much larger catalogue, or embedding search could change that. None of those was measured.

• That the Swedish numbers are precise. 5 Swedish and 3 Norwegian answerable tasks, three runs each, is 15 and 9 samples. The direction is large and consistent across runs; the exact percentages are not.

• That the tasks are unbiased. One author wrote all 66 after reading the catalogue, so they lean towards the vocabulary of the current descriptions. That bias favours published, which makes result 1 conservative and result 3's control win slightly less surprising.

• That BM25 is how If would search. It is the simplest lexical baseline. An embedding index would likely narrow the language gap on its own; that is the next experiment, not a result.

Found in the catalogue along the way

• One API published on two pages. Manage Insrt Ancillary Insurances appears under both other/ and self-service/ with the same slug.

• A page with no API product. Manage Used Car Warranty Sweden describes Check, Register and Cancel operations but publishes no product. An agent sent there finds nothing to call. The full-context agent correctly routed warranty tasks to Manage Insrt Warranty Insurances instead.

• Page text that describes another market. The Swedish car insurance page's related-content block talks about Norwegian customers. The scraper drops related-page blocks for that reason: an index built on the raw page would learn that Sweden means Norway.

• Product descriptions that repeat the name. Five products, for example Buy Car Insurance NO, have a description identical to their title. The full-context agent coped; a search index gets nothing from them.

The MCP server

hittbar/server.py exposes the snapshot to any MCP client over stdio. Two read-only tools:

Every design choice comes from a measurement, not from taste:

• The search tool's description says "The catalogue is written in English." and tells the agent to write the query in English and name the country. That is result 3 turned into a default. A test fails if the sentence is removed.

• It indexes the page text, not a generated aiDescription. That is result 3's control.

• Both tools return typed, schema-described output, and an unknown product_id is a tool error that tells the agent to search instead, not an empty answer.

The tests (tests/test_server.py) go through the MCP protocol, in-process and as a real stdio subprocess, and replay the README's own example: Opus's English query for task t47 ranks buy-home-insurance-se first; its Swedish query does not reach the top 5.

Connect it to Claude Code:

claude mcp add hittbar -e PYTHONPATH=/path/to/hittbar -- /path/to/hittbar/.venv/bin/python -m hittbar.server

or to any client that takes a stdio command, from the repo root: .venv/bin/python -m hittbar.server.

Not built, deliberately: access control. The public catalogue needs none, and a real version in front of If's partner specs would sit behind the portal's existing OAuth (client credentials or auth code with PKCE) rather than invent its own. Also not built: an embedding index, which is the next experiment above, not a result.

Cost

Run it

python -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python -m pytest              # 23 tests, no key, no network
.venv/bin/python -m hittbar.search      # free, deterministic
.venv/bin/python -m hittbar.server      # the MCP server, stdio

export ANTHROPIC_API_KEY=...
.venv/bin/python -m hittbar.evaluate --runs 3   # ~$10
.venv/bin/python -m hittbar.query --runs 3      # ~$1
.venv/bin/python -m hittbar.scrape              # refresh the snapshot
.venv/bin/python -m hittbar.describe            # regenerate the aiDescription drafts

The model is claude-opus-5 by default; set HITTBAR_MODEL to change it.

hittbar/scrape.py      public portal -> data/catalog.json
hittbar/catalog.py     the three views, byte-stable so the catalogue prefix caches
hittbar/describe.py    aiDescription drafts, grounded in each page's own text
hittbar/agent.py       pick one product or "none", schema-constrained
hittbar/evaluate.py    full-context eval, deterministic scoring, cost and latency
hittbar/search.py      BM25 lens, no model
hittbar/query.py       agent-written queries through the same index
hittbar/server.py      MCP server: search + product lookup, read-only
data/tasks.jsonl       66 tasks, frozen before generation
runs/                  every run behind every number above

MIT.