web2spec
Provides web search through DuckDuckGo's HTML backend for the research/search pipeline, enabling keyless page discovery for evidence-backed web research.
Provides a local metasearch backend option for web research, allowing queries to be routed through a SearXNG instance without paid search APIs.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
web2spec
English · Русский
Local, evidence-first web research for API models.
A large model writes only what it needs to find. The pipeline runs on your machine: a local analyst turns the goal into search queries, finds and reads pages, and returns compact, quote-backed facts. The caller never receives HTML, never reads dozens of links, and never pays for a search API.
task -> local search -> pages -> local models -> facts with exact evidenceThe reference machine is a laptop with 4 GB VRAM (RTX 3050 Ti). The runtime
uses GGUF models through llama-server and keeps exactly one model resident at
a time.
Why this exists
Token efficiency. A research task can cost hundreds of thousands of tokens if the caller reads raw pages. web2spec returns a small JSON document instead of HTML and page text.
Local search. SearXNG or DuckDuckGo HTML, no paid API keys, no query metering.
Evidence or nothing. Every returned fact must be an exact substring of the compressed page. Unverified values are moved to
missingorconflicts; the pipeline does not fill gaps from model memory.Small GPU. The fast
collectpath uses two 0.5B models (~380 MB each). Theresearchpath uses a 4B model (~2.4 GB) and unloads it before the collector starts.
Related MCP server: voltcrawl
Measured results
These are proxy measurements, not a general benchmark. The collector was evaluated on 50 held-out task/page pairs from 18 article/Habr URLs only. A separate dated real-page slice covers docs, API reference, pricing and product specs: 1/7 exact cases and 8/19 matched fields with Q4_K_M. That low result is kept visible because it exposes real table-row failures; it is not a general quality claim. Broad job, real-estate, news and tariff coverage is still absent.
Path | Task success | Fact F1 | Evidence | Wrong values after repair | Speed |
Collect, Q4_K_M | 100% (50/50) | 1.0000 | exact substring | 0 | 101 s / 50 pages (~2 s/page) |
Collect, PyTorch fp16 | 94% (47/50) | 0.9388 | exact substring | 0 | ~11.7 s/page |
Planner (test) | — | field F1 0.8220 | — | — | — |
Extractor v2 (test) | — | fact F1 0.9357 | 0.9524 evidence rate | — | — |
The Q4 collector result is high because the test slice is narrow: 18 unique pages, one host, one entity type. It is useful as a regression check, not as a claim that the model solves arbitrary websites.
research was smoke-tested offline on 20 Habr fixtures with Qwen3-4B Q4_K_M:
20/20 pages processed, 37 quote-backed notes, 20 entities, 35 gaps, 10 honest
note omissions (mostly missing authors), 206 s wall time, 24.7 tok/s generation,
7.0 s model load. Those fixtures are metadata-heavy, so this validates the loop
and the output contract, not multi-domain answer quality.
Full methodology, hardware and caveats: docs/BENCHMARKS.md
and the original Russian lab notes in docs/STATS.md.
Two paths, one contract
research — vague task, unknown sources
For questions such as “compare the current speech-to-text tariffs of three vendors” where neither the fields nor the URLs are known in advance.
goal
-> ResearchBrief: entities, questions, source hints
-> 5-8 queries -> local metasearch -> dedupe / rank / domain filter
-> 20-50 pages -> semantic chunks (headings, paragraphs, table rows)
-> 4B notes: entity / field / value / verbatim quote / URL
-> gap analysis -> new queries (up to 2 rounds)
-> deterministic aggregation: facts / conflicts / missing
-> final answer from the 4B model, grounded only in the notesThe analyst is Qwen3-4B-Instruct-2507 Q4_K_M. It runs for the whole research call, then exits before any 0.5B model is loaded.
collect — known URLs, fast path
For “extract title, author and publication date from these five official pages.”
URLs -> polite fetch + robots.txt -> compressed HTML + metadata header
-> planner 0.5B -> CollectionPlan (entity + fields)
-> extractor 0.5B -> FactSet (name / value / evidence)
-> deterministic repair -> strict validation -> facts / missingcollect does not crawl a site. It processes exactly the URLs it is given.
When the caller already knows the schema, pass --field name[:kind[:description]]
(repeatable) to skip the planner and keep field names exact:
web2spec collect --task "Collect price and availability" --url https://shop.example/item `
--field "price:number:price in RUB without discounts" --field in_stock:boolKinds: text, number, date, url, bool, null. Descriptions are shown
to the extractor and disambiguate rows such as "GitHub-hosted" versus
"self-hosted". The MCP collect tool accepts the same list as fields.
A fact is accepted only when its evidence is an exact substring of the
compressed page. repair_factset tries to re-anchor evidence to a nearby exact
fragment; if it cannot, the field becomes missing. Facts outside the plan are
dropped. This is why the output can be used without trusting the model's
memory.
Quick start
Requirements
Python 3.12
llama.cppwithllama-serveronPATH(or setWEB2SPEC_LLAMA_SERVER)~3.3 GB of disk for the three GGUF models
A GPU with ~4 GB VRAM is the tested target; CPU works but is much slower
Windows is the reference platform; Linux and macOS use the same Python code
Install
One command on Windows:
git clone https://github.com/vladmurnik/web2spec.git
cd web2spec
powershell -ExecutionPolicy Bypass -File scripts/00_install.ps1Add -InstallLlama to install llama.cpp through winget, -Dev to install
test dependencies, or -SkipModels to skip the download.
One command on Linux/macOS:
git clone https://github.com/vladmurnik/web2spec.git
cd web2spec
bash scripts/00_install.shAdd --dev for test dependencies or --skip-models to skip the download.
The scripts create .venv, install the package, verify the GGUF checksums, and
run web2spec doctor.
Manual fallback:
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e ".[dev]"
web2spec download-models --only all
web2spec doctorInstall with -e (editable). The CLI resolves scripts/ and models/ relative
to the repository checkout, so this is a repository-first project rather than a
library-only wheel.
download-models verifies SHA-256 hashes from
models/checksums.sha256. The two Q4_K_M collector
models are attached to the GitHub release; the 4B analyst comes from the
official Qwen GGUF repository. Optional Q3_K_M/Q2_K, f16 and merged artifacts
are local-only and are not distributed. Model files are not stored in git.
Expected doctor output:
[OK ] analyst Q4_K_M: .../models/gguf/Qwen3-4B-Instruct-2507-Q4_K_M.gguf (2381.6 MB)
[OK ] planner Q4_K_M: .../models/gguf/qwen05b-plan-Q4_K_M.gguf (379.4 MB)
[OK ] extractor Q4_K_M: .../models/gguf/qwen05b-plan-extract-v2-Q4_K_M.gguf (379.4 MB)
[OK ] llama-server: .../llama-server
[OK ] Python: 3.12.xCached repeat runs
collect, search, and research keep fetched pages and search results in
.cache/web2spec.sqlite for 24 hours. This makes repeated runs cheaper without
turning stale pages into permanent facts:
web2spec collect --task "..." --url https://example.com/page
web2spec collect --task "..." --url https://example.com/page --cache-ttl 1
web2spec collect --task "..." --url https://example.com/page --no-cacheUse --previous data/reports/old.json on collect or search to reuse pages
whose content hash did not change. The report contains content_hash,
cache_hit, and a cache block in stats, so reuse is auditable.
First run: known URLs
web2spec collect `
--task "Collect article titles, authors and publication dates" `
--url https://habr.com/ru/articles/1103276/ `
--out data/reports/demo.jsonFirst run: research
web2spec research `
--task "Compare Yandex SpeechKit, Sber SaluteSpeech and VK Cloud speech-to-text tariffs in 2026: price per hour, free tier, Russian language and diarization. Use official pages and report missing fields honestly." `
--domains developers.sber.ru,cloud.vk.com `
--min-pages 20 --max-pages 30 `
--out data/reports/research.jsonOne research call typically takes 2–4 minutes on the reference laptop. It
writes a JSON report with answer, entities, sources, gaps, errors,
stats and speed; --include-notes is available for audits.
MCP server
web2spec mcp starts a stdio MCP server. Any MCP-capable client can call two
tools:
Tool | Use it when | Main output |
| The task is vague and sources/fields are unknown |
|
| You already have official URLs | aggregated |
Copy configs/mcp.example.json into the client
configuration and replace the paths. Claude Desktop stores it in
claude_desktop_config.json; Cursor uses ~/.cursor/mcp.json.
{
"mcpServers": {
"web2spec": {
"command": "C:\\path\\to\\repo\\.venv\\Scripts\\python.exe",
"args": ["C:\\path\\to\\repo\\scripts\\23_mcp_server.py"],
"env": {
"WEB2SPEC_PROVIDER": "ddg",
"WEB2SPEC_LLM_MODEL": "qwen2.5:3b"
}
}
}
}WEB2SPEC_LLM_MODEL is optional and only affects the old search path; the
MCP research path uses the local 4B analyst. Do not run research and
collect concurrently on a 4 GB GPU. The server writes diagnostics to stderr;
stdout is protocol-clean JSON-RPC.
The response is compact JSON, never HTML. A typical collect result:
{
"task": "Compare speech-to-text tariffs",
"facts": { "russian": true },
"conflicts": {
"price_per_hour_rub": [
{ "value": 36, "sources": ["https://developers.sber.ru/..."] },
{ "value": 30, "sources": ["https://cloud.vk.com/..."] }
]
},
"missing": { "diarization": ["https://cloud.vk.com/..."] },
"by_source": {
"https://developers.sber.ru/...": { "price_per_hour_rub": 36 },
"https://cloud.vk.com/...": { "price_per_hour_rub": 30 }
},
"sources": ["https://developers.sber.ru/...", "https://cloud.vk.com/..."],
"errors": [],
"stats": { "requested": 2, "ok": 2, "failed": 0 }
}When sources disagree, both values are returned. The pipeline does not pick a winner.
CLI
web2spec doctor
web2spec download-models --only collector
web2spec collect --task "..." --urls data/urls.txt
web2spec search --task "..." --provider ddg --search-only
web2spec research --task "..." --domains vendor-x.ru --min-pages 20
web2spec mcp
web2spec benchmark
web2spec eval-q4 --limit 50Command | Purpose |
| fetch GGUF models and verify checksums |
| mirror fine-tuned artifacts to Hugging Face |
| run dated real-page cases through the collector |
| check models, |
| task + URLs -> structured facts |
| task -> queries -> local search -> collector |
| task -> 20–50 pages -> evidence report |
| MCP stdio server |
| Q4_K_M quality/speed evaluation |
| PyTorch runtime cascade evaluation |
| planner/extractor evaluation |
| QLoRA training and adapter merge |
Every numbered script in scripts/ remains runnable on its own.
See scripts/README.md for the full map.
Search providers
SearXNG JSON (recommended): self-hosted, aggregates public engines, no API key. Enable
formats: [html, json]insettings.ymland pass--provider searxng --searxng-url http://localhost:8080.DuckDuckGo HTML (fallback): no installation, but public endpoint and best-effort; it can rate-limit or return a captcha.
search can use an optional local LLM for query generation (Ollama or any
OpenAI-compatible server). Without it, the task text is used as the query.
research does not need Ollama: its 4B analyst writes the queries.
Models
Role | File | Size | Source |
Analyst ( |
| 2381.6 MB | Qwen official GGUF |
Planner ( |
| 379.4 MB | fine-tuned Qwen2.5-0.5B |
Extractor ( |
| 379.4 MB | fine-tuned Qwen2.5-0.5B |
Base models are Apache-2.0. The Q4_K_M collector GGUFs are attached to the
GitHub release; the same runtime artifacts can be mirrored to Hugging Face with
upload-models. See docs/MODELS.md for checksums, rebuild
steps, optional local Q3_K_M/Q2_K, f16 and merged artifacts, and the
quantization caveat. Rebuilding the collector requires the training data, which
is not committed.
The optional Q3_K_M/Q2_K, f16 and merged artifacts are not distributed. They are
built locally from the training pipeline; their hashes are in
models/checksums-extra.sha256.
The PyTorch fallback (--backend hf) requires the hf extra:
pip install -e ".[hf]"
web2spec collect --task "..." --urls data/urls.txt `
--backend hf `
--planner models/merged/qwen05b-plan `
--extractor models/merged/qwen05b-plan-extract-v2Training and full evaluation use .[train]:
pip install -e ".[train]"
web2spec train --config configs/train_plan_extract_reliable_4gb.yamlOutput contract
collect writes one JSON document:
{
"task": "Collect article titles and authors",
"generated_at": "2026-11-06T12:00:00+00:00",
"pages": [
{
"url": "https://example.com/article/1",
"status": "ok",
"entity": "article",
"facts": [
{ "name": "title", "value": "How to speed up data collection", "evidence": "How to speed up data collection" }
],
"missing": ["published_at"],
"repairs": { "evidence": [], "dropped": [] },
"dropped_foreign": [],
"error": null
}
],
"aggregate": {
"https://example.com/article/1": {
"status": "ok",
"entity": "article",
"facts": { "title": "How to speed up data collection" },
"missing": ["published_at"],
"error": null
}
},
"stats": { "requested": 1, "ok": 1, "failed": 0, "by_status": { "ok": 1 } }
}status is one of ok, fetch_error, plan_error, extract_error.
missing is not a failure of the run; it is the honest answer that the page
does not contain a verifiable value. Exit code 0 means at least one page is
ok, 1 means all pages failed, 2 means invalid arguments.
research adds goal, questions, entities (facts, conflicts,
evidence), notes, gaps, sources, errors, stats and speed.
Repository layout
src/web2spec/ library: schemas, HTML compression, research loop, CLI
scripts/ numbered pipeline: data -> training -> eval -> runtime -> MCP
configs/ YAML configs, MCP example, seeds
tests/ pytest suite; no GPU or network required
docs/ architecture, benchmarks, model notes, lab history
examples/ small task/URL examples
data/ datasets and reports (gitignored)
models/ GGUF weights (gitignored) and checksum manifest
outputs/runs/ adapters, checkpoints, training logs (gitignored)Key modules:
web2spec/html_compress.py— HTML compression and metadata extraction;web2spec/semantic.py— semantic chunks: headings, paragraphs, table rows;web2spec/research.py— question-first loop, notes, gaps, aggregation;web2spec/plan.py—CollectionPlan,FactSet,repair_factset, validation;web2spec/llama_runtime.py—llama-serverlifecycle and Q4 defaults;web2spec/tokens.py— ChatML prompts without a tokenizer dependency;web2spec/cli.py— unifiedweb2specentry point.
Development
pip install -e ".[dev]"
python -m ruff check .
python -m pytestThe suite is CPU/network-free and covers schemas, compression, plan repair, research aggregation, search parsing, the MCP protocol and the CLI. CI runs the same commands on Ubuntu and Windows with Python 3.12.
Training data and reports are not committed. The original experiments, plans
and metrics are preserved in docs/ and the numbered scripts. Read
CONTRIBUTING.md before changing the evidence contract or
adding a model path.
Limitations
Narrow quality evidence. Collector quality is measured only on article/Habr pages; research only on Habr fixtures. There is no multi-domain benchmark yet. Do not present the numbers as general accuracy.
No browser. The fetcher does not execute JavaScript, wait for SPAs, solve captchas, bypass WAFs, log in, or crawl pagination. A blocked page is reported as
blocked/fetch_error, not silently skipped.Best-effort search. Public metasearch endpoints can rate-limit. SearXNG is recommended for repeatable use.
Untrusted page text. Fetched pages may contain prompt injection. The pipeline returns data, never instructions or tool calls, and requires exact evidence, but a malicious page can still bias what the model chooses to note. Treat outputs as untrusted data and verify high-stakes numbers at the source.
One model at a time. On 4 GB VRAM, running the 4B analyst and the 0.5B collector together will exhaust memory. The CLI enforces sequencing by starting and stopping
llama-serveraround each stage.Not a crawler.
collectprocesses exactly the supplied URLs. It does not discover a site's full catalogue or deduplicate entities across pages.Model weights are separate. A clean clone contains no
.gguffiles. Runweb2spec download-modelsor build them from the training pipeline.
License
MIT — see LICENSE. The base Qwen models are Apache-2.0; model
licenses and redistribution notes are in docs/MODELS.md and
THIRD_PARTY_NOTICES.md. Datasets and scraped pages
are not redistributed.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Web search, fetch, extract, and research for AI agents. Markdown output + AI-synthesized answers.
LLM-ready web search + instant answers + URL-to-clean-text fetch for agents and RAG.
Web scraping, search, crawling, structured extraction and research for AI agents.
Live web research with related queries, browser-rendered sources, images, and evidence.
Related MCP Servers
- AlicenseNot gradedqualityAmaintenanceEnables coding agents to run local-first web research: intent-routed search across independent engines with reranking, a multi-stage fetch/crawl ladder, and document extraction. Results come back as signed-cursor, citation-bearing evidence envelopes, with an optional separately enabled profile for browser click/type actions.AGPL 3.0
- AlicenseNot gradedqualityBmaintenanceEnables coding agents to retrieve verified web evidence via headless Chromium rendering, sitemap navigation, and search candidate discovery, returning clean Markdown, structured JSON, and verbatim quoted proof.5 npmMIT
- AlicenseNot gradedqualityCmaintenanceEnables cited research synthesis from verified public sources, returning extractive quotations with source URLs and refusing topics that cannot be grounded.MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and applications to find authoritative primary data sources from natural-language queries, receive step-by-step website navigation guidance, and integrate evidence-backed external facts into their workflows.-