Skills Wiki (self-hosted)

Manage the AI skills you already installed. Skills Wiki routes each request to the right skill, remembers your corrections as learnings, and evaluates and benchmarks your skills on your own tokens. Everything runs on your machine, with no account and no server of ours.

• Route: one router for all your skills, instead of every skill's description competing in your agent's context.

• Learn: corrections you make ("from now on, sign off with Best") are attached to that skill every time it loads.

• Benchmark: generate a test suite for any skill, run it with and without the skill, and see whether it helps.

Optional: bring your own TypeSafe key and JEV, a calibrated decision model, routes your requests, checks your test suites and grades through a cascade. Without a key, everything still works.

Works with skills for Claude Code, Codex and Antigravity (folders containing a SKILL.md).

Beta (v0.1). Needs Python 3.11–3.13. Adopting a skill moves its folder, so back up your skills folders first (~/.claude/skills, ~/.agents/skills, ~/.gemini/skills, ~/.codex/skills). See Status and limits before you rely on it.

Quickstart

# needs pipx: brew install pipx   (or: python3 -m pip install --user pipx), then once: pipx ensurepath
pipx install git+https://github.com/appleaa123/skills_wiki_opensource.git
skillswiki scan                    # find installed skills
skillswiki adopt <slug>            # let Skills Wiki route and load it (reversible: skillswiki release <slug>)
claude mcp add skills-wiki -- skillswiki serve-mcp    # or any MCP client
skillswiki ui                      # the local management page

Full walkthrough, including the Claude Code hook and other agents: docs/quickstart.md.

Related MCP server: One Agent Plugin

How routing works

Adopt. Your agent triggers a skill natively by reading every installed skill's description. With dozens of skills those descriptions crowd the context and compete with each other. skillswiki adopt <slug> moves the skill folder from your agent's skills directory into ~/.skillswiki/library/. Your agent stops triggering it on its own, and Skills Wiki routes it instead. skillswiki release <slug> moves it back, byte for byte. Plugin-managed skills are listed but never moved. If the same skill is installed for several agents, identical copies move with it (and come back on release), so no agent keeps triggering it on its own; a copy that differs stays put and adopt warns you about it.

Without a key: keyword routing. Each request is matched (BM25) against the skill's own description, the trigger text its author wrote. Skills Wiki returns a shortlist and your agent decides. It never forces a skill.

Routing cards. Descriptions are often one terse sentence, so keyword matching misses synonyms ("this email sounds robotic" never mentions "rewrite"). A routing card adds example requests, keywords and what the skill is not for. Write one by hand (skillswiki card set), or click Enrich on the page (or run skillswiki enrich <slug>) to have your own AI draft one from the SKILL.md. That uses your tokens, so it is never automatic. The page shows "card: yes/no" for every skill. Cards live only in Skills Wiki's database, never in your skill files.

With a TypeSafe key: JEV routing. JEV asks whether the request needs a skill at all, picks among the candidates and checks the top picks' fit. It suggests a single skill only when it is confident (fit ≥ 0.85); otherwise your agent gets the shortlist. On Skills Wiki's own catalog and frozen task set, at that bar JEV was right 98.5% of the time when it suggested, suggested on about 65% of requests, and made 0% wrong-tool and 0% needless suggestions. Your library is different, so treat that as a strong guide, not a promise. If JEV is unreachable, or takes more than a few seconds, routing falls back to keywords. In a live test with five adopted skills, each routed request used about 2,000–2,700 JEV tokens (about $0.0001); "what's the capital of France?" was correctly told no skill was needed.

Ways in. The CLI (skillswiki ...) is the main interface. An MCP server over stdio gives agents five fixed tools (suggest_skill, load_skill, list_skills, learning_record, learning_list), the same five whether you have 5 skills or 500. A Claude Code UserPromptSubmit hook routes every prompt and adds one line of advice; it never pastes the skill and never blocks your prompt.

Use the hook. In real Claude Code and Antigravity sessions (2026-10-03), an agent that only had the MCP tools usually answered questions it felt sure about from its own knowledge, without checking for a skill. With the hook on, Claude Code loaded the suggested skill and followed it, and learnings recorded earlier showed up in the answer. Antigravity has no prompt hook yet: ask it to check Skills Wiki, and allow its MCP calls (in non-interactive mode it blocks them unless permissions are approved).

How evaluation works

A skill should make your AI's answers measurably better. Skills Wiki tests that the way you'd test any change: same tasks, with and without the skill, graded blind.

1. Generate a test suite. Skills you install don't come with tests, so skillswiki eval generate <slug> asks your own AI to draft one from the SKILL.md: six tasks a real user would type, plus a rubric to grade them. The prompt carries hard-won rules. Tasks must not restate the skill's own instructions (if the task tells the AI how to do the job, the no-skill answer scores just as well and the test measures nothing). At least two tasks are plain requests with no style hints, to measure the skill's default behaviour. At least one tests a failure the skill warns against. Criteria are phrased as what a good answer does, and each is tagged with its kind (factual, constraint, count, legal or style), which decides how far JEV may ever be trusted on it. Review it with skillswiki eval show <slug> and edit freely; it is yours.

2. Check the suite before spending tokens. skillswiki eval check <slug> validates the files, lints the rubric, and flags task prompts that copy the skill's wording. With a TypeSafe key, JEV reads every prompt and criterion in one batched call and flags prompts that restate the skill, prompts that don't read like a real request, and criteria a grader couldn't decide from the answer alone.

3. Run it. skillswiki eval run <slug> --tier screen (quick) or --tier publish (each task, with and without the skill, three times). Your AI CLI does the work on your plan; the token estimate is shown first.

4. Grade it. Every answer is graded against the rubric by a judge, ideally a different AI from the one that did the work (--judge gemini while Claude works). The judge also compares the two answers to each task blind, in both orders, and picks the better one. With a key, --cascade lets JEV grade every criterion too. Its grade becomes final only where your own runs show it agrees with your judge; until then your judge decides and JEV's grades are stored as evidence (skillswiki eval calibration <slug>).

5. Read the benchmark. A verdict (gain, noise or loss), the change in pass rate with a 95% confidence interval, blind wins/losses/ties, token cost, and named findings such as a check that only the skill passes. skillswiki eval accept <id> makes a run the baseline; later runs are compared with it only when measured the same way (same AI, judge, settings and rubric).

6. Learn. Corrections you record (skillswiki learn add, or your agent via learning_record) are attached to the skill every time it loads. --tier learning_screen measures whether they actually help.

A real run

llm-approach-advisor (one of our own skills: it recommends prompting vs RAG vs fine-tuning for an LLM project), on 2026-10-03. Claude did the work, Gemini judged, and the TypeSafe key was on.

• Suite: generated in 50 seconds (~49,000 tokens). Six tasks, e.g. "A couple of people on the team think we should fine-tune a model on the docs so it 'really knows' the product. Is that the right call?" JEV's check cost $0.00015 and flagged two criteria as hard to grade from the answer alone.

• Benchmark (publish tier, two runs, ~137,000 Claude tokens each, 6–10 minutes): verdict gain both times. The judge preferred the answer written with the skill 18 of 18 times in each run. Pass rate went from 33% to 67% in the first run and to 83% in the second, under identical settings: run-to-run noise is real, which is why the interval (−22 to +78 points, then +6 to +89) is shown next to every delta.

• What the skill fixed: naming rejected alternatives with reasons (100% with the skill vs 44% without) and including a safety note. What it still missed: stating the constraints it relied on (11% with the skill).

• Learning: one recorded learning targeting that gap ("list each constraint and label it from the inputs, inferred, or unknown") took the pass rate from 67% to 100%, preferred 6 of 6 times blind (one quick run of six tasks; a small sample).

• JEV in the cascade: ~116,000 JEV tokens, about $0.005 per run. JEV made the final call on 0 of 468 grades. That is the system working as designed: the generated rubric left criteria untagged, untagged criteria are treated as factual, and on factual criteria JEV may only fail an answer, never pass it. JEV rarely failed these good answers, so most criteria are still collecting evidence, and on two criteria it disagreed with the judge, so those stay with the judge. Since this run, the generator tags every criterion's kind: style (tone, clarity, structure) only where a lenient grade is harmless, which is where JEV can earn the right to settle grades after enough agreeing runs. As a safety net, a style tag on wording that matches a list of legal, medical, financial and safety words is set back to factual; it is a word list, so review the kinds before trusting them.

JEV, honestly

JEV (by TypeSafe) is a decision model: it does not write text. It answers typed questions (pick one, score against levels, true or false) with calibrated probabilities. In Skills Wiki it:

• routes requests to your adopted skills (above);

• checks generated test suites before you spend tokens on them;

• grades in a cascade: it grades every criterion, but its grade only becomes final where your own runs have shown it agrees with your LLM judge. Fact rules and "no X" rules can only ever be failed by JEV, never passed.

What it does not do: replace your judge. On Skills Wiki's own study JEV's grade was final on about 0–21% of grades, and every output still needed one LLM call. Its value is cheap, calibrated triage, and routing. JEV is not deterministic either: about 5.6% of picks changed between identical runs, so 1–2 point differences are noise.

Bring your own key: put TYPESAFE_API_KEY= in ~/.skillswiki/.env (see .env.example). You are TypeSafe's customer under their terms; Skills Wiki never sees your key. SKILLSWIKI_JEV=off turns it off.

Costs and privacy

• Tokens. Every eval run, routing-card enrich and suite generation runs on your own AI plan. eval run shows a token estimate first and asks before the larger tiers. Tips: start with --tier screen, and judge with a different CLI (codex or gemini) when you have one, since a model grading its own outputs tends to favour them. If only one CLI is installed, pass --judge claude (for example) to opt in.

• Privacy. Everything stays on your computer (~/.skillswiki/). Nothing is sent anywhere except to your own AI CLI and, with a key, to TypeSafe.

• Scripts. A skill folder can contain scripts your agent may run. The page flags skills that contain scripts and skills whose files changed since you adopted them.

Status and limits

What has been tested, and what to know before relying on it:

• Use the Claude Code hook. Without it, agents usually answer from their own knowledge and never check for a skill. With it, routing is automatic.

• Antigravity works through MCP, but has no prompt hook yet, so ask it to check Skills Wiki. In non-interactive mode it blocks MCP calls unless you approve permissions.

• Codex skill folders and the Codex judge are covered by tests only, not by a live Codex session yet.

• JEV is not fully deterministic. About 5.6% of its picks changed between identical runs, and a borderline request is sometimes judged as needing no skill; the shortlist and keyword fallback still apply.

• The benchmark numbers above come from one skill, so treat them as an example, not a promise.

• The JEV grading cascade had not yet made a final grade in a live run when this was written (0 of 468). Generated suites now tag criterion kinds so it can earn trust on style criteria; that has not been confirmed live.

• Python 3.11, 3.12 and 3.13 are tested in CI.

Uninstall

skillswiki release --all           # put every adopted skill back where your agents expect it
pipx uninstall skillswiki
rm -rf ~/.skillswiki               # optional: your learnings, cards, eval results and settings

Uninstalling never deletes your skills. If you uninstall before releasing, adopted skills stay in ~/.skillswiki/library/, and reinstalling then running skillswiki release --all puts them back. Without Skills Wiki at all, ~/.skillswiki/library/RESTORE.json lists where every folder came from, so you can move them back by hand.

Want it done for you?

Run it yourself, free. Or let skillwiki.app find, vet, maintain and benchmark skills for you, with suites already written and tested, working in claude.ai, ChatGPT and other web chats, not just your terminal.

Licence

Elastic License 2.0 with an Additional Limitation: source-available (fair-code), not OSI open source. Free for your own use, including inside a for-profit business. You may not sell it, offer it as a hosted service, or install, run or support it for paying clients. Commercial licence: aliu@skillwiki.app. See LICENSE.

Contributing: see CONTRIBUTING.md. Security: see SECURITY.md.