Assay

Read-quality scoring for AWS Builder Center. An assay is the test that tells you how much actual metal is in an ore; this one tells you how much substance is in an article.

Builder Center publishes around 307 articles a week. Nothing on the site tells you which are worth opening, and the signal it does show you carries no information about quality — across a full week (n = 302), likes correlate with measured quality at r = −0.09, and the top 20 by likes and the top 20 by score share zero articles.

Assay gives every article a Read-Quality Score (RQS, 0–100) and a verdict — READ · SKIM · SKIP — plus one blunt sentence explaining it, and tells authors exactly what would raise theirs.

Live: https://d1m5fcrjmmi9ue.cloudfront.net

What it actually measures

The question is the reader's — is this worth my time? — not can the author prove they ran it. Being written with AI assistance is not a defect. Being empty is.

Handing an article to a model and asking whether it is any good reproduces the exact failure being detected: models reward fluency, and fluent-but-empty is the thing to catch. So eight deterministic tools run before any model does, and their output is ground truth the judges may interpret but never contradict:

Then a Strands multi-agent fleet on Bedrock AgentCore scores fifteen dimensions, grouped into four families. The question every one of them serves is the reader's — is this worth my time? — not can you prove you ran it.

Fifteen dimensions, but only four judge calls — each judge returns its whole family in one structured response. Granularity for the author-facing advice without paying for it in tokens.

An earlier version weighted proof-of-execution at 30 and scored a substantial project write-up as SKIM for linking a repository instead of pasting a terminal session. Execution proof is now one signal inside evidence, weighted to the strength of the claims made: an opinion piece needs none, a claimed 10x speedup needs the measurement. Judges are told to calibrate against what the article is trying to be — a news post, an explainer and a deep dive succeed differently.

Author credibility

Authors who carry accountability for what they publish get an explicit bonus on the finished score — AWS Heroes are vetted and renewed annually, Amazon employees write under their employer's name:

Three properties make this safe to have:

• It is applied to the final score, never inside a judge. The author's status is deliberately withheld from the judges' evidence block. If a judge knew who wrote a piece it would inflate five separate rationales invisibly; as one number it is auditable — base_rqs and author_bonus are stored separately and shown in the score breakdown.

• It cannot rescue an evidence-free post. No bonus applies below the SKIM floor (SIR_BONUS_FLOOR, default 40).

• It cannot turn SKIP into READ on its own. The largest bonus applied at the SKIM floor still lands short of the READ threshold; a test asserts this.

Tune with SIR_BONUS_HERO, SIR_BONUS_AMAZONIAN, SIR_BONUS_CB, SIR_BONUS_FLOOR. Set them all to 0 to score purely on evidence.

The RQS is arithmetic, not a model output. Judges score their own dimensions; the weighted sum happens in Python. And every dimension is capped by what was measured — services named but never operated hold aws_service_depth at 49; an API that exists in no SDK holds code_quality at 29 and accuracy at 35, however persuasive the prose. That guardrail is what makes the score hard to talk your way past.

Caps are demanded by content type, not universally. An opinion piece with no citations is not penalised; a reference article asserting facts with none is. An article containing no code is not capped at all — it simply is not the kind of article code was expected from.

Stability

An LLM judge whose variance you have not measured is a judge whose scores you cannot interpret. Scoring one article five times, at temperature 0, originally gave:

Measured against the rubric of the time, one sample returned 0 for evidence on an article containing five pasted terminal sessions, then 80 a minute later.

Two fixes, in the order that mattered. Each judge is sampled SIR_JUDGE_SAMPLES times (default 3) and the per-dimension median is taken, with the disagreement recorded in signals.sample_spread. And every one of the fifteen dimensions now carries explicit score bands anchored to the measured counts — judgement operates inside a band, never to move between bands. The bands did most of the work; more compute alone did not fix it.

python analysis/stability.py --samples 5                 # measure it yourself
python analysis/stability.py --samples 5 --judge-samples 1   # without the median
python analysis/likes_vs_quality.py --run week           # do likes predict quality?

stability.py exits non-zero if the verdict changes across identical runs.

Related MCP server: gts-repo-guardian

Quick start

uv venv --python 3.13 && uv pip install -e ".[dev]"

# score one article
assay score https://builder.aws.com/content/<id>/<slug>

# score the 15 most recent articles from a rolling 3-day window (the default corpus)
assay run --run latest --days 3 --limit 15

# build the leaderboard and the filtered feed
assay report --run latest
assay page   --run latest
assay feed   --run latest --min-rqs 70 --topics bedrock serverless

Recommendations: turning the score into advice

Scoring answers "is this worth reading?". The more useful question for an author is what would move it, and the caps make that computable rather than hand-wavy.

$ assay review https://builder.aws.com/content/<id>/<slug>

  Weekend Agent Challenge: Opportunity Finder Agent
  RQS 50/100 -> SKIM   (20 points from READ)
  addressing everything below reaches about 75 (READ)

  already working:
    + 4 code blocks parse cleanly
    + includes complete runnable files

  recommendations, highest impact first:

  1. Support the claims you are making   [~4.0 RQS]
     evidence - proportionate to the claims
     Not every article needs terminal output - but every claim needs something
     behind it. Show the measurement for results, and say plainly what you did
     not test.

  2. Cite the factual claims   [~3.9 RQS]
     sources and citations

  3. Operate the services, do not just name them   [~3.5 RQS]
     AWS depth - operating or naming
     One parameter that mattered, a quota you hit, an IAM condition or a cost
     figure is worth more than another paragraph describing a service.

  [= exact: this cap is provably costing that much]
  [~ estimate: realistic headroom on that dimension]

Because the weights and caps are deterministic, a binding cap has an exact price. A cap holding sources at 50 when the judge wanted 85 costs precisely (85 − 50) × 6 / 100 = 2.1 RQS. Recommendations are ranked by what they are actually worth rather than by how important they sound, and marked exact or estimate so the difference is visible.

Available four ways: the assay review CLI command (accepts a local draft or a published URL), the /review.html page, GET /api/review?url=, and the review_article MCP tool.

A test asserts that every cap the scorer can emit has matching advice, so a new cap cannot ship as a finding nobody can act on.

Surfaces

Every surface applies the same Preferences filter, so the feed and the browser extension can never disagree.

The published site is built through Assay's own MCP server. publish_site.py seeds DynamoDB, then reads the corpus back out via the export_corpus MCP tool and renders the leaderboard, feed and extension index from that — over stdio by default, or against the deployed AgentCore runtime with --via agentcore. It is not decoration: if the MCP tools ever drift from what the pages show (a different filter, a lapsed attribution rule) the publish step breaks loudly instead of the two quietly disagreeing. Use --via direct to bypass it.

python scripts/publish_site.py --run week --replace                  # via local MCP
python scripts/publish_site.py --run week --via agentcore            # via hosted MCP

• Leaderboard — static page on CloudFront + S3, filterable by verdict and topic

• Atom feed — /feed.xml, filtered by topic, minimum RQS, author kind and age

• MCP server — nine tools: get_reading_queue, score_url, review_article, explain_score, search_scored, set_preferences, get_preferences, export_corpus, corpus_stats. Runs over stdio locally, or streamable HTTP on AgentCore Runtime for Quick Suite and any other MCP client.

• Chrome extension — MV3 content script that injects score badges onto builder.aws.com cards and article pages. Purely additive; the site itself is untouched.

python -m shouldiread.mcp_server            # stdio
python -m shouldiread.mcp_server --http     # :8000/mcp, AgentCore shape

See docs/connect-mcp.md for Quick Suite, Claude Desktop and raw SigV4 connection details.

Attribution policy

Articles scoring READ or SKIM are named and linked — naming good work is the point. Articles scoring SKIP publish their score and their reasons but never the author or title. The aggregate problem is worth being blunt about; individual community members are not the target. This is enforced in one place (publish._public) and covered by tests that assert redacted titles never reach the rendered HTML or the feed.

How content is read

All public, no scraping tricks and no login:

• /rss — Atom, 30 newest, hourly. The delta source.

• /sitemaps/articles/YYYY-M.xml — 55 monthly files back to 2022. Full backfill.

• api.builder.aws.com/cs/v2/articles?articleId=… with the header builder-session-token: dummy — the literal anonymous token builder.aws.com's own web app sends for logged-out readers. Returns raw markdown, which is far better for scoring than rendered HTML.

Article pages themselves are a 2.5 KB SPA shell with no server-rendered body, so HTML scraping yields nothing and no headless browser is used in production. Requests are rate limited and concurrency-capped; robots.txt disallows only /profile/.

Deploy

cd infra && cdk deploy                      # S3, CloudFront, API GW, Lambdas, DynamoDB
python infra/deploy_agentcore.py build      # push arm64 images
python infra/deploy_agentcore.py deploy     # scorer + MCP runtimes

Bedrock access note: on the account this was built against, every Anthropic inference profile returns ResourceNotFoundException, so scoring runs on Nova (nova-2-lite for triage, nova-pro for the judges). Swap the ids in config.py if your account differs.

Tests

pytest

Covers the tool belt against real-article fixtures (including the assertion that an invented s3.turbo_upload_object is flagged while ordinary non-AWS Python is not), the scoring caps, preference filtering, and the attribution policy.