Carl

Carl is a local evidence store and work queue for scraping and reviewing marketplace listings. It currently has Facebook Marketplace and eBay adapters. It retains acquisitions, extracted observations, images, and analysis reports, and assembles them into source-linked listing views.

The main interface is MCP. A client can request searches, queue follow-up work, inspect retained evidence, and maintain review workspaces. A separate worker process executes queued collection, extraction, and analysis. The CLI also provides acquisition, reprocessing, inspection, and monitoring commands.

Evidence and derivation

Carl separates acquiring evidence from interpreting it:

1. Acquisition sends HTTP requests through a configured transport and network route. It retains response metadata and collected content, including failed or challenged responses where available. A search result, an item page, and an image request are separate acquisitions.

2. Extraction turns retained content into versioned records: search occurrences, listing observations, gallery references, and other source-specific fields. Extraction can run again on saved evidence without fetching the source again.

3. Composition selects usable evidence for a listing's current read model. It does not rewrite the acquisitions or observations that supplied the values.

4. Analysis, when requested, gives Claude an exact retained evidence selection and a versioned product guide. Its report identifies the item, compares seller claims with visible evidence and researched specifications, describes condition and missing parts, and records uncertainty.

Records and artifacts are linked by named operation inputs and outputs. Operations retain the component and output-schema version, code and runtime provenance, configuration, and execution status. Later observations and derivations add history rather than replace earlier evidence. Identical content may share storage without collapsing the separate acquisition histories.

Metadata, JSON records, and ordinary response bodies live in SQLite. Validated images use database-relative, content-addressed files. Retained response bodies are collected, decoded content, not packet captures; transport redaction, response limits, and incomplete downloads affect what is available. The database and image files together form the evidence store.

Related MCP server: Marketplace Finder MCP Server

Composed listing views

Carl calls the final read model a composed projection. Its components are selected independently from layered evidence; they need not have been observed at the same time.

• Descriptive fields prefer usable item-page evidence, with search cards filling gaps.

• Price uses the newest usable price observation from a search card or item page. A newly observed card price can therefore coexist with an older description.

• Status comes from explicit retained status evidence. Absence from a search is not evidence that an item sold or became unavailable.

• The gallery uses a coherent retained reference set, not an arbitrary union of photos from different observations. A search thumbnail is not a collected gallery.

• Completed analyses retain their own input observation and guide version. An older analysis can remain visible with an applicability label; that does not mean it analyzed the newest evidence.

• Search membership adds first/last-seen and refresh-lineage information for the selected context.

Selected components carry evidence references and observation/completion times. Missing values do not erase older usable evidence, and re-extracting an old response does not make its acquisition fresh. Reading a projection performs no acquisition or AI analysis.

Workspaces combine related search tracks, including tracks from both marketplaces, into a deduplicated listing set. Tracks have stable identities and versioned search specifications; revising a watch preserves its history. Review state compares normalized, review-relevant component values, so an identical re-observation does not by itself make a review stale. Worksets and selection snapshots preserve explicit groups for later review or analysis.

Scraping and follow-up work

Acquisition uses HTTP transports, including wreq browser-profile emulation; it does not launch a Chrome browser. Network routes and acquisition stacks select the configured proxy/VPN provider and transport. Routes can differ between search, item pages, and images. The currently supported setup and route defaults are documented in the setup and usage guide.

Search-only requests retain search results. They do not automatically fetch every item page or download every photo. An explicit search pipeline can request bounded follow-up through details, images, or analysis. Work progresses per listing: one listing can become ready for analysis while another is still collecting images. Fixed-selection analysis can also operate on retained workspace evidence after previewing the selection.

Overlapping requests can share acquisition work, and follow-ups reuse usable retained item pages and validated matching images. This is evidence reuse, not remote unchanged-page detection. An ordinary search refresh may reuse item details without checking whether the remote item page changed; explicit item-detail refresh is a separate request. Reused evidence keeps its original age.

Running Carl

From a checkout, with uv and Python 3.13 or newer:

uv sync --locked --all-groups
uv run --locked carl locations
uv run --locked carl init

locations reports the per-user configuration, database, image, and diagnostic paths. init creates or migrates the evidence database. It does not generate network configuration or install credentials. Configure the required routes before requesting live collection; install and authenticate Claude CLI in the worker's environment before requesting analysis. See the setup and usage guide for configuration, credentials, database selection, MCP client launch, and upgrade instructions.

The processes have separate roles:

Normally, keep one worker process running and let the MCP client launch its server separately. Both must use the same evidence database and per-user configuration. Multiple MCP connections do not create additional worker pools. A queued work ID acknowledges a request, not completion; inspect root and child outcomes before treating collection or analysis as successful.

Workers lease persisted work, apply scheduling constraints, and retain checkpoints and attempt history. Temporary failures have bounded retries; detected Internet outages pause acquisition without consuming outage retry budgets. Retained reads remain useful without a worker. Recovering terminal work is explicit, and downstream recovery can reuse a completed search rather than rerun it.

The agent skill describes the MCP workflow and evidence/identifier conventions. The server also publishes tool schemas, instructions, identity, and capability versions for connected clients. Unexpected MCP failures return a redacted diagnostic and are logged locally; handler errors retain structured validation details and the latest durable checkpoint.

Limits and interpretation

Searches are bounded observations, not exhaustive marketplace inventories. Anonymous acquisition can encounter challenges, rate limits, layout changes, or incomplete responses. Facebook's search radius is not a reliable hard geographic filter. New eBay sold/completed searches are currently disabled because the acquisition path does not support their sign-in/challenge gating; retained closed-listing evidence remains readable.

An absent listing, an empty result, an idle queue, and successful collection are different states. Read the response classifications, coverage warnings, budgets, and child failure counts. Incomplete galleries require an explicit analysis override and remain marked incomplete.

Product guides currently contain identification instructions, not typed field schemas. Analysis reports are prose, and external research references are reported by the analysis agent rather than independently archived source pages. Seller claims, visible facts, researched specifications, and later judgments about value should remain distinguishable. General SQL queries, typed analysis fields, and saved queries are planned work, not current interfaces.

Development and design

mise run check
mise run pre-commit

The implementation separates sans-I/O parsing and models in carl.core, effect adapters in carl.io, and application orchestration. Async work uses AnyIO with Trio. See the architecture notes, composed projection design, and review workspace design for implementation details and explicitly marked future work.

Carl is under active development and is available under the MIT or Apache-2.0 license.