Agent Pointed Context

Point your own AI agent at what you see in an app, without sharing the screen.

Try it: apc.tineira.com is the connection chooser and opens no sandbox. Choose V1 for the original landing, which also opens no sandbox, or V2 to connect through your dock. Entering the V1 demo creates or resumes a private sandbox with synthetic data, with no sign-up; it is deleted after 6 hours without use. The how-to takes about five minutes with Claude Code, claude.ai, ChatGPT, Grok, Cursor or any agent that accepts a remote MCP server by URL.

This repo is the open reference implementation and demo of Agent Pointed Context (APC):

T. Neira. Agent Pointed Context: letting a person point an AI agent at what they see, without sharing the screen. Zenodo, 2026. https://doi.org/10.5281/zenodo.23382909

T. Neira. Agent Pointed Context: design records and payload specification. Zenodo, 2026. https://doi.org/10.5281/zenodo.23383101

The idea

An app exposes three small channels to any agent connected over the Model Context Protocol, and they carry identifiers and keys only, never pixels, text or amounts:

1. View: the screen the person is on, as identifiers (page, month, filters, at most one selected row).

2. Pin: the person drags their agent's mark onto a row, a cell or one step of a computed figure. The pin carries a key and the exact tool call that reads it (next, next_args). It reaches the agent on its next tool result, with no polling by the model, or right away through wait_for_focus.

3. Presence: each agent is a mark on screen and lights up on what it reads.

The agent reads what a pin points at through its ordinary tools, under the access it already has. And because it is your own agent, it already knows things the app does not: your files, your other tools, your memory. In the demo, Pulse (a fictional SaaS analytics app) knows customers and seats but not revenue; you give your agent a file of MRR per customer, point at a bar, and ask how much MRR it is. Your agent matches the customer ids behind the bar with your file and answers. Pulse never receives an MRR value.

Related MCP server: Little Mouse Pointer

What the demo shows

Measurements (load test up to 150 visitors at once, CPU per request) are in DESIGN.md §9b.

Use the pattern in your app

The packages are TypeScript sources in this workspace (not on npm yet). They are small, so copying them is fine too.

On the server, register the two APC tools next to your own and attach the envelope to every result while a pin is unread:

import { registerApcTools, attachAttention, attentionFor, isUnread } from "@apc/server";

registerApcTools(server, {
  readView: async () => myViewTrimmedTo(grant),          // identifiers only, cut to the caller's access
  readFocus: async () => myPinFor(grant),                 // { kind, id, part, params, next, next_args, ... }
  focusUnread: async () => isUnread(await myPinRow(grant)),
  markSeen: async () => stampSeen(grant),
  sleep: (ms) => new Promise((r) => setTimeout(r, ms)),
});

// in your tool wrapper, after each tool runs:
if (isUnread(row) && focus) result = attachAttention(result, attentionFor(focus));

In the page, mark what can be pointed at and let the person drag their agent's mark onto it:

import { focusAttrs } from "@apc/protocol";
import { enablePinning } from "@apc/web";

<g {...focusAttrs({ kind: "figure", params: { key: "active_customers", month: "2026-03" } })}>…</g>
<tr {...focusAttrs({ kind: "customer", id: "cus_0042" })}>…</tr>

enablePinning({ onPin: (grantId, target) => savePin(grantId, target) });

The demo is the full worked example: apps/demo/src/worker (the sandbox, the MCP endpoint, grant trimming, the figure registry) and apps/demo/src/web (marks, pins, presence).

The agent's manual (optional)

The server already tells every agent the essentials (its instructions, the tool descriptions, the attention and then keys). The full manual is opt-in, in two forms with the same text:

• MCP prompt guide, for one session: /mcp__pulse__guide in Claude Code, the + menu in Claude Desktop, or wherever your client lists MCP prompts.

• Skill pulse-agent, installed once for your CLI agent: npx skills add https://github.com/tineira/agent-pointed-context --skill pulse-agent (skills/pulse-agent/SKILL.md).

The text lives in one place (apcGuide() in packages/server/src/prompts.ts and Pulse's part in apps/demo/src/worker/guide.ts); node scripts/build-skill.mjs regenerates the skill file, and a test fails if the two drift.

Run it

Node 24 and pnpm through corepack:

corepack pnpm install
corepack pnpm typecheck
corepack pnpm test
corepack pnpm dev          # http://localhost:5173 (Vite + the local Workers runtime)

The public root / is a chooser: Connect app by app opens the original V1 landing at /v1; Connect once links to the separate v2 dock. The two cards have connection diagrams and a comparison table. Neither / nor /v1 opens a sandbox, even when you already have one. The V1 landing keeps its explanation, prerequisites and steps; its demo links create or resume your sandbox. /how-to and /overview keep their existing routes.

Deploy your own copy to Cloudflare Workers (the free plan is enough to try it):

cd apps/demo
npx wrangler login
# in wrangler.jsonc, remove the "routes" entry or point it at your own domain
node node_modules/vite/bin/vite.js build && npx wrangler deploy

apc.tineira.com itself deploys from CI: every push to main that passes the checks runs wrangler deploy (.github/workflows/ci.yml, with the repository secrets CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID).

On Windows the local Workers runtime needs the Microsoft Visual C++ Redistributable, and it sometimes crashes when a new sandbox starts; the page retries. Deployed, this does not happen.

Layout

What the hosted demo stores

Per sandbox, until it is deleted 6 hours after its last use: its seed, its agents' grants (with the client name and, if the client sent one, its small raster icon), your current view (identifiers only, and you can switch it off), pins, and an event log of tool calls including their arguments (so you can check what your agent sent). The MRR file is made in your browser and never sent. One count outlives the sandbox: when an agent connects (MCP initialize), the Worker writes a point to Cloudflare Analytics Engine with the client's name and version, the grant's label, its user agent, and two keyed hashes (of the sandbox and grant, and of the IP), never the IP, the sandbox id or the grant secret; Analytics Engine keeps it about three months. node scripts/mcp-sessions.mjs reads back distinct agent sessions per day and client. Cloudflare keeps its usual request logs for the Worker.

Status

Cut 1 is live; the canary check over the event log is node scripts/canary.mjs <events.jsonl> <mrr-by-customer.csv>. Open work is tracked in issues: the plan and seat screens, reusing the MCP server per agent, the client compatibility matrix across agents, per-client connection guides, and the evaluation harness (cut 2).

APC v2 (one agent across apps: a dock you drag your agent from onto any APC app) is developed in parallel in apps/dock, deployed on its own Worker at v2.apc.tineira.com, with a second deploy of Pulse at pulse.v2.apc.tineira.com, so apc.tineira.com is never touched; CI deploys v1 when anything outside apps/dock changes and v2 when apps/dock, apps/signals, apps/demo or the packages change. What v2 needs from the @apc/* packages goes in as additive, opt-in options that v1 does not turn on; a breaking change would version APC (0.3 / 1.0) with v1 pinned to 0.2. To try the dock prototype (TIN-370): open apc.tineira.com/overview?apc=v2, click Dock, and drag an agent from that window onto a bar in Pulse. Pulse and the dock talk over postMessage; the drag carries only the agent's public handle. To try APC v2: open v2.apc.tineira.com and click Open your dock. You only need an agent that can add a remote MCP server; nothing to install first. A checklist stays beside your agent and checks itself as you try each feature, in any order. Connect your agent once by dragging the plug into its chat, or using its MCP URL or QR. Open Pulse, an example APC app, from the checklist; drag your agent onto a bar, choose Allow in Pulse and then Allow in the dock (check the app and its address), and ask your agent “what is this?”. The dock creates a demo world silently, so Pulse and Signals open with matching made-up data; Change world is in the menu. Next open Signals, the second example app, drop your agent on a post, allow both sides and ask what explains the churn: no new MCP connection is needed, and your agent reads both apps and joins them while the apps share nothing. The checklist also offers the browser extension (download and install it step by step, without building anything): drop the agent on any paragraph, and its first page reading checks that item. Keep it at hand is optional: Float, install the dock as an app, the Windows dock with desktop pets, or Pulse for Windows. For Pulse for Windows, paste the dock’s Pulse link into the app, click Open world, then drag the agent from the dock onto a step, bar or customer. The dock stores, for 7 days after its last use, its agents (client name and icon), the apps each is paired with (with the MCP URL each app gave it), each app's last view, where each agent points, and each agent's last 300 calls through it (the tool, the app, the time, the pin it read, errors; TIN-419, shown on the agent's page under Details); never the calls' arguments or app results. With the browser extension (TIN-421), the dock also stores, for each agent, the browsers you allowed to send it page readings, and the last reading from each: the page's address (no query string), its title and the text you dropped the agent on, kept 10 minutes at most. That text passes through Cloudflare, where the dock runs; the dock deletes it on time, but Cloudflare's point-in-time recovery can restore a Durable Object's storage up to 30 days back, so a deleted reading stays recoverable there by whoever controls the account until then (the dock never does this). Each reading also leaves a line in a log of the agent's last 50, with the site, the time and the size only (TIN-422), listed with Clear in the extension's popup and on the agent's page; a reading's text is never logged. Like Pulse v1, when an agent connects to the dock (MCP initialize), the dock's Worker writes a point to the same Analytics Engine dataset, marked as the dock's (TIN-416): the client's name and version, the agent's label, its user agent, and two keyed hashes (of the dock and agent, and of the IP), never the IP, the dock id or the agent's secret. node scripts/mcp-sessions.mjs lists these sessions too, marked [dock].

To try the browser extension (TIN-421, TIN-465, DESIGN.md §13), follow the illustrated install steps: download the ready-to-use zip, unzip it into a folder you keep, and load that folder in Chrome or Edge. For developers building from source: clone this repo, run pnpm install, then node build.mjs in apps/extension, open chrome://extensions (or edge://extensions), turn on Developer mode, click Load unpacked and pick apps/extension/dist. A welcome page opens: click Let me drop my agent on any page (one grant; you can also allow sites one by one from the extension's icon instead), and choose when to be asked. Then drag your agent from the dock or Float onto a paragraph on any page. The first drop on a site asks on a card in the page: Just this once, Always on this site or Not here; government sites (.gob, .gov) and .bank and .insurance sites ask every time, and so does any site you set to Always ask from the extension's icon (your bank, email or health sites are not recognised by name: set them yourself), unless you turn on Never ask, which sends every drop at once. The first drop of an agent in this browser also asks in the dock to allow this browser. Then ask your agent about "this" and it reads what you dropped it on (page__get_reading). Your agent's mark stays pinned on that paragraph: orange until your agent reads it, green once it has, grey once you drop it somewhere newer; × removes it, and your agent forgets that reading. Drag the pin to another paragraph to point your agent there instead; Esc puts it back. The extension's icon also has Open dock (Alt+Shift+A) and Float (Alt+Shift+F), which opens your dock ready to float your agents with one click. The page's text goes to your agent through your dock and is untrusted: the agent is told never to follow instructions in it. To update the extension after a new build, click its reload button in chrome://extensions.

Security issues: please use GitHub's private vulnerability reporting (Security tab), not a public issue.

Credits

The look follows hue-switch-console by the same author, reused here under MIT. Its colour themes come from the Omarchy palettes (MIT). Fonts: Geist (SIL Open Font License).

Cite

This software: T. Neira. Agent Pointed Context: reference implementation and demo. Zenodo. https://doi.org/10.5281/zenodo.23404049 (all versions; v0.1.0 is https://doi.org/10.5281/zenodo.23404050). See also CITATION.cff. Please cite the paper for the pattern itself.

License

MIT for the code (see LICENSE). The paper and its companion record are CC BY 4.0.