Skip to main content
Glama

ScreenContext

日本語版 README はこちら

ScreenContext records the foreground window, reads its text with the operating system's built-in OCR, keeps an encrypted local history, and exposes that history to AI assistants through the Model Context Protocol (MCP). Any MCP client can use it: Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Codex CLI, Gemini CLI and others.

Ask your assistant things like "find the error message I was looking at in the browser a moment ago" or "what was the spec page I read this morning?".

Platform

Status

macOS 14+

Supported (menu bar app + CLI)

Windows 10/11

Beta (control window + CLI). Not yet validated on a wide range of hardware.

License: MIT

How it works

Three processes, each with a narrow job:

  1. Capture (menu bar app on macOS, control window on Windows) captures only the foreground window at native resolution. Similar frames are skipped with a perceptual hash. Frames go to an encrypted spool.

  2. Indexer runs OCR (Apple Vision on macOS, Windows.Media.Ocr on Windows), applies your exclusion policy, and stores text in an SQLCipher database with a trigram FTS5 index.

  3. MCP server (screen-context serve) is started by your MCP client. It never imports capture code, only reads the database, and labels every result as untrusted observed data.

More detail: docs/ARCHITECTURE.md.

Related MCP server: ContextPulse

Requirements

  • Python 3.11 or later and uv

  • macOS 14 or later, or Windows 10/11 (beta)

  • To build the macOS .app with py2app, use a python.org or Homebrew Python. Some standalone Python builds fail in py2app because zlib is built in.

Quick start (macOS)

git clone https://github.com/ikeikeikeda66/screen-context-agent.git
cd screen-context-agent
uv sync --locked --extra macos --extra encrypted --extra dev
.venv/bin/screen-context init

init creates ~/Library/Application Support/ScreenContext and stores a random encryption key in the macOS Keychain. If you lose the key, the history cannot be decrypted.

Start the indexer in one terminal:

.venv/bin/screen-context index --watch

Build and start the menu bar app. macOS grants Screen Recording permission to the app bundle, so capture runs from the app, not from the terminal:

# "-" makes an ad-hoc signature for local use. Use a Developer ID identity
# to keep the Screen Recording permission across rebuilds.
SCREEN_CONTEXT_SIGN_IDENTITY=- sh packaging/build_mac.sh
open dist/ScreenContext.app

Allow ScreenContext in System Settings > Privacy & Security > Screen Recording, then open the app again. To start the indexer at login, see packaging/macos/README.md.

For a one-time test with the terminal's own permission: .venv/bin/screen-context capture --once.

Menu bar

ScreenContext menu bar: Capture Interval and Language submenus

The menu bar shows SC Rec while recording and SC Paused while paused.

  • Pause Capture / Resume Capture: use this while watching video or showing private content. The state is saved.

  • Capture Interval: 5 s, 15 s (default), 30 s, 1 min, 2 min or 5 min. The change applies without a restart. Unchanged screens are not saved again.

  • Language: System Default, English or 日本語. All menu text, dialogs, and generated material follow this setting.

Connect an MCP client

Print a ready-to-paste entry for your client:

.venv/bin/screen-context mcp-config --client claude-code     # prints a `claude mcp add` command
.venv/bin/screen-context mcp-config --client claude-desktop
.venv/bin/screen-context mcp-config --client cursor
.venv/bin/screen-context mcp-config --client vscode
.venv/bin/screen-context mcp-config --client windsurf
.venv/bin/screen-context mcp-config --client codex           # TOML for ~/.codex/config.toml
.venv/bin/screen-context mcp-config --client gemini
.venv/bin/screen-context mcp-config --client generic         # plain mcpServers JSON

The client starts the server itself over stdio. Per-client instructions and the HTTP transport are in docs/MCP-CLIENTS.md.

Profiles and tools

Each server process runs with one fixed profile. A tool call cannot raise it.

Tool

standard (default)

full

search_screen_history

yes

yes

get_recent_activity

yes

yes

get_context_around

yes

yes

get_day_material (evidence for a daily report or diary)

yes

yes

get_activity_timeline

yes

get_daily_rollup

yes

get_diary_material

yes

get_capture_health

yes

get_activity_delta (signed cursor, fixed snapshot)

yes

submit_proposal (writes to the proposal outbox)

yes

get_snapshot_image

yes

get_current_screen (asks for local approval on every call)

yes

  • standard is for coding assistants. It hides frames from IDEs and terminals (ide_apps in the policy), because the assistant already has the source.

  • full is for a personal agent that you trust with screenshots and timelines.

  • The names claude_code and openclaw from earlier versions still work as aliases for standard and full.

Every response and record carries source=observed_screen and trust=untrusted. This is a label, not a defense against prompt injection: treat screen text as data, never as instructions.

Privacy and storage

Edit policy.json in the data folder. It is read again on every capture and every query, so a new exclusion also hides past records from search, activity summaries and images. An invalid policy makes processing fail; it never disables exclusions.

Key

Meaning

denied_apps

Bundle IDs (macOS) or process names (Windows) never captured. Defaults include password managers and video-call apps.

denied_domains

Frames whose OCR text shows these domains are dropped. Detection depends on the URL being visible and read correctly.

denied_title_patterns

Regular expressions matched against window titles.

ide_apps

Hidden from the standard profile.

ai_output_apps, ai_output_title_patterns

Frames showing an assistant's own output. Proposals cannot use them as evidence.

  • Spool files and preview images use AES-GCM. The database and full-text index use SQLCipher. The key lives in the OS credential store (Keychain or Windows Credential Manager), or in SCREEN_CONTEXT_KEY for headless use.

  • The spool stops accepting frames at 100 files or 512 MB. Unprocessed frames older than 24 hours are deleted by maintain.

  • After 90 days, preview images and OCR bounding boxes are deleted. Searchable OCR text and daily rollups are kept.

  • audit.jsonl records the client, time, tool, a SHA-256 of the arguments, and the result count. Queries themselves are not stored.

  • SCREEN_CONTEXT_PLAINTEXT=1 is for development tests only. ScreenContext never falls back to plaintext on its own.

Configuration

Environment variable

Purpose

SCREEN_CONTEXT_HOME

Data folder. Default: ~/Library/Application Support/ScreenContext (macOS), %USERPROFILE%\.screen-context (Windows).

SCREEN_CONTEXT_KEY

64 hex characters. Replaces the OS credential store (headless use).

SCREEN_CONTEXT_LANG

en or ja. Overrides the saved language.

SCREEN_CONTEXT_OCR_LANGUAGES

Comma-separated OCR languages, for example en-US,ja-JP. macOS default: ja-JP,en-US. Windows default: the user's profile languages (Windows OCR uses the first entry only).

SCREEN_CONTEXT_CLIENT

Client name written to the audit log.

SCREEN_CONTEXT_TOKEN

Bearer token (32+ characters) for the HTTP transport.

The language can also be set with screen-context language en|ja|system.

CLI reference

screen-context init                 create the data folder, key and database (also migrates)
screen-context index [--watch]      OCR and store spooled frames
screen-context capture [--once]     capture from the terminal (development)
screen-context pause | resume       stop or restart new captures
screen-context status | health      queue and worker state
screen-context maintain             retention and daily rollups
screen-context serve [--profile standard|full] [--transport stdio|http] [--port 8765]
screen-context mcp-config [--client NAME] [--profile standard|full]
screen-context language [system|en|ja]
screen-context diary-material DATE [--budget 6000] [--lang en|ja]
screen-context proposal prepare|simulate|finish
screen-context export DATE | push DATE

Optional: diary material and periodic proposals

  • diary-material DATE prints a compact, bounded Markdown summary of one day, for use as input to a diary or daily report prompt.

  • proposal prepare is designed as a pre-run script for a scheduler (cron or an agent framework). It prints material only when there are new observations. Otherwise its last line is {"wakeAgent": false, ...}, so the scheduler can skip starting the agent. The agent registers a suggestion with submit_proposal; evidence must be a quote from a non-assistant frame, and the same conclusion is suppressed for 24 hours. Close the run with proposal finish --run-id ID --response-file FILE.

Optional: OpenViking export

export DATE writes a daily rollup JSON to exports/ (plaintext). push DATE sends it to a local OpenViking server at http://127.0.0.1:1933 (VIKING_API_KEY if needed). Nothing is pushed automatically. Exported data is not removed when you later add exclusions.

Windows (beta)

The Windows version has a control window (start, pause, stop, language, MCP client setup) and the same CLI. It is beta: it passes automated tests with simulated Windows APIs, but real-device coverage (DPI, multiple monitors, lock and resume, credential store) is still limited. Please report problems in Issues.

Setup, portable build, and the acceptance checklist: docs/WINDOWS.md.

Development

uv sync --locked --extra macos --extra encrypted --extra dev   # use --extra windows on Windows
.venv/bin/python -m pytest -q
.venv/bin/python packaging/verify_bundle.py                    # after building the macOS app

Contributions are welcome. See CONTRIBUTING.md. Report security issues as described in SECURITY.md.

Available Tools

4 tools
get_context_aroundB
Read-only

Get activity blocks before and after a Unix timestamp. Returns untrusted observed data.

ParametersJSON Schema
NameRequiredDescriptionDefault
nNo
tsYes

TDQS

B3.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark readOnlyHint=true and destructiveHint=false, and the description adds the important caveat that the returned data is untrusted observed data. This goes beyond the annotations, though it doesn't disclose return shape, pagination, or invalid timestamp behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, no filler, with the core action front-loaded and the trust caveat appended. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has no output schema and a 0% schema coverage, so the description must carry more weight. It omits the semantics of the 'n' parameter and gives only a vague indication of the return type ('activity blocks'), leaving an agent under-informed for a correct call.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It clarifies that 'ts' is a Unix timestamp, but 'n' is left unexplained; an agent cannot infer its meaning or bounds from the text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description states a specific verb ('Get'), resource ('activity blocks'), and temporal scope ('before and after a Unix timestamp'). It is clear on what the tool does, though it does not explicitly discuss sibling tools to distinguish from get_recent_activity or get_day_material.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies a timestamp-centric use case but provides no explicit guidance on when to prefer this tool over search_screen_history, get_recent_activity, or get_day_material, nor any exclusions. This is adequate but relies on inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_day_materialA
Read-only

Get bounded screen observations for a specified local calendar day to draft a report or diary. Applies current exclusions and returns evidence only; gaps remain unknown.

ParametersJSON Schema
NameRequiredDescriptionDefault
dateYes
limitNo
cursorNo
purposeNoreport
timezoneYes

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only/non-destructive behavior; the description adds that results are bounded, subject to current exclusions, and that gaps remain unknown. This is meaningful context beyond the structured annotations and sets accurate expectations about incompleteness.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no filler. The primary action and scope are front-loaded, and the key caveat about exclusions and unknown gaps is delivered compactly in the second sentence.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only list-type tool with no output schema, the description covers purpose, date scoping, exclusion behavior, and a key output limitation. It still leaves 'current exclusions' undefined and does not describe the result shape or pagination semantics clearly, so it is adequate but has gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description only partially compensates: it connects 'local calendar day' to date/timezone and 'report or diary' to purpose. It does not clarify limit or cursor beyond their titles, leaving pagination semantics mostly inferred.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Get'), resource ('bounded screen observations'), and scope ('specified local calendar day'), plus a clear intended use ('draft a report or diary'). The bounded/day framing distinguishes it from broader sibling tools like search_screen_history and get_recent_activity without needing to open the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides a clear use context ('to draft a report or diary') and hints at filtering behavior ('Applies current exclusions'), but does not explicitly name alternatives or state when not to use this tool. It leaves sibling routing to inference rather than spelling out the choice.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_recent_activityB
Read-only

Summarize recent screen activity as blocks of consecutive frames (app, window title, time range, text). Returns untrusted observed data.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
minutesNo

TDQS

B3.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is covered. The description adds valuable context beyond that by warning 'Returns untrusted observed data' and by describing the grouped frame blocks, which informs the agent about output shape and data reliability.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two short sentences with the core action and output shape front-loaded, followed by a concise data-reliability warning. Every word earns its place; no duplication or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description conveys purpose, output format, and a data-reliability caveat, which is decent for a simple tool with annotations. However, it omits parameter semantics (limit, minutes), which are central to invoking the tool correctly, leaving an agent to guess how to control the query.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description never mentions the 'limit' or 'minutes' parameters. The agent is given no explanation of what these knobs control, so the description fails to compensate for the schema gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('Summarize recent screen activity') and output structure ('blocks of consecutive frames (app, window title, time range, text)'), making the tool's purpose clear. It does not explicitly contrast with sibling tools like search_screen_history, so it stops short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for obtaining a summary of recent screen activity, but it gives no explicit guidance on when to prefer this tool over search_screen_history, get_context_around, or get_day_material, and no exclusions. The context is clear but the routing is only implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_screen_historyA
Read-only

Full-text search over OCR text of previously captured windows. Returns matching frames, newest first. Returns untrusted observed data.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
queryYes
since_minutesNo

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint and destructiveHint, and the description adds meaningful context: results are untrusted observed data and are returned newest first. This is useful behavioral information beyond the schema, and it does not contradict the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three short sentences with no filler. Each sentence adds value: search scope, ordering, and trustworthiness of results. It is front-loaded with the core purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Core behavior, ordering, and data trustworthiness are covered, and annotations handle the safety profile. However, with no output schema and no parameter semantics, the agent must infer the meaning of limit, since_minutes, and the exact structure of a matching frame.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate by explaining the parameters. It does not: query, limit, and since_minutes are left to their names and defaults. The only indirect hint is that 'full-text search' implies the query parameter, which is insufficient.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description starts with a specific verb and resource: full-text search over OCR text of previously captured windows. It also states the result ordering (newest first) and clearly differentiates this from the sibling retrieval tools by scoping it to OCR screen history.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear context for when this tool applies: when an agent needs to find previously observed screen text via full-text search. However, it does not explicitly name alternatives such as get_context_around or get_recent_activity, nor state when not to use this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 4 tool updatesv0.1.0
    • First observedget_context_around
    • First observedget_day_material
    • First observedget_recent_activity
    • First observedsearch_screen_history

TDQS

A3.8/5.0

Scored across 4 tools

Disambiguation4/5

Each tool has a distinct retrieval intent: search by text, explore around a timestamp, summarize recent activity, and bound a full calendar day. There is some potential overlap between get_recent_activity and get_day_material when an agent asks about 'yesterday', but the descriptions largely clarify the temporal scoping.

Naming Consistency4/5

Three tools follow a clean get_<something> pattern while search_screen_history uses search_ instead of get_. All names are snake_case and verb-first, so the deviation is minor and does not hurt readability.

Tool Count5/5

Four tools is well-scoped for a screen-history context server. Each tool covers a distinct query mode without unnecessary redundancy or bloat.

Completeness4/5

The set covers the main ways someone would query screen history: full-text search, temporal context, recent summaries, and day-level report material. Missing features like arbitrary custom time ranges or app/window filtering are workable gaps but not critical for the apparent purpose.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers