Clippy Vision
Clippy Vision's MCP server lets an agent query your locally captured screen/activity history and manage personal memory, notes, and workspace file lookups.
Search activity history: ask natural-language questions over session summaries or granular events (OCR text, messages, URLs, clipboard, app usage).
Bounded keyword search: search events or sessions with explicit start/end dates and pagination (
limit/offset).Timeline listing: get a deduped chronological session list for a day or week.
Coverage & time analytics: see hourly event-count buckets to detect capture gaps, and approximate per-app foreground time.
URLs & screenshots: list distinct URLs seen, list screenshots (with expiry info), and fetch a specific screenshot's redacted screen text.
Long-term memory recall: list memory clusters by freshness or run semantic recall by query, then fetch all facts in a cluster.
Save & manage memory: store identity facts, add/remove list items, override facts, remember durable facts from the user's words, and save or delete notes.
File discovery: list trusted workspace roots and search them for filenames or substrings.
Clippy Vision
Clippy Vision watches your work on this computer, stores a searchable activity history locally, and lets you ask about it through a connected MCP client such as Cursor, Claude Desktop, or VS Code. The desktop app captures, summarizes, and shows the timeline.
Ask in a connected MCP client, for example:
"What was I working on before lunch?"
"Which hotels did I look at yesterday?"
"What was the error I encountered in my terminal?"
Download
Windows is tested for 2.0. macOS validation is pending, so macOS 2.0 installers are not included yet. macOS downloads below are the previous release (v1.3.1). Linux is experimental and X11 only; an unsigned x86_64 AppImage is built in CI (see docs/linux.md). The README describes current main; some installer features differ from 1.3.1.
The installers are unsigned: on Windows choose More info → Run anyway, and on macOS right-click → Open the first time. First run needs internet for dependencies and for the chat model you pick in setup (for example qwen3:8b is ~4.7 GB).
Trying Clippy? Share feedback (5 min).
Requirements
Minimum | Recommended | |
OS | Windows 10 / 11 (64-bit), macOS 12+, or Linux (Ubuntu 22.04+, experimental, X11 only) | Windows 11 or a current macOS |
System RAM | 8 GB | 16 GB |
Graphics memory | Not required. Integrated graphics or Apple unified memory is enough | 4 GB+ dedicated on Windows, or 16 GB unified memory on Apple silicon |
Free disk | 8 GB | 10 GB+ |
Python runtime | System Python 3.11+ on PATH (not bundled). Windows auto-install uses 3.11; CI uses 3.12 | Python 3.12 |
Python is not bundled in the Windows installer or Linux AppImage. requirements.txt does not pin a Python version. The packaged app runs CLIPPY_PYTHON if set, otherwise python3 (Linux/macOS) or python (Windows) from PATH (electron-ui/electron/lib/paths.js). CI unit tests and the release dry-run install Python 3.12. The setup wizard parses python --version and requires Python 3.11+.
When Python is missing, the setup wizard (electron-ui/electron/lib/setup.js) does this:
It parses
--versionon that interpreter and requires Python 3.11+. Preflight text says "Clippy needs Python 3.11+ on PATH." It then re-resolves known install locations before declaring failure.Windows: tries
winget install Python.Python.3.11 --silent. If winget is missing or the install cannot be found on PATH, it shows python.org/downloads and probes%LOCALAPPDATA%\Programs\Python\Python311\python.exe.macOS: tries Homebrew
brew install pythonwhen Homebrew is present. If Homebrew is missing, it shows a manual-install message with the python.org link.Linux: stays a manual install. The wizard does not try Homebrew or apt; it shows a python.org link.
Advanced: point Clippy at a specific interpreter before launch:
$env:CLIPPY_PYTHON = "C:\path\to\python.exe"export CLIPPY_PYTHON=/path/to/venv/bin/python3Linux support is experimental and X11 only. Wayland is not supported. On X11, screenshots use mss (optional crop via xdotool). Window titles use xdotool, clipboard uses wl-paste/xclip/xsel. AT-SPI accessibility text is implemented; it is covered by CI unit tests, not a real desktop. An unsigned Linux AppImage is built in CI. Install and run steps: docs/linux.md.
The setup wizard checks these numbers before installing. Below minimum → setup is blocked. Between minimum and recommended → you can continue with a warning that summaries may feel slower. Homebrew, dependency, and permission troubleshooting: docs/usage.md. Lower-spec machines: CONTRIBUTING.md.
Related MCP server: macos-screen-mcp
Quick Start
Download and install Clippy.
Complete the setup wizard and grant required permissions.
Leave Clippy running to build your activity history.
Open Settings → Connect apps, connect your agent, and ask about your activity.
More detail: docs/usage.md · docs/mcp.md · client guides: Claude Desktop · LM Studio
Build from source: CONTRIBUTING.md
Features
Searchable activity history across apps
Local summaries and long-term memory
Access through MCP-compatible agents
A timeline for reviewing and deleting activity
Capture and privacy controls
Privacy
Clippy captures foreground windows, clipboard, typing metrics, and screenshots.
Capture, summarization, and storage run locally on this computer.
Connected cloud agents can receive retrieved memory through MCP tool replies.
Connected clients have access to all exposed Clippy tools.
Redaction is best effort. Private-browsing detection is on by default: Chrome, Edge, and Brave on Windows and macOS; Firefox and Opera on Windows (and attempted but untested on macOS); Vivaldi on Windows. Vivaldi, Safari, and Linux are not covered. See the coverage table.
You can pause capture and delete stored activity from the tray or timeline.
Full guide (retention defaults and detailed behavior): docs/usage.md.
Documentation & Roadmap
Guide | Contents |
Capture controls, privacy, accessibility, install troubleshooting | |
Experimental Linux install and run (X11 only; AppImage, Python, capture tools) | |
Supported clients, Copy JSON, tool reference, access permissions | |
Step-by-step Claude Desktop setup: config location, restart, verification | |
Step-by-step LM Studio setup: MCP configuration, local model, test query, privacy | |
Tech stack, capture pipeline, summarization, distillation, schema | |
Build from source and contribution guide | |
Roadmap, principles, and known limitations |
Contributing
See CONTRIBUTING.md for setup steps, building from source, and good first issues.
Contributors
Thanks to everyone who has shipped code, docs, design, or ideas.
Contributor | What they built | |
@protocorn💻 📖 🎨 🤔 🚧 | Designed the core app: agent, vision pipeline, memory system, and the Electron desktop shell. | |
@rusetiq💻 📦 | Brought Clippy Vision to macOS: native screen capture, permissions, and Apple Silicon + Intel packaging. | |
@cursoragent💻 | See their commits → | |
@Afrin-banu-A💻 | See their commits → | |
@ABarpanda💻 | See their commits → | |
@vitorparras💻 | See their commits → | |
@Furox-Art💻 | See their commits → | |
@vaishn4vi💻 | See their commits → | |
@adity982💻 | See their commits → | |
@Draoui-Haroun💻 | See their commits → | |
@KeuqS💻 | See their commits → | |
@shaurya703💻 | See their commits → | |
@cyforkk💻 | Made errors readable: replaced bare HTTP status codes with real API error messages in chat. | |
@tadaokatzuki💻 | See their commits → | |
@Buzepool💻 | See their commits → | |
@BitsRevenge💻 | See their commits → | |
@icn5381💻 | See their commits → |
How to get on this wall: CONTRIBUTING.md.
License
AGPL-3.0. See LICENSE. Releases already published through v1.3.1 stay under MIT. A commercial license for companies that cannot use AGPL is not written yet.
Available Tools
17 toolsactivity_coverage_toolB
Hourly (or custom) event-count buckets so capture gaps are honest. Empty buckets mean no events that hour — often capture was off. start required (epoch or YYYY-MM-DD); end defaults to now.
Inputs: start (required), end, bucket_hours. Returns: result
app_time_summary_toolB
Approximate per-app foreground time from event gaps in a window. start/end: epoch or YYYY-MM-DD. Durations are approximate (gaps capped at 5 min).
Inputs: start (required), end. Returns: result
delete_note_toolA
Delete a note or memory fact the user wants forgotten. Use when the user says 'forget', 'delete', 'remove', or 'don't remember that'. Matches by substring — pass the key phrase or exact text from the note.
Inputs: note_text (required). Returns: result
fetch_cluster_toolA
Get all facts stored in a named memory cluster (with freshness annotations). Use after recall_memory_tool to get the full content of a specific topic. Pass the cluster label exactly as returned by recall_memory_tool.
Inputs: label (required). Returns: result
find_files_toolC
Search trusted workspace roots for a filename or substring.
Inputs: name (required), limit, under. Returns: result
get_screenshot_toolB
Fetch one screenshot by filename or unix timestamp. Returns redacted screen text. The image file path is not included. Screenshots use adaptive TTL (important frames kept longer, capped).
Inputs: filename, timestamp. Returns: result
list_screenshots_toolB
List screenshots (or OCR-backed stubs) in a time window. image_available=false means the JPEG expired; OCR preview may still be present.
Inputs: end, limit, start. Returns: result
list_sessions_toolA
Chronological deduped session list for a time window. Prefer this over search_sessions_tool for day/week timelines. start/end: epoch or YYYY-MM-DD.
Inputs: end, limit, start, offset. Returns: result
list_urls_toolB
Distinct URLs seen in active_url / OCR within a time window. pattern filters substring (e.g. 'greenhouse'). Deduped with counts.
Inputs: end, limit, start, pattern. Returns: result
list_workspace_roots_toolA
List folders Clippy may search with find_files_tool.
Inputs: no named parameters. Returns: result
recall_memory_toolA
List long-term memory clusters ranked by freshness score. Pass query to run semantic recall instead of listing. Clusters below min_freshness are hidden unless include_stale=true.
Inputs: query, include_stale, min_freshness. Returns: result
remember_turn_toolA
Store durable facts from the user's own words. Call after the user tells you something about themselves, their preferences, relationships, plans, or anything they would want recalled later. Pass only the user's message, not your reply. Skip for activity questions, file search, and one-off tasks. Secrets, passwords, and API keys are not stored.
Inputs: user_message (required). Returns: result
save_identity_toolA
Save a personal fact about the user. op='set' for scalar facts (name, location, job). op='add_items' with items=[] for adding to a list (hobbies, skills). op='override' only when the user explicitly corrects a previous fact. op='remove_items' with items=[] to remove from a list.
Inputs: op (required), field (required), items, value. Returns: result
save_note_toolB
Save a free-form note or reminder the user wants remembered.
Inputs: note (required). Returns: result
search_bounded_toolA
Keyword search with explicit time bounds (no LLM date guessing). start/end: epoch seconds or YYYY-MM-DD (local). table: 'events' or 'sessions'. Use when you know the calendar window and need pagination via limit/offset.
Inputs: question (required), end, limit, start, +2 more. Returns: result
search_events_toolA
Search individual events in the activity database. Use for: specific messages, OCR screen text, exact URLs, clipboard content, app usage, WhatsApp/email content, fine-grained timestamps, copy-paste history. Returns raw event rows with screen/OCR data. If the result says the info isn't there, call search_sessions_tool next. For exact date ranges prefer search_bounded_tool.
Inputs: question (required). Returns: result
search_sessions_toolA
Search session summaries in the activity database. Use for: broad time windows (yesterday, this week), daily/weekly overviews, what-did-I-work-on questions, project topics, task recaps. Returns paragraph summaries — NOT granular event detail. If the result says the info isn't there, call search_events_tool next. For exact date ranges prefer list_sessions_tool or search_bounded_tool.
Inputs: question (required). Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 17 tools
Several tools have overlapping search purposes: search_sessions_tool, search_events_tool, search_bounded_tool, and list_sessions_tool all query the activity database, and descriptions only partially disambiguate (e.g. 'if result says not there, call the other one'). Memory tools recall_memory_tool, fetch_cluster_tool, save_identity_tool, remember_turn_tool, save_note_tool, and delete_note_tool overlap around memory storage/retrieval.
All 17 tools follow a consistent snake_case verb_noun or noun_verb pattern (get_screenshot_tool, recall_memory_tool, find_files_tool, list_urls_tool), with the uniform '_tool' suffix making the set predictably uniform.
17 tools is on the heavy side for a screen/memory assistant, and several pairs (search_sessions vs list_sessions, search_events vs search_bounded) could likely be collapsed without losing capability.
The surface covers capture (screenshots), memory storage/retrieval, note lifecycle (save/delete), file search, and rich activity querying with coverage/app-time/URL listing. Minor gaps like note editing/updating or explicit screenshot deletion exist but agents can work around them.
Maintenance
Related MCP Connectors
Self-hosted AI workspace: semantic RAG document search, long-term memory, and AI chat via MCP.
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
Shared memory MCP server for Claude Code, Codex, Gemini CLI and Cursor. Cited, editable.
Shared memory for your AI tools: tasks, notes and context every tool can read and write over MCP.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceGUI automation MCP server that enables AI agents to see and control the Windows desktop using a local Vision LLM (Ollama), supporting screenshot analysis, mouse/keyboard actions, and autonomous task execution.5MIT
- AlicenseAqualityDmaintenanceAn MCP server that lets AI assistants see your macOS desktop, capture screenshots, read browser tabs, and preview files via local macOS tools.48 npmMIT
- AlicenseNot gradedqualityDmaintenanceA comprehensive MCP server that gives AI assistants full control over your desktop — monitor system resources, manage windows, capture screenshots, control the clipboard, launch applications, and more.MIT
- AlicenseNot gradedqualityBmaintenanceAn MCP server that gives any AI assistant eyes and hands on your desktop — screenshots, clicking, typing, OCR, window management, accessibility-tree queries, workflow recording.5Apache 2.0