seshat

An MCP server that records a screen and narrates the take.

seshat captures an output (or a region of one) into a silent artifact, ingests the timeline streams that other tool servers publish while the take runs, and then muxes a synthesized narration track — aligned to those real events, with styled captions burned in — over the recorded video. The prose is the caller's; the timing, the speech, the captions and the container are seshat's.

It is named after the Egyptian goddess of writing, measurement and record-keeping.

Sesh is the mascot: an ivory scribe-keeper crowned with Seshat's star and horns, the recording lens set in her chest, the goddess's notched measuring rod — the take timeline — in her hand, and the events other servers publish trailing from her headcloth. She keeps the book; she never invents a word of it.

What it is, and what it is not

• It is a recorder and a narrator. It owns capture, the take timeline, speech synthesis, caption layout and muxing.

• It is not a desktop controller. It never clicks, types, or focuses anything. The actions a take documents are performed by other tool servers; they reach seshat as published event streams. That split is deliberate: a recorder that also drives the desktop cannot record a desktop driven by anything else.

• It embeds no model. Narration prose is written by the calling agent and passed in. seshat owns timestamping, scheduling, synthesis and multiplexing: the same inputs, timeline and synthesized clips always produce the same schedule, and narration always rebuilds its deliverable from the take's finalized master, never from a previous narration's output.

Related MCP server: casefile-mcp

Requirements

• Python 3.10+ (standard library only — dependencies = [] is a hard contract).

• A live Sway or Hyprland session (Omarchy is Hyprland). The compositor is detected from the session environment and can be forced with SESHAT_COMPOSITOR=sway|hyprland; a missing IPC variable (SWAYSOCK, HYPRLAND_INSTANCE_SIGNATURE) is recovered from the runtime directory, which is how a harness-launched server still finds the desktop.

• wf-recorder for capture, ffmpeg + ffprobe for finalization, muxing and probing; hyprctl or swaymsg to resolve output geometry. Caption burn-in additionally needs an ffmpeg built with libass — the subtitles filter — which minimal builds ship without. A missing binary is reported as a clear tool error, never an import failure.

• Optional: piper (offline — the engine="auto" default needs a voice model) or edge-tts (network — runs only when engine="edge" is passed explicitly, because it sends the narration prose to Microsoft); tesseract for OCR of scene keys.

On Arch or Omarchy:

sudo pacman -S --needed wf-recorder ffmpeg tesseract   # hyprctl ships with Hyprland
uv tool install edge-tts                               # or piper + a voice model

make check runs the unit tests, bytecode compilation, and the file-length check. make help lists every target; make ci-local runs the whole gate plus a distribution build, and make runtime shows the streams, recordings and active take when something looks wrong.

Install

uv venv
uv pip install -e .
seshat --doctor     # what this host can record and narrate with
seshat --self-test  # non-mutating checks

Register with an MCP harness:

hermes mcp add seshat --command /path/to/venv/bin/seshat
codex mcp add seshat -- /path/to/venv/bin/seshat

Agent skill

skill/seshat/SKILL.md is the operating procedure for an agent driving this server: the take lifecycle, the anchor rules, and the failure modes that produce silent or unanchored takes. It ships in the source distribution, and the copy in this repository is the source of truth — installed copies are copies.

# opencode
mkdir -p ~/.config/opencode/skill
cp -r skill/seshat ~/.config/opencode/skill/

# Hermes
mkdir -p ~/.hermes/skills/media
cp -r skill/seshat ~/.hermes/skills/media/

Re-run the copy after changing the skill; a stale installed copy is how an agent ends up calling tools this server no longer has.

Recording a take

recording_start(format="mp4", max_duration_seconds=120)
   ... the demonstration happens ...
recording_stop()
recording_status()        # poll until phase == completed | failed
recording_timeline()      # the ingested events, with recording-relative t_ms

One take may be active at a time. Artifacts are written to $XDG_RUNTIME_DIR/seshat/recordings (0700, files 0600); the live capture intermediate lives there because tmpfs is fastest. A completed take is adopted into the durable store $XDG_STATE_HOME/seshat/takes/<id>/ (default ~/.local/state/seshat/takes/<id>/, 0700/0600) — master, timeline sidecar, manifest and every narration request — and is rehydrated by the next server session: takes survive restarts, and only recording_discard deletes them. recording_export is the only way take files leave the store:

recording_list()                       # every take this process knows
recording_export(destination="/path")  # copy master, deliverable, sidecars, manifest
recording_discard(id="...")            # delete one stored take (irreversible)

Takes are recorded silent by default. To capture desktop audio into the artifact, ask for it at start:

recording_start(format="mp4", audio="monitor")   # the desktop's own sound
recording_start(format="mp4", audio="mic")       # the host's default microphone

• audio="monitor" records the default output sink's monitor — application and system sound, never the microphone — resolved from pactl info and passed to wf-recorder -a <sink>.monitor. A host without a resolvable default sink fails with a clear error; seshat_info.audio reports what would be captured.

• audio="mic" records PulseAudio's default source (the default input device).

• audio="auto" is monitor where a default sink exists, mic otherwise.

• audio="off" (default, or null) records no audio.

• Audio is for mp4/webm; gif cannot carry audio and refuses the argument.

• Narration added later blends the synthesized voice with the captured sound in the artifact's single audio stream — see Narrating a take.

A completed take separates three lifecycle facts: capture_elapsed_seconds runs from recorder launch to the stop request, shutdown_latency_seconds measures how long the recorder then took to exit, and media_duration_seconds is the playable picture. termination_stage and recorder_returncode expose how the process ended. Capture is damage-independent, so static screen intervals still advance the video; latest_event_ms and events_beyond_media identify any remaining timeline/media mismatch rather than hiding it.

make check                         # unit tests, compilation, file-length ceiling
SESHAT_INTEGRATION=1 make integration   # records the live screen; opt in explicitly

The timeline stream contract

seshat cannot timestamp actions it does not perform. Any tool server that wants its actions to be narratable publishes them — one JSON object per line — to $XDG_RUNTIME_DIR/seshat/streams/<source>.jsonl:

{"at_monotonic": 12345.678, "tool": "click", "ok": true,
 "payload": {"x": 640, "y": 360}, "source": "computer-use-sway"}

• at_monotonic is required: CLOCK_MONOTONIC seconds, the value of time.monotonic() in the emitting process. That clock is host-wide, which is what makes an unrelated process's timestamps directly comparable with the recording epoch — no handshake, no session id.

• tool is required. ok (default true), payload (default {}) and source (default: the file stem) are optional.

• Each emitter owns its file and should truncate it when a new session starts.

• Events are filtered to the take's own capture window. Anything published before recording_start or after capture stopped describes something the video does not contain, and is not an anchor.

• Malformed, oversized or unreadable lines are counted in the timeline's sources report and skipped. A broken stream can never destroy a take.

The contract is driver-agnostic. On Sway, computer-use-sway plays this emitter role; on Hyprland or Omarchy, whatever drives the desktop can publish the same lines from the process that performs the actions. A driver that publishes nothing still records fine, but the take has no event anchors — narrate with at_ms or fall back to recording_scenes.

Pass timeline_sources to recording_start to ingest specific files instead of everything in the streams directory; explicit paths are validated when the take starts, not when it is finalized.

Narrating a take

Read the timeline first, then write prose against the event ids that are really there (recording_status(wait_seconds=45) below blocks until terminal instead of forcing a poll per turn):

recording_timeline()
recording_voiceover(segments=[
  {"anchor": {"event_id": 1}, "text": "..."},
  {"anchor": {"at_ms": 8200},  "text": "..."}
])
recording_status()        # poll until phase == completed | failed

recording_voiceover synthesizes each segment, builds one audio track placed at the resolved anchors (with offset_ms, optional tempo compression via fit="compress", and lead-silence trimming), mixes it with whatever audio the take already carries — over a take recorded with audio=monitor|mic, the captured desktop sound is ducked under the voice by a sidechain compressor and the whole mix is normalized to −16 LUFS; over a silent take, that stream is the voice alone — and muxes it over the video. The voice is tunable: rate, volume and pitch are passed to edge-tts, length_scale to piper, and the result reports what was used (character).

Captions are split into sentence-shaped phrases, timed by the engine's own cues where they match the phrase and proportionally by characters otherwise, burned unless subtitles=false, and written as SRT and WebVTT sidecars beside every deliverable either way — so players, YouTube and translators get the timings even when this ffmpeg build lacks libass. position="auto" lifts a caption to the top when the events inside its window acted in the bottom third of the picture (the events are read for placement, never for prose).

Plan before you record. dry_run=true answers synchronously with the schedule — every segment's synthesized duration and any overruns past the master's extent — without muxing or publishing, so a demonstration can be paced to the script before it is recorded.

The deliverable is written to <id>.narrated.<fmt> (or <id>.<variant>.<fmt> with variant="it", for narration in another language from the same take and master) and published only after validation; the finalized master at the take's master_path is never modified, so re-rendering always starts from the clean capture. The video stream is re-encoded only where work is done on it — caption burn-in, or extending the picture with cloned frames when the narration runs past the recorded extent (reported as narration.extended_ms) — and stream-copied otherwise. The result is re-validated: exactly one video stream plus exactly one audio stream. A narration failure never destroys the take: the master stays in place and the reason lands in result.narration.error — for example, an ffmpeg build without the subtitles filter cannot burn captions (the sidecars are still written). Retry that take with subtitles=false. To burn captions, start the server with a libass-enabled ffmpeg on PATH before recording; restarting the server to change PATH forgets its in-memory state.

Anchors are checked against the master's picture. event_id, at_ms and match ({"tool", "payload", "nth"} — exactly one real event, so a script survives a re-record) are validated against the playable video extent — the video stream's own duration, falling back to the container's — so speech cannot be anchored after the last frame. A segment that would run past that frame is not dropped: the picture is extended with cloned frames to cover it. An event id that exists but resolves past the end of the master's video is still refused, with both numbers in the message.

If recording_timeline reports no events, the demonstration was driven by a tool server that does not publish a stream. Anchor segments with at_ms, or use recording_scenes for approximate cuts. Scene cuts are secondary evidence; they are never the sync source.

Post-production

recording_render post-produces a completed take over its own ingested timeline — nothing in the edit looks at the picture, so every cut traces back to an event:

• trim keeps first-event to last-event plus a pad (default 300 ms).

• idle speeds up every event-free gap past min_gap_ms (the picture and the captured audio re-time together).

• zoom holds a hard-cut zoom window centered where the events acted, for hold_ms, then releases; markers pulses a box at the same points.

• profile rescales to 1080p/720p, or crops native to vertical (9:16) around the median pointer position.

• poster, teaser and chapters derive a poster PNG, a GIF teaser, and chapters from the events whose payloads carry a chapter label — muxed into the render and written as a YouTube-style list beside it.

• stitch concatenates whole finalized masters of the same shape in order, at the container level; a shape or codec mismatch is a refusal, not a half-re-encoded catalog.

• narrate (with the narration arguments: engine, voice, fit, position, subtitles and the voice character) speaks the cut itself: every segment's anchor resolves on the render's own event map — at_ms against the render's extent, an event_id as a position in the map, a content match against the remapped rows — and the track is scheduled against the render and muxed to <id>.render.narrated.<fmt>, with the cut's own audio as the ducked bed. The cut is published before its narration runs, so a narration failure leaves a complete picture with the reason in the render's narration.error.

The master is never modified; a render is a new <id>.render[.<profile>].<fmt> file in the take's store directory, validated before it is published. The result reports every event's remapped t_ms, so a script narrates a render with at_ms anchors against the render's own timeline — or passes the prose straight to the render in narrate and takes the narrated cut from one call. The render answers synchronously.

What the events do and do not prove

An event records that a tool call was dispatched, not that it visibly worked. The driving server is the only party that knows whether its action had the intended effect; a driver that returns sent: true without verifying the result will publish an event for an action that did nothing. Two open issues in the sibling project are exactly this shape: a modified key that arrived as an unmodified one, and a keystroke delivered to a window that had stolen focus.

So narrate what the take shows. Use event ids to place a line in time, and ground its content in the video, in the driver's own verified payload, or in something you observed yourself — never in the mere existence of an event.

Security notes

• Audio capture is opt-in and source-resolved. seshat records sound only when recording_start says so, and records the desktop output's monitor — never the microphone — unless you explicitly ask for mic. A take never silently starts listening to anything.

• edge-tts runs only when asked for. engine="auto" resolves to the offline piper engine (install one and a voice model, or provide voice); engine="edge" is an explicit opt-in to send the narration prose to Microsoft. The narration result records engine, voice and network.

• Narration text is the caller's; seshat never reads it from the screen, so nothing visible on screen becomes narration unless the agent says so.

• Recordings live under $XDG_RUNTIME_DIR/seshat while captured and move to $XDG_STATE_HOME/seshat/takes/<id>/ when finalized — 0700 / 0600 everywhere. Files leave the store only through recording_export and are deleted only through recording_discard.

FAQ

Why doesn't my recording have any sound? Because you didn't ask for it. Takes are silent by default — recording_start captures only a picture unless you pass audio="monitor" (desktop sound), audio="mic" (default microphone) or audio="auto". This is deliberate: a recorder that silently starts listening to a microphone is a security problem, not a convenience. seshat_info.audio shows what a take would capture.

I passed audio="mic" and got my microphone, but I wanted the desktop's sound. Which is which? monitor = what your speakers play (application and system audio, not the microphone). mic = the host's default input device, whatever that is. On a host with several microphones, the default one wins; pick input routing in your audio mixer (pavucontrol / wpctl) if you need a different one. There is one exception to know about: if no default output sink exists (headless session, no audio hardware), auto falls back to mic.

Where is my microphone in the recording? Nowhere unless you asked: a take that captured the microphone was started with audio="mic" or audio="auto" on a mic-only host. Desktop audio comes from the sink's monitor and never carries the microphone, and vice versa — one -a source per take.

GIF refused my recording with "gif cannot carry audio" — why? GIF containers cannot carry any audio stream, so there is no such thing as a sound-encoded GIF. Record format="mp4" or "webm" for takes that need audio; keep GIF for silent, looping previews.

My narration is too loud / too quiet against the desktop audio. Can I mix the levels? The mix is automatic: the captured desktop sound is ducked while the voice speaks and the whole deliverable is normalized to −16 LUFS integrated, so takes are level-consistent across a channel. The voice itself is tunable with rate, volume and pitch (edge) or length_scale (piper). If it is still hard to mix against unusually loud desktop sound, narrate a silent take (default audio="off") when the voice is the point.

Can I get narration and desktop audio in the same file? Yes — that is exactly what an audio-captured take narrated later produces. Desktop audio is recorded into the master by recording_start(audio=...), and recording_voiceover mixes the synthesized voice with it, ducking the desktop sound while the voice speaks; both are audible in the deliverable's single audio stream. The captured track and the voice are not separate selectable tracks in the file — that one stream is their mix — and the master keeps its own track untouched for a later re-render.

Why does seshat_info.audio show null for my sink? pactl info answered without a default sink: no audio server is running, or the session's user owns none. Recording silently still works. Ask for audio="mic" only if default_source resolves, and otherwise record silent.

The recording fails with "could not resolve a default audio sink" when I asked for audio="monitor". That host has no default output sink to monitor. Record with audio="mic", or without audio at all. Setting an output sink (wpctl) does not change seshat's mind mid-take: it resolves the sink at the moment the take starts; a take that already started keeps the source it resolved.

My take says audio_included: true but I hear nothing when I play it. Two normal causes: the desktop was quiet when the take ran (the desktop's sound is recorded only if it played during the capture window — seshat does not create sound), or the player you tried doesn't handle the container's audio track (MP4 = AAC, WebM = Opus; both are common, try VLC or mpv).

Related

• computer-use-sway — drives a Sway session and publishes the timeline stream that this server ingests. Recording and narration were extracted from that project so that both could stand on their own.

• omarchy-computer-use — gives an agent its own nested Hyprland session on Omarchy. It does not publish a timeline stream, so takes it drives narrate via at_ms or recording_scenes unless something else publishes for them.