# story-agent

`story-agent` is a local, MCP-first writing assistant backend for maintaining
long-form story continuity.

It keeps established story state in SQLite, protects unfinished writing with
persistent drafts, retrieves relevant older passages with local embeddings,
and gives an MCP client controlled tools for reading and updating the story.
The MCP client—Codex or another compatible agent—does the creative writing and
reasoning. The server does not call an LLM.

> This project is still under development, but the workflows documented here
> are implemented and tested.

## Features

- Multiple isolated story projects, created and selected through chat
- Transactional story initialization with metadata, a main character, and
  optional facts, lore, and an opening event
- Canonical characters, facts, relationships, chapters, events, and lore
- Persistent chapter drafts and automatic chunk-by-chunk saves
- Proposed canon changes kept separate until explicit review
- Deterministic continuity validation against established canon
- Atomic chapter commit with accepted story changes
- Bounded scene-context gathering for long writing sessions
- Local semantic search over chapters, lore, events, and character facts
- MCP tools, resources, and reusable workflow prompts
- Client-independent operation over the standard MCP `stdio` transport

## Architecture

```text
Codex or another MCP client
        |
        | stdio MCP
        v
story-agent MCP server
        |
        +-- safe project registry
        |       `-- selected isolated story database
        +-- deterministic domain services
        +-- SQLite canonical state
        +-- SQLite persistent drafts and proposals
        `-- local derived semantic index
```

- **MCP client:** conversation, prose, story reasoning, identifying claims,
  producing continuity reports, and proposing durable changes.
- **MCP server:** validation, bounded retrieval, controlled writes,
  transactions, and project selection.
- **SQLite:** canonical story state and persistent draft state.
- **Semantic index:** derived retrieval data; it is never canonical truth.

The core writing rule is:

```text
autosave protects work
commit changes canon
```

## Requirements

- Python 3.11 or newer
- An MCP client supporting `stdio`
- Node.js only for the optional MCP Inspector

Embeddings use the free local `BAAI/bge-small-en-v1.5` model through
`fastembed`. No paid API is required. A first-time model download must be
explicitly allowed from the admin CLI; normal searches never download a model
or call an external service.

## Installation

```bash
cd story-agent
python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/story-agent --help
```

You can replace `.venv/bin/story-agent` with `.venv/bin/python main.py` in the
CLI examples if the console command is not installed.

## Connect an MCP client

The entry point is `mcp_server.py`. Two environment variables control storage:

- `STORY_AGENT_PROJECTS_DIR` holds the safe project registry and each managed
  story's isolated database.
- `STORY_AGENT_DATABASE` is the legacy single-database fallback when no
  managed project is selected.

### Codex configuration

Add this to `~/.codex/config.toml`:

```toml
[mcp_servers.story-agent]
command = "/absolute/path/to/story-agent/.venv/bin/python"
args = ["/absolute/path/to/story-agent/mcp_server.py"]
default_tools_approval_mode = "approve"

[mcp_servers.story-agent.env]
STORY_AGENT_PROJECTS_DIR = "/absolute/path/to/story-agent/data/projects"
STORY_AGENT_DATABASE = "/absolute/path/to/story-agent/data/story_agent.db"
```

Replace `/absolute/path/to/story-agent` with the repository's location on your
machine, then restart the MCP session. For MCP Inspector and second-client
setup, see
[docs/mcp-clients.md](docs/mcp-clients.md).

## Normal use: chat first

Normal writers do not need the terminal for story setup.

### Create a story

Say:

```text
Create a new fantasy story.
```

The client should ask once for genuinely missing required information:

- project display name
- story title and premise
- genre and tone
- setting summary
- main character name and description

Initial character facts, lore, and an opening event are optional. The client
then calls `create_story_project`, which creates and selects a safe isolated
database, and `initialize_story`, which writes the initial canon in one
transaction.

Project creation and initialization are intentionally separate. An empty
project can exist before its story details are supplied. Initialization refuses
to overwrite an already initialized story.

### Work with multiple stories

```text
List my stories.
Create another story called Moon Gate.
Switch to The Glass Harbor.
Which story is selected?
```

Every project has its own SQLite database. The model receives generated project
IDs, never arbitrary filesystem paths. Creating a project selects it; listing
does not. Switching requires an explicit `select_story_project` call.

Selection is stored in `project_registry.db`, so it survives restarts. Clients
using the same `STORY_AGENT_PROJECTS_DIR` share the registry, current selection,
and canonical data.

### Write a long chapter

```text
Start chapter 1. Mara arrives at the abandoned lighthouse.
Continue with her meeting in the harbor office.
Continue, but keep the argument restrained.
```

After every generated chunk, the client should call `append_to_draft`. The
complete draft remains locally stored across prompts and server restarts but is
not yet a canonical chapter.

The client can keep context bounded by using the draft summary, recent excerpt,
pending changes, focused canonical scene context, semantic retrieval, and
`get_draft_segment` only when an older portion is needed. It does not need to
send the entire chapter on every turn.

### Finish and commit

Only explicit requests start the commit workflow:

```text
I'm done with this chapter.
Finish chapter 1.
Commit this chapter.
Make this chapter canon.
```

`Continue`, `next`, `looks good`, and `okay` are not commit approval.

The intended flow is:

1. Load the full stored draft.
2. Extract and validate continuity-relevant structured claims.
3. Present the continuity report.
4. Review every proposed persistent change.
5. Ask the user to accept or reject each proposal.
6. Mark the draft ready.
7. Commit the chapter and accepted changes in one transaction.
8. Read canonical state back to verify it.

A commit rolls back on failure. It cannot proceed until continuity is reviewed
and every pending change has an explicit decision.

## Canon versus drafts

Canonical state contains story metadata, characters, explicit facts,
relationships, committed chapters, timeline events, and lore.

Draft state contains an in-progress or ready chapter, its full autosaved text,
a bounded summary, and proposed new characters, facts, relationships, lore,
and events. Saving a draft never changes canonical story state. Proposals remain
pending until explicit review during commit.

## Continuity checking

```text
draft prose
  -> client identifies structured claims
  -> server compares claims with canonical SQLite state
  -> client writes the final continuity report
```

The validator supports character existence, character facts, explicit
knowledge, relationship state, event order, chapter existence, and lore
content. Results are `confirmed`, `contradicted`, or `unknown`. Missing canon is
unknown, not an error. Knowledge checks use explicit facts such as `Knows: ...`
and `Does not know: ...`.

The server does not interpret raw prose, rewrite drafts, or persist corrections.
Those reasoning and writing decisions remain with the MCP client.

## Exact and semantic retrieval

Use exact tools for authoritative identities, facts, relationships, chronology,
chapter numbers, and lore. Use semantic search to locate unstructured passages
whose wording differs from the query. For example, a search for `Alex distrusts
authority` can retrieve a passage where Alex avoids a commander and refuses an
officer's orders.

Semantic sources include chapter content and summaries, lore passages, event
descriptions, and character facts. Results contain traceable source metadata.
Similarity is only a lead; verify important structured claims with exact tools.

The index is derived and can always be rebuilt from canonical SQLite state:

```bash
.venv/bin/story-agent --database data/my-story.db search-index-rebuild

# Permit the first local model download when needed:
.venv/bin/story-agent --database data/my-story.db \
  search-index-rebuild --allow-model-download

.venv/bin/story-agent --database data/my-story.db search-index-status

.venv/bin/story-agent --database data/my-story.db \
  semantic-search "Alex distrusts authority" --limit 5

.venv/bin/story-agent --database data/my-story.db \
  semantic-search "the red stone" \
  --source-type chapter --source-type lore
```

## MCP interface

### Project tools

| Tool | Purpose |
| --- | --- |
| `list_story_projects` | List registered projects. |
| `create_story_project` | Create an isolated empty project and select it. |
| `select_story_project` | Switch explicitly to a registered project ID. |
| `get_current_story_project` | Report selection and initialization state. |

There is no project-delete tool and no path-based selection.

### Canonical read and validation tools

| Tool | Purpose |
| --- | --- |
| `get_character` | Read a character and facts by name. |
| `get_chapter` | Read one canonical chapter by number. |
| `get_recent_chapters` | Read bounded recent chapters. |
| `get_timeline` | Read events with optional chapter bounds. |
| `search_lore` | Perform case-insensitive exact lore search. |
| `get_relationship` | Read a relationship between two characters. |
| `get_scene_context` | Gather bounded characters, relationships, chapters, events, and lore. |
| `semantic_search_story` | Locate semantically related canonical passages. |
| `validate_story_claims` | Compare structured claims with canon. |
| `validate_story_change_proposal` | Validate a client-authored proposal without saving it. |

### Setup and draft tools

| Tool | Purpose |
| --- | --- |
| `initialize_story` | Atomically create metadata and initial canon. |
| `get_story_status` | Read bounded story, chapter, draft, and proposal status. |
| `get_current_draft` | Read bounded active-draft context. |
| `get_draft_segment` | Read one bounded part of the full draft. |
| `save_draft` | Create or replace an in-progress draft. |
| `append_to_draft` | Append a chunk and autosave it. |
| `save_pending_draft_changes` | Store proposed durable changes outside canon. |
| `mark_draft_ready` | Mark the active draft ready for commit. |
| `commit_draft` | Atomically promote the chapter and accepted changes. |

Pending change types are `new_character`, `character_fact`, `relationship`,
`lore`, and `event`. Each requires a short evidence excerpt.

### Controlled canonical writes

| Tool | Purpose |
| --- | --- |
| `create_character` | Create a canonical character. |
| `save_chapter` | Save canonical chapter text directly. |
| `add_story_event` | Add a canonical timeline event. |
| `add_character_fact` | Add an explicit canonical character fact. |
| `update_relationship` | Create or update a canonical relationship. |
| `add_lore_entry` | Add a canonical lore entry. |

These remain useful for administration and explicitly approved changes. Normal
chapter writing should use drafts and reviewed commit.

### Resources

| URI | Content |
| --- | --- |
| `story://summary` | Story overview, latest chapter, and bounded state. |
| `story://characters` | Bounded canonical character directory. |
| `story://timeline` | Bounded recent canonical timeline. |
| `story://chapter/{number}` | Metadata and bounded chapter content. |
| `story://character/{name}` | Description and bounded character facts. |

Resources are concise and read-only.

### Reusable prompts

| Prompt | Purpose |
| --- | --- |
| `write_next_chapter` | Context retrieval, chunked writing, autosave, review, and commit workflow. |
| `check_draft_continuity` | Evidence-based continuity review without mutation. |
| `review_story_changes` | Review proposed canon changes before writes. |

Prompts are declarative instruction templates returned to the client. They do
not call a model or execute tools themselves. Presentation varies by client: a
prompt may appear as a menu item, slash command, API feature, or an instruction
an agent chooses during conversation.

## Admin and development CLI

The CLI is optional and works with an explicit database path. Put `--database`
before the command:

```bash
.venv/bin/story-agent --database data/my-story.db COMMAND
```

Different unmanaged stories should use different files. Multi-project chat use
is managed through MCP, not these direct CLI paths.

### Setup, status, and drafts

```bash
# Terminal setup fallback; refuses to overwrite an initialized story
.venv/bin/story-agent --database data/my-story.db init

.venv/bin/story-agent --database data/my-story.db status
.venv/bin/story-agent --database data/my-story.db draft
.venv/bin/story-agent --database data/my-story.db \
  draft --start 0 --max-characters 4000

.venv/bin/story-agent --database data/my-story.db \
  draft-append 1 "Chapter title" "New chapter text"

.venv/bin/story-agent --database data/my-story.db \
  draft-append 1 "Chapter title" "New chapter text" \
  --summary "Summary so far"

.venv/bin/story-agent --database data/my-story.db draft-ready
.venv/bin/story-agent --database data/my-story.db \
  commit --continuity-reviewed

# Every pending ID must be decided
.venv/bin/story-agent --database data/my-story.db \
  commit --continuity-reviewed \
  --accept-change 1 --accept-change 2 --reject-change 3

# Does not remove canonical chapters
.venv/bin/story-agent --database data/my-story.db discard-draft
```

### Direct canonical commands

```bash
# Characters and facts
.venv/bin/story-agent --database data/my-story.db \
  character-create "Alex" --description "A reluctant courier"
.venv/bin/story-agent --database data/my-story.db character-get 1
.venv/bin/story-agent --database data/my-story.db fact-add 1 "Fears the tribunal"
.venv/bin/story-agent --database data/my-story.db fact-list 1

# Chapters
.venv/bin/story-agent --database data/my-story.db \
  chapter-save 1 "Arrival" "Chapter content"
.venv/bin/story-agent --database data/my-story.db chapter-get 1
.venv/bin/story-agent --database data/my-story.db chapter-list

# Timeline timestamps require a timezone
.venv/bin/story-agent --database data/my-story.db \
  event-add "Alex reaches the city" "2040-01-01T12:00:00+00:00" \
  --chapter-id 1 --character-id 1
.venv/bin/story-agent --database data/my-story.db timeline

# Relationships
.venv/bin/story-agent --database data/my-story.db \
  relationship-save 1 2 "distrustful" \
  --description "They suspect each other"
.venv/bin/story-agent --database data/my-story.db relationship-get 1 2

# Lore
.venv/bin/story-agent --database data/my-story.db \
  lore-create "Red Stone" "The relic glows near broken oaths"
.venv/bin/story-agent --database data/my-story.db lore-search "relic"
```

```bash
.venv/bin/story-agent --help
.venv/bin/story-agent commit --help
```

## Cross-client test

To prove state belongs to the backend rather than one agent:

1. Configure two clients to launch the same `mcp_server.py`.
2. Give both the same `STORY_AGENT_PROJECTS_DIR`.
3. Create or select a disposable test project.
4. Add a harmless character fact from client A.
5. Read it from client B.
6. Add another fact from client B and read it from client A.

The clients need not run simultaneously. Sharing only the source code is not
enough; they must share the project registry directory. Tested Inspector
commands are in [docs/mcp-clients.md](docs/mcp-clients.md).

## Tests

Run service/application and MCP transport tests separately:

```bash
.venv/bin/python -m pytest -q --ignore=tests/test_mcp_transport.py
.venv/bin/python -m pytest -q tests/test_mcp_transport.py
```

Coverage includes initialization and rollback, canonical services, autosave,
atomic commit, proposals, continuity, semantic retrieval, resources, prompts,
project isolation, persistent selection, and MCP transport discovery.

## Data safety

- Project filenames use random internal IDs, not user input.
- Selection accepts only registered, validated project IDs.
- MCP responses do not expose full internal database paths.
- Duplicate display names are rejected case-insensitively.
- Project creation never overwrites an existing database.
- Story initialization and chapter commit are transactional.
- Reads, resources, prompts, validation, and search do not mutate canon.
- No delete-project, delete-all, raw SQL, automatic persistence, or automatic
  rewrite interface exists.

Back up `STORY_AGENT_PROJECTS_DIR` to preserve managed stories. For an unmanaged
CLI story, back up its `.db` file. Do not manually edit `project_registry.db` or
move its story databases independently.

## Troubleshooting

### The MCP server is not visible

- Check that the configured Python and `mcp_server.py` paths are absolute.
- Confirm dependencies are installed in `.venv`.
- Restart the MCP session after configuration changes.
- `mcp_server.py` speaks protocol messages over standard I/O; it is not an
  interactive terminal program.

### Two clients see different stories

Ensure both use exactly the same `STORY_AGENT_PROJECTS_DIR`, then inspect the
selection with `get_current_story_project`.

### Initialization is rejected

Initialization needs a selected empty project. Create one first. An initialized
project cannot be overwritten; create or select another project.

### Semantic search is unavailable or stale

Run `search-index-status`, then rebuild the affected story's index. Add
`--allow-model-download` only when you intentionally permit the local model
download.

### A draft will not commit

Confirm that the draft is ready, continuity was reviewed, every pending change
was accepted or rejected, and referenced canonical entities are valid. A failed
commit should leave the draft and canon unchanged.

## Project layout

```text
story-agent/
├── database.py                  # SQLite schema and connection boundary
├── models.py                    # Domain models
├── main.py                      # Admin/development CLI
├── mcp_server.py                # Thin MCP interface
├── services/
│   ├── story_service.py         # Canonical operations
│   ├── workflow_service.py      # Setup, drafts, proposals, atomic commit
│   ├── project_service.py       # Multi-story registry and selection
│   ├── continuity_service.py    # Structured claim checks
│   ├── proposal_service.py      # Proposal validation
│   └── retrieval_service.py     # Chunking, embeddings, semantic search
├── tests/
├── docs/mcp-clients.md
└── data/
```

## Current boundaries

The project intentionally has no hidden/server-side LLM, paid embedding API,
GUI, automatic planning, automatic draft rewriting, automatic canon promotion,
raw filesystem/SQL access, project deletion, destructive reset, or autonomous
background behavior. Structured canon remains authoritative; embeddings only
support retrieval.

## License

See [LICENSE](LICENSE).