rolefence-scan

Find out what your AI assistant can really see in your database, before someone else does.

Part of RoleFence. Free and open source (Apache 2.0). Made by VTSOL INC.

What is it?

Many companies now let an AI assistant (Claude, ChatGPT, Codex and others) answer questions from the company database. The AI talks to the database through an "MCP server": a small program that holds a list of questions the AI may ask.

The problem: these setups are new, done by hand, and easy to get slightly wrong. A tool meant to answer "how many enquiries did we get in June?" can often also read every customer's phone number, see the supplier costs that sales staff should never see, or even change data.

rolefence-scan is a checker. You point it at the address your AI uses, and it tells you, in plain English:

• what it saw (for example: "the server lists 6 tools to anyone, with no login")

• why it matters (for example: "anyone on this address can ask those questions, and nobody is recorded as having asked")

• how to close it (for example: "put a login in front; here is which setting")

It only reads. It never changes anything in the database or the server it checks.

Related MCP server: AgentAudit

What is the goal?

Before you give an AI access to company data, you want three things to be true:

1. The AI can only read, never write.

2. Each role (sales, finance, support, …) can reach only its own data, and the hidden fields really stay hidden.

3. There is a login, so you know who asked what.

rolefence-scan checks whether that is true today, so you can fix the gaps. Run it again after every change, or in your deployment pipeline, so a mistake never reaches production without a warning.

How do I use it?

1. Install

You need Python 3.11 or newer.

uvx rolefence-scan --help

(or pipx install rolefence-scan)

2. The easy way: the wizard in your browser

rolefence-scan ui

This opens a page on your own computer (http://127.0.0.1:8765, reachable only from this machine) that walks you through five short steps: the server address, an optional login token, an optional database login (server, database name, username, password, with a "Test login" button), an optional config file, then a review and the run. Each check appears as it finishes; the findings are shown as cards you can download as an HTML or JSON report. Tokens and connection strings are kept in memory for that one run and never saved.

Run it on the machine where the MCP server runs and it finds things itself: running Toolbox and agentgateway processes (address, port, config file), the database-login variable the server was started with (used without ever being shown), and the MCP servers in Claude Code, Claude Desktop, Cursor and Codex settings: both the ones with an address and the ones the app starts as a program on this computer (shown as "Claude Desktop: crm · runs as a local program"). Each shows as "Found on this computer" with a "Use this" button.

The list is sorted from the settings alone (nothing is started, no value is shown):

• Talks to a database first: a database connection in its settings (a mongodb://, postgres://, mysql://… string, or settings named like MONGO_…, DATABASE_…, DB_…), or a known database MCP server such as MCP Toolbox for Databases.

• Unknown next, with "may reach your data, check it".

• Well-known tools that are not databases (browser automation, documentation lookup, code hosting and the like) folded away under "N other MCP servers that do not look like databases — show". They can still be scanned.

The same server listed in several places shows once, with every app that lists it ("Claude Code, Cursor"); a scan of it checks every copy's settings for plain-text passwords.

To teach the scanner another well-known server, add a line to src/rolefence_scan/data/known_servers.json, under databases or tools: a name, and the patterns that identify it. A pattern is the program's name (toolbox), a package (@scope/package, any @version is ignored), a Docker image (mcp/name), or a web address's host (mcp.example.com, which also matches its subdomains). Pull requests welcome.

Not sure where to find the address, the connection string or the config file? Each step has a "Where do I find it?" link. In short: the address is in your AI app's MCP settings (for Claude Code, .mcp.json; for Claude Desktop, Settings → Developer → Edit Config); the config file (tools.yaml) and the connection string live on the machine where the MCP server runs, so ask whoever set it up. Everything but the address is optional.

2b. Or the command line

The simplest run: just the address your AI assistant is configured with.

rolefence-scan --mcp http://127.0.0.1:5000/mcp

That already tells you whether the server answers without a login, whether a "sales" address leaks the "finance" tools, and whether the server is open to the whole office network.

3. Give it more, get more

Each extra piece of information unlocks more checks. Passwords and tokens are never typed on the command line: you put them in an environment variable and give the scanner the variable's name.

A full run looks like this:

export SALES_TOKEN='...'                                   # a login token for the sales role
export DB_URI='mongodb://user:pw@host/db?authSource=db'   # the connection the MCP server uses

rolefence-scan --mcp http://127.0.0.1:3001/mcp --token-env SALES_TOKEN --role sales \
               --backend http://127.0.0.1:5002/mcp \
               --db-env DB_URI --config tools.yaml --json report.json

The exit code tells a script how it went: 0 nothing found, 1 low or medium findings, 2 at least one high finding, 3 it could not run. Use it in CI to stop a deploy that opens something up.

4. Or just ask your AI assistant to run it

The scanner can itself be an MCP server. Register it with Claude Code, Codex or any MCP client, and ask in plain words: "scan my MCP server at http://127.0.0.1:3001/mcp as the sales role". The assistant runs the checks and explains the report.

{"mcpServers": {"rolefence-scan": {"command": "rolefence-scan", "args": ["serve"]}}}

Set the environment variables in the shell that starts the assistant. The assistant only ever sees variable names, never the secrets.

5. Local-program servers

Most MCP servers are not at a web address at all: the AI app starts them as a program on your computer, from its settings file (command, args and env in claude_desktop_config.json, .mcp.json, Cursor's mcp.json, or [mcp_servers.<name>] in Codex's config.toml). The risk there is the database login the program holds, which is often for a database on another computer and often written into that settings file in plain text.

# read the settings file only; nothing is started
rolefence-scan --stdio-from ~/Library/Application\ Support/Claude/claude_desktop_config.json

# also start one server once, the way the AI app does, to list its tools (no tool is run)
rolefence-scan --stdio-from .mcp.json --server crm

# or a command you type yourself
rolefence-scan --stdio "npx -y some-mcp-server"

What it checks:

• A password or key written in the settings file. A connection string with a password in it, or a setting named like a password, token or API key. The report names the server, the file and the setting's name, and says whether the database is on this computer or another one. It never shows a value, a host name or any part of the connection string.

• Tools that change data, by name and description (insert, update, delete, create, send…).

• Every tool is open to whoever uses this computer: a local program has no login and no roles; every caller gets every tool.

To list the tools the scanner starts the program with the settings from the file, sends initialize and tools/list, and stops it (after 30 seconds at most; --stdio-timeout changes that). It has no way to send tools/call. Anything the program prints is discarded unread. The wizard asks before it starts anything ("This will start crm the way your AI app does, ask for its tool list, and stop it. No tool is run."); in serve mode the scan_local_program tool only starts a server named in a settings file, never a free command. The network checks (login, gateway, open ports) are listed as "does not apply to a local program".

What does a report look like?

This is a real run against the "weak" demo setup that ships with the scanner (made-up data):

rolefence-scan report for http://127.0.0.1:5010/mcp/sales

11 finding(s): 6 high, 3 medium, 2 low

1. [HIGH] Tools are listed without a login
   What we saw:    The server answered with 3 tool(s) when no token was sent.
   Why it matters: Anyone who can reach this address can see every question the AI can ask
                   and call them. There is no record of who asked.
   How to close:   Put a login in front: a gateway that checks tokens, or the question
                   server's own auth setting.

2. [HIGH] The bare /mcp address serves every tool
   What we saw:    /mcp/sales lists 3 tool(s); /mcp lists 6, including 3 not meant for this caller.
   Why it matters: Per-role toolsets are a menu, not a boundary. Anyone who drops the toolset
                   name from the address gets everything.

3. [HIGH] The AI's database user can write
   What we saw:    The user the question server connects with holds write rights.
   Why it matters: A tool mistake, a prompt injection, or a pipeline with $out/$merge can
                   change or delete data.
   How to close:   Create a user with a custom read-only role and point the server at it.

4. [HIGH] Server-side JavaScript runs for the AI's database user
   ...

The same company, set up properly (one read-only user per role, fixed questions, a login in front), gets 4 findings instead of 11, and the scanner says which ones remain.

NIST AI RMF

Risk and audit teams often work from the NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1, and its Generative AI Profile, NIST AI 600-1). Every report shows which of its items the scan gives evidence for:

• each finding has an AI RMF: line, for example MEASURE 2.7, MEASURE 2.10, MANAGE 1.3;

• at the end, one row per item with what this scan showed and a status: Evidence, Gaps to fix, or Not checked in this scan (the checks for it did not run this time), plus a fixed row for what this tool does not cover at all: accuracy, bias and model testing.

  MEASURE 2.10 Privacy: personal and secret data the AI can reach
               Status: Gaps to fix
               What this scan showed: 3 finding(s) to fix: Tools accept any query from the model; ...

The items used are GOVERN 1.6, GOVERN 6.1, MEASURE 2.7, MEASURE 2.8, MEASURE 2.10, MANAGE 1.3, MANAGE 3.1 and MANAGE 4.1. The AI RMF is voluntary guidance and has no certification: the report gives evidence for its data-access parts and does not certify compliance. The JSON report has the same in ai_rmf (per finding and a summary). Which check counts towards which item is one data file, src/rolefence_scan/data/ai_rmf.json; corrections are welcome.

Examples

"Our sales team's AI must never see supplier cost." Run with --db-env set to the sales user's connection. If the scanner reports "cross-collection reach" into orders or "the AI's database user can write", the database user is too powerful, whatever the tools say. If it reports "Cross-collection reach is contained by the database role", the data layer is doing its job.

"We put a gateway with login in front. Are we done?" Run with --mcp on the gateway and --backend on the server behind it. "The question server can be reached without the gateway" means anyone who knows the inside address skips the login.

"Users say they get 401 even with a valid login." Run with --token-env. "A valid token is refused" with the hint about pass-through means the gateway checked the token and then threw it away before forwarding, so the server behind it saw none. One setting fixes it.

"A developer set this up months ago. Is it still safe?" Run it with everything you have, save --json report.json, and run it again after each change. Compare the two.

The full list of checks

Works today with MongoDB and MCP Toolbox for Databases, with or without agentgateway in front. Other databases are planned.

Try it on the demo

demo/ holds a made-up removals company, Brightline Movers: 2,000 customers, 3,200 enquiries, 1,110 orders, all invented. It comes with two setups:

• demo/toolbox/weak.yaml: how a first attempt usually looks. One over-powered database user, tools that accept any query, no login. Expected: 11 findings.

• demo/toolbox/governed.yaml: fixed questions, one read-only user per role, hidden fields removed by a view. Expected: 4 findings, all about the missing login (that is the gateway's job, below) and a database setting.

export DEMO_ADMIN_URI='mongodb://<admin>:<pw>@127.0.0.1:27017/admin'
uv run python demo/seed.py                    # writes database rolefence_demo
mongosh ... demo/setup_users.js               # see the header of that file for the command
toolbox --config demo/toolbox/weak.yaml --port 5010     # with DEMO_RW_URI set
rolefence-scan --mcp http://127.0.0.1:5010/mcp/sales --db-env DEMO_RW_URI --config demo/toolbox/weak.yaml

With a login and a gateway

demo/gateway/ adds the identity side: a Keycloak realm (rolefence-demo, roles sales and finance, scopes demo:sales / demo:finance), a third Toolbox config with Toolbox's own login (demo/toolbox/governed_auth.yaml, port 5012), and two agentgateway configs in front of it: gateway_good.yaml (port 3011, passes the token on) and gateway_weak.yaml (port 3012, drops it, which is agentgateway's default). Passwords are typed into the terminal, never written to a file; get_token.sh fetches a token into a variable and prints only who it is for.

The whole thing in one command (asks for the passwords, starts everything, runs the three scans, stops what it started):

demo/gateway/run_live.sh        # set AGENTGATEWAY=/path/to/agentgateway if it is not on PATH

Or step by step:

# 1. Keycloak (Docker) and the demo realm
docker run -d --name keycloak -p 127.0.0.1:8080:8080 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin \
  -e KC_BOOTSTRAP_ADMIN_PASSWORD="$KC_ADMIN_PW" quay.io/keycloak/keycloak:26.7.1 start-dev
export KC_ADMIN_PW DEMO_SALES_USER_PW DEMO_FINANCE_USER_PW   # typed with read -s, see the script header
demo/gateway/setup_keycloak.sh

# 2. Toolbox with its own login (needs DEMO_SALES_URI / DEMO_FINANCE_URI as for governed.yaml)
toolbox --config demo/toolbox/governed_auth.yaml --port 5012 --toolbox-url http://127.0.0.1:5012

# 3. The two gateways (binary from https://github.com/agentgateway/agentgateway/releases)
agentgateway -f demo/gateway/gateway_good.yaml &
agentgateway -f demo/gateway/gateway_weak.yaml &

# 4. A sales token, then scan
source demo/gateway/get_token.sh sales                        # sets DEMO_SALES_TOKEN
rolefence-scan --mcp http://127.0.0.1:3011/mcp --token-env DEMO_SALES_TOKEN --role sales \
               --backend http://127.0.0.1:5012/mcp --config demo/toolbox/governed_auth.yaml
rolefence-scan --mcp http://127.0.0.1:3012/mcp --token-env DEMO_SALES_TOKEN --role sales \
               --backend http://127.0.0.1:5011/mcp
rolefence-scan --mcp http://127.0.0.1:5012/mcp --token-env DEMO_SALES_TOKEN --role sales

What each run shows:

All three were run live on 15 Oct 2026 with demo/gateway/run_live.sh and matched this table.

What it does not do yet

• Secrets that can be worked out from other fields. If sales can see price and the AI can find cost somewhere, it can work out the margin. The scanner cannot know which fields are secret in your business. Planned: a short wizard where you tick the secret fields and the scanner checks each role against them.

• Watching the server over time, for example telling you that a config change was rejected and the old config is still running.

• Fixing things. It finds and explains; you (or the RoleFence console) make the change.

• PostgreSQL and other databases. MongoDB first.

Is it safe to run on a live system?

Yes, by design:

• It only reads: it asks the database what the user is allowed to do, runs tiny read queries, and calls tools with empty arguments. Nothing is inserted, updated or deleted.

• The report never contains data values, only names of fields, collections and tools, and counts.

• Secrets are never on the command line, so they do not end up in shell history or logs.

• reports/ and *.report.json are git-ignored, because a report can contain host names.

Where the checks come from

GAPS.md is the log of 30 things that did not hold when we built this setup and tried to break it, with the scanner check that catches each. The story version, for people who would rather read than scan, is docs/what-a-model-can-reach.md.

Questions or ideas

Open an issue. If you found a gap the scanner missed, that is the most useful thing you can tell us: every new check started as someone's "it shouldn't have been able to do that".