ControlsGuard

A grounded, agentic GenAI assistant that answers questions about company policies, controls, risks and the systems they protect, and finds control-coverage gaps. Built as a portfolio project for a GenAI Developer role in a controls-technology team: it applies hosted foundation models through context engineering, RAG, Graph RAG and tool-using agents, and treats guardrails, evaluation and observability as first-class.

It runs offline with no API keys (a deterministic mock model + local embeddings), and switches to a real model with three environment variables.

Quick start

pip install -r requirements.txt
export PYTHONPATH=src

python -m controlsguard.cli "How quickly must a leaver's access be removed?"
python -m controlsguard.cli "Are there any control gaps on ledger-core?"

python -m unittest discover -s tests -t .      # 70 tests
python evals/run_evals.py                      # quality + safety gates, writes evals/report.md

uvicorn controlsguard.api:create_app --factory --port 8000
python -m controlsguard.mcp_server             # MCP server (stdio); add --http for streamable HTTP
docker compose up --build

Use a real model: CG_LLM_PROVIDER=anthropic|openai, CG_LLM_MODEL=<model id>, CG_LLM_API_KEY=... (CG_LLM_BASE_URL for any OpenAI-compatible endpoint, including Gemini's).

Related MCP server: KaaS MCP Server

How it works

question -> guard_input -> retrieve -> graph_expand -> build_context -> act (LLM <-> tools) -> verify -> respond
              | injection?   | off-topic?                                                      | grounding,
              v blocked      v abstain / clarify                                               v PII, leak check

Guardrails and responsible AI

• Prompt injection blocked before any retrieval or model call; indirect injection (instructions hidden inside retrieved documents or tool output) scrubbed line by line

• PII redaction (email, PAN, Aadhaar, SSN, Luhn-validated cards, phone) on input, model output, memory and logs; digit patterns are anchored so a long number can't be partially redacted

• Grounding check: every substantive sentence needs a valid citation and lexical support in the cited text, or the answer is replaced by a safe fallback instead of being shown

• Abstention on off-topic questions and clarification on dangling references

• System-prompt leak detection, tool allow-list, schema validation, API-key auth, input size limits

Evaluation (evals/)

47 labelled cases (see evals/report.md after a run) with CI gates: retrieval hit@k/MRR with a BM25 vs vector vs hybrid ablation, answer correctness + citation correctness, graph and coverage questions, refusals, injection blocking, PII leakage (including partial leaks), and an indirect-injection test against a deliberately poisoned document. The run fails the build if any gate regresses, mirroring the corpus's own control C-AI-02 (prompt and model changes need evaluation evidence).

Honest limitations

• The default model is a mock. It writes extractive, cited answers so the full pipeline is testable offline; it is not a language model. Quality numbers describe the retrieval, graph, guardrail and harness layers. To judge answer quality with a real LLM, run the eval against CG_LLM_PROVIDER=anthropic|openai and extend the dataset.

• The corpus is synthetic (6 policies, 19 controls, 8 risks, 6 systems). One gap, R-VENDOR on ledger-core, is planted so the coverage check has something real to find.

• Offline embeddings are lexical-semantic (LSA). With a hosted embedding model, semantic recall (e.g. eval case ret-19, "root rights" vs "administrator and root credentials") should improve; it is kept in the report as a known miss.

• Guardrails are transparent heuristics (regex, lexical overlap), a first layer rather than a replacement for model-based classifiers. The vector store is in memory (swap for pgvector/OpenSearch at scale).

• langgraph_app.py and the live Anthropic/OpenAI adapters were not run against the real services here (no network access to package indexes or APIs in the build environment). The adapters are covered by tests with a fake transport. The A2A surface follows the 0.3.0 Agent Card / message/send shape; verify against a real A2A client.

JD skill map