Use the Newton MCP Gateway to query and embed physical-world sensor data via Newton and to propose validated physical action contracts without executing them.
Ask Newton text-reasoning questions with newton_query, grounded in inline text/JSON events or previously uploaded file IDs; optionally force structured JSON via system_prompt.
Encode channel-first sensor windows with newton_embed_timeseries into one 768-dim embedding per channel; optionally normalize.
Analyze a single PNG/JPEG image with newton_analyze_image using inline base64 or an existing file ID; read-only and stateless.
Propose exactly one schema-validated Physical Action Contract from an observation with newton_propose_action; can restrict allowed goals; returns a valid contract or explicit failure with raw model text/errors, never executes.
All tools are read-only and open-world; mock results are labelled backend: mock / [mock], and real Newton calls require an ATAI_API_KEY.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Newton MCP Gateway Why did kitchen vibration spike at 3pm? Check these events."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
An experimental Model Context Protocol bridge for Archetype AI Newton, the foundation model for physical-world sensor data.
This project explores two complementary integration patterns:
Newton as an MCP capability — make physical-world intelligence available to any
MCP-compatible agent (Claude, ChatGPT, enterprise copilots, custom hosts) through a small,
auditable gateway over Newton's publicly documented /query API.
MCP as an action boundary for Newton — connect physical-world understanding to external digital and physical capabilities through policy-controlled, auditable and verified actions.
Status: experimental, independent project. The MCP server and mock backend are functional. The real Newton adapter is implemented against public documentation but has not yet been validated against a live account; it activates only with an authorized
ATAI_API_KEY. The action runtime and the smart-home testbed are mock-validated only -- no live actuator has been driven by this runtime. This project is not affiliated with or endorsed by Archetype AI.
Newton turns raw sensor streams into physical understanding. MCP is becoming the common interoperability boundary for agent capabilities. Connecting the two creates value in both directions:
DIGITAL AGENTS PHYSICAL WORLD
Claude / ChatGPT / enterprise agents sensors, machines, environments
│ ▲
│ MCP │ actuators (via MCP servers)
▼ │
newton-mcp-gateway ──────► Newton ──────► MCP Action Runtime
(Direction B: implemented) physical (Direction A: proposal)
understanding policy · approval · verifyThe deeper idea is not MCP itself. It is that Physical AI needs an open, safe, auditable and closed-loop boundary between understanding the physical world and changing it — and MCP may be a good foundation for that boundary.
Related MCP server: Grok Bot MCP Bridge
git clone https://github.com/mctlhq/newton-mcp-gateway
cd newton-mcp-gateway
uv sync
NEWTON_BACKEND=mock uv run newton-mcp # stdio MCP serverClaude Desktop / Claude Code configuration:
{
"mcpServers": {
"newton": {
"command": "uv",
"args": ["--directory", "/path/to/newton-mcp-gateway", "run", "newton-mcp"],
"env": { "NEWTON_BACKEND": "mock" }
}
}
}With real Newton access (variable names follow Archetype's docs):
NEWTON_BACKEND=api ATAI_API_KEY=... ATAI_API_ENDPOINT=https://api.u1.archetypeai.app/v0.5 uv run newton-mcpThe gateway defaults to the stdio transport for local MCP hosts. To reach it over HTTP
instead — for a host that is not a local Claude Desktop process on the same machine — set
NEWTON_MCP_TRANSPORT=streamable-http:
NEWTON_BACKEND=mock NEWTON_MCP_TRANSPORT=streamable-http NEWTON_MCP_PORT=8000 uv run newton-mcp
# -> http://127.0.0.1:8000/mcpOr run the container, which enables streamable-http by default:
docker build -t newton-mcp-gateway .
docker run --rm -p 8000:8000 newton-mcp-gateway
# -> http://127.0.0.1:8000/mcpThe container binds 0.0.0.0, which is intended for a container network. The gateway
ships no authentication, so exposing it to an untrusted network is the operator's
responsibility — put a proxy or firewall in front of it.
Deliberately few. Each maps to a documented Direct Query pattern.
Tool | Newton model family | What it does |
| Newton C (text / image / video reasoning) | Natural-language question grounded in inline text/JSON events or uploaded |
| Omega encoder | Channel-first sensor window → one 768-dim embedding per channel. |
| Newton C (image reasoning) | Ask a question about one image: inline |
| Newton C (text reasoning) | Read-only: observation → exactly one validated Physical Action Contract, or a failure with the raw model text. Proposes only; executes nothing. |
Planned (see issues): running Newton Agent bundles (osm, anomaly-discovery,
rare-event-detection, task-verification) and paging their results.
Mock results are labelled backend: "mock" and text outputs start with [mock]. They are never
presented as real Newton output.
A successful MCP tool call is not a successful physical action. set_temperature(23) can return
200 OK while the AC is offline, the wrong zone was changed, or the room simply does not cool.
Physical actions need policy, approval and outcome verification on top of ordinary tool calling.
This project proposes a tool-independent Physical Action Contract (schemas/, v0.2):
{
"goal": "reduce_room_temperature",
"reason": "The occupied kitchen reached 29.4 C",
"confidence": 0.96,
"target": { "type": "environment", "location": "kitchen" },
"constraints": { "desired_temperature_c": 23, "minimum_temperature_c": 20, "maximum_temperature_c": 25 },
"risk": "low",
"verification": {
"condition": { "path": "temperature_c", "op": "le", "value": 24 },
"timeout_seconds": 600
}
}verification.condition is a structured object, never an expression string: a predicate
{path, op, value} (op one of eq | ne | lt | le | gt | ge), or an {all: [...]} / {any: [...]} composite of predicates. There is no expression parser and no eval anywhere in the
repo -- a reviewer audits a condition by reading the JSON.
Newton says what should happen. The MCP environment knows how. A small action runtime sits in between:
contract → capability match (MCP tool discovery) → policy (auto / confirm / deny)
→ human approval if required → execute → observe → verify → succeed / retry / escalateThe contract can be produced by a Newton /query with a strict JSON system prompt — a documented
usage pattern — or by a Newton Agent's output. The full pipeline above — capability resolver,
policy engine, approval, executor and verifier — is implemented and mock-validated; see
docs/action-runtime.md.
The newton_propose_action MCP tool implements the first link of that chain: it sends the caller's
observation (text_events / json_events) to Newton C with a strict JSON system prompt that embeds
the contract's schema verbatim and, when allowed_goals is given, restricts which goals the model
may propose. The model's response is parsed with no repair — no fence stripping, no substring
extraction. If it does not validate against PhysicalActionContract, the tool retries exactly once
with the validation errors appended to the prompt, then gives up: it returns status: "failed" with
the raw model text and the accumulated errors, never a guessed or partially filled contract. A
backend-reported failure is terminal with no retry. This is mock-validated only — the mock backend
returns the repo's example contract, clearly labelled backend: "mock" and with reason prefixed
[mock] , so it can never be mistaken for real inference.
The first real actuator testbed is a smart-home MCP server (lights, HVAC, speaker), chosen because it is real hardware with benign, reversible actions. The interface itself is domain-independent: Home Assistant, BMS, OPC-UA, ROS or enterprise workflow MCP servers plug in the same way.
src/newton_mcp/runtime/ implements the first box of that pipeline: the gateway process also
becomes an MCP host/client. Given a reviewable runtime.yaml allow-list, it discovers tools
on the configured MCP servers (list_tools only, never call_tool) and deterministically
resolves a PhysicalActionContract into ranked, explainable candidate tool calls.
src/newton_mcp/action/policy.py then decides auto/confirm/deny against a reviewable
policy.yaml, and src/newton_mcp/action/approval.py can bind an Approval to one exact
resolved action (server, tool, args, action id, policy version and expiry) via a sha256 binding
that a re-pointed server or a changed argument invalidates. src/newton_mcp/runtime/lifecycle.py
tracks an approved action through an explicit ActionState machine (including an UNKNOWN state
for a tool-call timeout, which can never go straight back to EXECUTING) with four correlation
ids, and src/newton_mcp/runtime/audit.py writes one append-only, redacted JSONL line per
accepted transition. src/newton_mcp/runtime/executor.py then calls the chosen MCP tool with a
bounded timeout, and src/newton_mcp/runtime/verifier.py re-observes the world through the
capability's read_tool, evaluating the contract's structured verification.condition
(src/newton_mcp/action/conditions.py) before ever deciding to retry a non-idempotent action.
Mock-validated only -- no live actuator or live Newton credentials. See docs/action-runtime.md.
examples/smart-home/ is a self-contained, runnable artefact for the full Direction A pipeline
above: propose_action → capability resolution → policy → approval → execution → verification,
against an in-process fake smart-home actuator (fake_alice.py) and a simulated kitchen. No
socket, no subprocess, no credential.
The recording is the mock backend with the in-process fake actuator -- no live Newton account and
no real device. It is regenerated from docs/demo.tape with VHS.
uv run python examples/smart-home/demo.py --mockSuccess path excerpt:
policy: decision=auto reason="matched rule 'ac-reduce-temperature'"
terminal state: SUCCEEDED
actuator tool calls: 1
read polls: 3--ac-offline demonstrates the failure this project exists to catch -- a digitally successful
tool call ({"accepted": true}) with no physical effect, so the run ends ESCALATED rather than
SUCCEEDED:
uv run python examples/smart-home/demo.py --mock --ac-offline
...
terminal state: ESCALATED
actuator tool calls: 2
read polls: 20The two actuator calls are the first attempt and the one retry allowed by the mock contract's
retry_limit: 1 (the AC capability is idempotent); the read polls are the verifier's
get_room_state calls between and after them, counted separately.
Both excerpts are mock-validated. See examples/smart-home/README.md for the full capability
table, every flag and exit code, the trace.jsonl regeneration command, and why a real
smart-home server is not yet drivable by this runtime.
Source | |
| Archetype public docs |
Agents API (blueprints → bundles → runs, results/events) | Archetype public docs |
This gateway's MCP tool mapping | this project |
Physical Action Contract, policy engine | this project's experimental proposal |
Lifecycle ( | this project's experimental proposal |
Open questions for Archetype engineers | see |
Nothing here reverse-engineers private endpoints or circumvents access controls.
Document | What it covers |
Archetype's own terminology, Direction A vs. Direction B, and the lifecycle summary. | |
The MCP host, tool discovery, resolver, policy, approval, execution, verification and audit -- in detail, traced to | |
The safety defaults tables, the invariants a reviewer can check, demo-safe actions, and known limitations. | |
What is confirmed from Archetype's public documentation versus what this project proposes, plus the open questions for an Archetype engineer. | |
Traceability notes for the | |
The runnable, mock-validated closed-loop demo: capabilities, flags, exit codes, and why a real actuator is not drivable yet. |
uv sync --group dev
uv run pytestLayout: src/newton_mcp/newton/ (backend protocol, mock, real adapter) ·
src/newton_mcp/action/ (contract, policy) · src/newton_mcp/server.py (MCP server) ·
schemas/ · examples/ · docs/.
MIT
Ask Newton C a natural-language question about exactly one image. Provide exactly one of: image_base64 (with mime_type image/png or image/jpeg), sent inline as a data.base64_img event; or file_id, an existing Files API file_id (must end in .png/.jpg/.jpeg). This tool never uploads or stores anything -- it makes one read-only call to /query.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| file_id | No | ||
| question | Yes | ||
| mime_type | No | ||
| image_base64 | No | ||
| system_prompt | No | ||
| max_new_tokens | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=true, and the description reinforces this by disclosing that it 'never uploads or stores anything' and makes 'one read-only call to /query', adding endpoint and data-retention context beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences that front-load the core action and then the input contract. Every clause carries information; no filler or restatement of the name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, return-value explanation is unnecessary, and the read-only safety profile is covered by both annotations and the endpoint note. The remaining gap is the undocumented optional parameters.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 7 parameters, so the description must compensate; it documents image_base64, mime_type values, and file_id suffix requirements, but says nothing about model, system_prompt, or max_new_tokens, leaving several parameters undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Ask Newton C a natural-language question about exactly one image') and constrains scope to a single image, which cleanly separates it from the sibling text/query and embedding tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly specifies the mutual-exclusion rule: provide exactly one of image_base64 or file_id, with format constraints on each. It does not discuss when to prefer this over newton_query, but the input-selection guidance is concrete.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Encode a sensor window with the Newton Omega encoder. Input is channel-first: outer list = channels, inner lists = samples. Returns one 768-dim embedding per channel. Leave normalize=false unless cross-window amplitude is irrelevant.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| channels | Yes | ||
| normalize | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnlyHint, openWorldHint), so the bar is lower; the description still adds real behavioral context by specifying the input layout semantics ('channel-first') and the output cardinality (one 768-dim embedding per channel), which the annotations cannot convey. It does not disclose the model default behavior, but that is minor.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with purpose, then input shape, then the normalize caveat. No filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the return value needn't be re-explained, and the description covers the non-obvious input structure and the normalize tradeoff. The undocumented 'model' parameter is the only gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must carry the load, and it does for the critical 'channels' parameter ('outer list = channels, inner lists = samples') and gives a decision rule for 'normalize'. Only the 'model' parameter is left unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Encode a sensor window'), names the concrete encoder ('Newton Omega'), and its scope is narrow enough to distinguish it from the sibling newton_query without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to choose this tool over newton_query, and no prerequisites described. The only conditional ('Leave normalize=false unless...') is a parameter-level instruction, not a when-to-use rule.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Propose exactly one validated Physical Action Contract from a physical-world observation, by calling Newton C with a strict JSON system prompt. Returns either a schema-valid contract or a failure with the raw model text and validation errors -- never a guessed, defaulted or partially filled contract. This tool only proposes an action; it never executes or authorises one.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| json_events | No | ||
| text_events | No | ||
| allowed_goals | No | ||
| max_new_tokens | No | ||
| observation_id | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=true, and the description adds real behavioral context beyond them: it returns either a schema-valid contract or a failure carrying raw model text and validation errors, and it never guesses, defaults, or partially fills a contract. It still omits anything about permissions, rate limits, or model/network failure behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences that are front-loaded with the core action and end with the key non-execution guarantee. It is appropriately sized, though 'calling Newton C with a strict JSON system prompt' is implementation detail that is slightly less essential to a caller.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return structure need not be spelled out, and the safety profile is covered by annotations and the description. But with six parameters at 0% schema coverage and no compensating parameter guidance, the definition is incomplete for actually invoking the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for six parameters (model, json_events, text_events, allowed_goals, max_new_tokens, observation_id), so the description carries the full burden. It only indirectly alludes to observation input and the 'Newton C' model, but explains none of the individual parameters or their formats, so an agent cannot tell what to pass for most fields.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Propose exactly one validated Physical Action Contract from a physical-world observation'), and clarifies the boundary that it proposes but does not execute or authorise. It does not explicitly distinguish itself from the siblings (analyze_image, embed_timeseries, query), but the purpose is unambiguous on its own.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'from a physical-world observation' implies the input context and usage setting, and the no-execution statement scopes it. However, there is no explicit when-to-use/when-not guidance and no mention of the alternative sibling tools, leaving routing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Ask Newton (text-reasoning model) a natural-language question about physical-world data. Ground it with inline text/JSON events or previously uploaded file_ids. Use system_prompt to force structured JSON output.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| query | Yes | ||
| file_ids | No | ||
| json_events | No | ||
| text_events | No | ||
| system_prompt | No | ||
| max_new_tokens | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and openWorldHint, so safety and open-domain behavior are covered. The description adds genuinely useful behavioral context (it is a reasoning model that can be grounded with events/files and steered via system_prompt), but says nothing about latency, cost, token limits, or failure modes of an LLM call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the core action and then the grounding/steering options. Every sentence contributes, with only minor duplication of the 'model' concept.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return values need not be explained, and annotations cover the read-only profile. Still, with 7 parameters at 0% coverage, the omission of model and max_new_tokens and of event-array formats leaves an agent guessing on nontrivial inputs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 7 parameters, so the description carries the burden. It explains query, file_ids, json_events, text_events, and system_prompt, but leaves model and max_new_tokens entirely undocumented and gives no format details for the event arrays.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Ask) and resource (Newton, a text-reasoning model) scoped to physical-world data, which is clearly distinct from the sibling newton_embed_timeseries. It stops short of explicitly naming the sibling to differentiate, so it lands just below the top band.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives concrete invocation guidance — ground the query with inline text/JSON events or previously uploaded file_ids, and use system_prompt to force structured JSON. However, it never states when to prefer this over newton_embed_timeseries or any prerequisites for the grounding inputs, leaving the choose-a-tool decision to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.1newton_analyze_imagenewton_propose_actionv0.1.0newton_embed_timeseriesnewton_queryScored across 4 tools
Tools have mostly distinct purposes: analyze_image for image Q&A, propose_action for action contracts, embed_timeseries for embeddings, and query for general text/JSON Q&A. Potential overlap exists between query and analyze_image since both ask natural-language questions, and query's structured JSON capability could be confused with propose_action, though descriptions clarify boundaries.
All tools share a consistent 'newton_' prefix, and three of four follow a verb_noun pattern (analyze_image, propose_action, embed_timeseries). The bare verb 'newton_query' is a minor deviation but remains readable and predictable.
Four tools is a well-scoped set for a model gateway, each covering a distinct capability (image, action, embedding, query). The count is within the ideal 3–15 range, and no tool feels redundant or missing from the core set.
The surface covers key interactions with Newton C: querying, image analysis, action proposal, and timeseries embedding. A file upload tool is absent, but the tools accept base64 inline or pre-existing file_ids, so agents can work around it; no execution tool is expected by design.
OCR, transcription, file extraction, and image generation for AI agents via MCP.
Persistent memory, knowledge graph, task management and more for AI agents via MCP.
Analytical memory for AI agents: a real Postgres queried in plain English over MCP. One command.
Human-input bridge for AI agents with voice-first answer links, MCP tools, and HTTP APIs.