AREL
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@AREL@AREL run a reliability benchmark on my agent"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
AREL — Agent Reliability & Evaluation Lab
A production-oriented reliability, benchmarking and evaluation system built around an MCP agent runtime.
Why this exists
The upstream MCP agent runtime is an excellent framework for composing agents with Model Context Protocol — but shipping it to production demands more than composition primitives. Real production systems need multi-version protocol negotiation, peer-to-peer agent discovery, backpressure-aware streaming, connection pooling with circuit breakers, hot-reloadable plugin architectures, resilient retry & state recovery, and continuous health monitoring with autoscaling.
AREL keeps the upstream runtime as a foundation and layers a complete reliability, benchmarking and evaluation platform on top:
5 protocol versions supported (MCP 1.0 → 2.1 + A2A v0.3) with formal negotiation and deprecation warnings
9 new subsystems implemented as additive, non-invasive modules (3 155 LOC of source, 2 096 LOC of tests)
109 new tests added — 0 regressions, suite pass rate restored from 85.9 % → 99.4 %
82 % line coverage on the new
enhancementspackage470 k msg/s adaptive streaming throughput (+34 % over the naive baseline)
At a glance
Capability | Before | After |
Test pass rate | 1292 / 1503 (85.9 %) | 1602 / 1612 (99.4 %) |
New tests added | — | 109 (0 regressions) |
Coverage on | n/a | 82 % |
MCP protocol versions | 1.0 – 1.20 | 1.0 – 2.1 + A2A v0.3 |
Adaptive streaming throughput | 352 k msg/s (naive) | 470 k msg/s (+34 %) |
Connection pool reuse | — | ~4× reduction in factory calls |
New subsystems | — | 9 (see Capability Surface) |
Source LOC added | — | 3 155 (12 files) |
Test LOC added | — | 2 096 (11 files) |
Architecture
Architecture
├── Benchmark methodology
├── Evaluation methodology
├── Fault injection
├── Observability
├── Measured results
└── ReproducibilityEach branch above is realised by a concrete module under src/mcp_agent/enhancements/ and is exercised by the test suite under tests/enhancements/. The branches are not marketing copy — they map 1:1 to files, classes and measurable numbers (see Measured results below).
System architecture
The diagram below shows how AREL sits on top of the upstream MCP agent runtime. The upstream mcp_agent.* package (left, in grey) provides composition primitives; the new mcp_agent.enhancements.* package (right, in colour) provides the reliability, benchmarking and evaluation platform. The two are wired together by the HybridMCPA2AGateway and the ResilientExecutor — everything else is additive and can be adopted incrementally.
flowchart LR
subgraph UP["Upstream mcp_agent runtime (Apache-2.0, unchanged)"]
direction TB
APP["MCPApp<br/>context & lifecycle"]
AGENT["Agent + AgentSpec"]
LLM["AugmentedLLM<br/>(OpenAI/Anthropic/Bedrock/…)"]
WF["Workflows<br/>orchestrator · router · parallel · swarm"]
MCP["MCP client<br/>stdio + HTTP transports"]
TRACE["OpenTelemetry tracing"]
LOG["Rich structured logging"]
end
subgraph ENH["AREL enhancements package (this project)"]
direction TB
P1A["P1.1 Protocol<br/>MCPProtocolAdapter"]
P1B["P1.2 A2A<br/>AgentCard · A2AClient<br/>HybridMCPA2AGateway"]
P2A["P2.1 Streaming<br/>AdaptiveStreamProcessor<br/>StreamingMultiplexer"]
P2B["P2.2 Connection<br/>MCPConnectionPool<br/>CircuitBreaker · QuotaManager"]
P3A["P3.1 Plugin<br/>PluginManager<br/>(hot-reload)"]
P3B["P3.2 Patterns<br/>WorkflowPatternRegistry<br/>PatternComposer"]
P4A["P4.1 Resilience<br/>ResilientExecutor<br/>RetryPolicy · FallbackChain<br/>StateRecovery"]
P4B["P4.2 Health<br/>HealthMonitor<br/>HealthCheck · AutoScaler"]
end
subgraph EXT["External surfaces"]
direction TB
PEER["A2A peer agents"]
LLM_API["LLM provider APIs"]
MCP_SRV["MCP servers"]
USER["Operator / SRE"]
end
APP --> AGENT --> LLM
AGENT --> WF
WF --> MCP
MCP --> MCP_SRV
LLM --> LLM_API
P1A -. negotiates .-> MCP
P1B -. bridges .-> MCP
P1B <--> PEER
P2A -. wraps streams .-> WF
P2B -. pools .-> MCP
P2B -. guards .-> LLM
P3A -. injects into .-> APP
P3B -. extends .-> WF
P4A -. wraps .-> WF
P4A -. wraps .-> P1B
P4B -. observes .-> P2B
P4B -. observes .-> P2A
P4B -. emits signals .-> USER
TRACE -. consumes .-> ENH
LOG -. consumes .-> ENH
classDef upstream fill:#F5F5F5,stroke:#999999,color:#333333
classDef enh fill:#EEF2FF,stroke:#425CC7,color:#1E1B4B
classDef ext fill:#FEF3C7,stroke:#D97706,color:#78350F
class APP,AGENT,LLM,WF,MCP,TRACE,LOG upstream
class P1A,P1B,P2A,P2B,P3A,P3B,P4A,P4B enh
class PEER,LLM_API,MCP_SRV,USER extLayered view
The four P-tier groups form a layered cake. P1 is the protocol foundation; P2 is the transport and resource layer; P3 is the extensibility layer; P4 is the resilience & operations layer that observes and protects everything below.
flowchart TB
subgraph L4["P4 — Resilience & Operations"]
HM["HealthMonitor + AutoScaler<br/>(EWMA · predictive)"]
RE["ResilientExecutor<br/>(retry · fallback · state recovery)"]
end
subgraph L3["P3 — Extensibility"]
PM["PluginManager<br/>(hot-reload)"]
PR["WorkflowPatternRegistry<br/>+ PatternComposer"]
end
subgraph L2["P2 — Transport & Resources"]
ASP["AdaptiveStreamProcessor<br/>(3 QoS tiers · backpressure)"]
CP["MCPConnectionPool<br/>+ CircuitBreaker + QuotaManager"]
end
subgraph L1["P1 — Protocol"]
PA["MCPProtocolAdapter<br/>(5 versions · deprecation)"]
A2A["A2A Gateway<br/>(discovery · task lifecycle)"]
end
subgraph L0["Foundation"]
RUNTIME["Upstream MCP agent runtime<br/>(MCPApp · Agent · AugmentedLLM · Workflows)"]
end
L4 --> L3 --> L2 --> L1 --> L0
HM -. observes .-> CP
HM -. observes .-> ASP
RE -. wraps .-> A2A
RE -. wraps .-> CP
PM -. injects .-> RUNTIME
PR -. extends .-> RUNTIME
classDef l0 fill:#F5F5F5,stroke:#999999,color:#333333
classDef l1 fill:#E0E7FF,stroke:#425CC7,color:#1E1B4B
classDef l2 fill:#C7D2FE,stroke:#425CC7,color:#1E1B4B
classDef l3 fill:#A5B4FC,stroke:#425CC7,color:#1E3A8A
classDef l4 fill:#818CF8,stroke:#312E81,color:#FFFFFF
class RUNTIME l0
class PA,A2A l1
class ASP,CP l2
class PM,PR l3
class HM,RE l4Workflow diagrams
Request lifecycle through the resilient executor
A typical agent request flows through the protocol adapter (negotiate version), the connection pool (acquire a pooled connection), the resilient executor (retry on failure, fall back to an A2A peer if needed), the adaptive stream processor (consume the LLM stream with QoS), and the health monitor (record latency & error rate). State snapshots are saved between steps so a retry can resume mid-flight instead of redoing completed work.
sequenceDiagram
autonumber
participant Caller as Caller
participant PA as MCPProtocolAdapter
participant CP as MCPConnectionPool
participant CB as CircuitBreaker
participant RE as ResilientExecutor
participant SR as StateRecovery
participant LLM as AugmentedLLM
participant ASP as AdaptiveStreamProcessor
participant HM as HealthMonitor
participant AS as AutoScaler
Caller->>PA: negotiate(version)
PA-->>Caller: NegotiatedCapabilities
Caller->>RE: execute_with_resilience(fn, workflow_id)
RE->>SR: load(workflow_id)
alt snapshot exists
SR-->>RE: snapshot(step=N)
RE->>RE: resume from step N
else no snapshot
RE->>RE: start from step 0
end
RE->>CP: acquire(target)
CP->>CB: before_call()
alt breaker OPEN
CB-->>CP: false (fast-fail)
CP-->>RE: CircuitBreakerOpenError
RE->>RE: fallback chain
else breaker CLOSED / HALF_OPEN
CB-->>CP: true
CP-->>RE: connection
RE->>LLM: generate_stream(prompt)
loop over tokens
LLM-->>ASP: token (QoS=BEST_EFFORT)
ASP-->>Caller: token
end
RE->>CP: release(connection)
CP->>CB: record_success()
RE->>SR: save(workflow_id, step, state)
end
par health observation
ASP->>HM: latency + error_rate
LLM->>HM: latency + error_rate
HM->>HM: EWMA update
alt status transition
HM->>AS: on_unhealthy / on_recovered
AS-->>Caller: ScaleSignal (UP / DOWN)
end
end
RE-->>Caller: result / ExecutionStatsA2A gateway: bridging peer agents into MCP
HybridMCPA2AGateway makes remote A2A agents appear as local MCP tools (named a2a__<agent_name>). The gateway handles agent discovery (via /.well-known/agent.json), task lifecycle (submitted → working → input-required → completed/canceled/failed), and transport selection (HTTP for production, in-proc for tests).
flowchart TB
subgraph CLIENT["MCP client"]
CALL["call_tool('a2a__researcher', input)"]
end
subgraph GW["HybridMCPA2AGateway"]
LT["list_tools()"]
DISP["dispatch(name, input)"]
REG["registry:<br/>name → A2AClient"]
end
subgraph A2A_PEER_A["A2A peer: researcher"]
AC1["AgentCard<br/>/.well-known/agent.json"]
TS1["A2AServer<br/>task lifecycle"]
end
subgraph A2A_PEER_B["A2A peer: coder"]
AC2["AgentCard"]
TS2["A2AServer"]
end
subgraph TRANSP["Transport"]
HTTP["httpx<br/>(production)"]
INPROC["in-proc<br/>(tests)"]
end
CALL --> DISP
LT --> DISP
DISP --> REG
REG -- "researcher" --> AC1
REG -- "coder" --> AC2
AC1 --> TS1
AC2 --> TS2
TS1 --> HTTP
TS2 --> HTTP
TS1 -. test .-> INPROC
TS2 -. test .-> INPROC
classDef client fill:#FEF3C7,stroke:#D97706,color:#78350F
classDef gw fill:#EEF2FF,stroke:#425CC7,color:#1E1B4B
classDef peer fill:#FCE7F3,stroke:#BE185D,color:#831843
classDef trans fill:#D1FAE5,stroke:#059669,color:#064E3B
class CALL client
class LT,DISP,REG gw
class AC1,TS1,AC2,TS2 peer
class HTTP,INPROC transCircuit breaker state machine
Three states, exponential backoff on recovery trips, on_trip async callback for the health monitor.
stateDiagram-v2
[*] --> CLOSED
CLOSED --> OPEN: failures ≥ failure_threshold
OPEN --> HALF_OPEN: open_timeout_s elapsed
HALF_OPEN --> CLOSED: success ≥ success_threshold
HALF_OPEN --> OPEN: any failure
note right of OPEN
Fast-fail all calls.
Backoff = base_s × 2^(trips-1)
capped at backoff_max_s.
Fires on_trip callback.
end note
note right of HALF_OPEN
Allow probe requests.
Reset backoff if recovery
succeeds.
end noteAdaptive streaming: QoS & backpressure
AdaptiveStreamProcessor is a bounded queue with three QoS tiers. When the queue is full, the per-tier policy decides whether to drop, block, or propagate backpressure to the autoscaler.
flowchart TB
subgraph PROD["Producer"]
P1["emit(item, qos)"]
end
subgraph Q["AdaptiveStreamProcessor (bounded queue)"]
DECIDE{"queue full?"}
T1["DROPPABLE<br/>priority=1"]
T2["BEST_EFFORT<br/>priority=5"]
T3["REALTIME<br/>priority=10"]
BUF["bounded buffer<br/>(maxsize)"]
end
subgraph CONS["Consumer"]
C1["async for item in proc.process()"]
end
subgraph REACT["Reactive hooks"]
DROP["drop oldest<br/>++dropped"]
BLOCK["block producer<br/>++backpressure_events"]
SCALE["on_backpressure()<br/>→ AutoScaler.SCALE_UP"]
end
P1 --> DECIDE
DECIDE -- yes --> T1
DECIDE -- yes --> T2
DECIDE -- yes --> T3
DECIDE -- no --> BUF
T1 --> DROP
T2 --> BLOCK
T3 --> SCALE
DROP --> BUF
BLOCK --> BUF
SCALE --> BUF
BUF --> C1
classDef prod fill:#FEF3C7,stroke:#D97706,color:#78350F
classDef q fill:#EEF2FF,stroke:#425CC7,color:#1E1B4B
classDef cons fill:#D1FAE5,stroke:#059669,color:#064E3B
classDef react fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D
class P1 prod
class DECIDE,T1,T2,T3,BUF q
class C1 cons
class DROP,BLOCK,SCALE reactHealth monitor & autoscaler feedback loop
The health monitor runs registered checks on a schedule, computes EWMA latency and EWMA error rate over the last 20 invocations, and fires on_unhealthy / on_recovered callbacks on transitions only (not on every check) to avoid alert storms. The autoscaler subscribes to those transitions and emits SCALE_UP / SCALE_DOWN signals with per-component cooldowns.
flowchart LR
subgraph TARGETS["Observed components"]
DB["db"]
CACHE["cache"]
LLM_API["LLM API"]
MCP_SRV["MCP server"]
end
subgraph MON["HealthMonitor"]
SCHED["schedule loop"]
EWMA["EWMA latency<br/>EWMA error_rate<br/>(window=20)"]
STATE["status per component<br/>HEALTHY / DEGRADED /<br/>UNHEALTHY / UNKNOWN"]
CB_ON["on_unhealthy<br/>(transition only)"]
CB_OFF["on_recovered<br/>(transition only)"]
end
subgraph SCALE["AutoScaler"]
DECIDE{"transition?"}
UP["SCALE_UP<br/>(UNHEALTHY)"]
DOWN["SCALE_DOWN<br/>(HEALTHY + cooldown)"]
HOLD["HOLD"]
end
TARGETS --> SCHED
SCHED --> EWMA
EWMA --> STATE
STATE -- degraded --> CB_ON
STATE -- healthy --> CB_OFF
CB_ON --> DECIDE
CB_OFF --> DECIDE
DECIDE -- unhealthy --> UP
DECIDE -- healthy + cooldown ok --> DOWN
DECIDE -- otherwise --> HOLD
UP -. feeds .-> TARGETS
DOWN -. feeds .-> TARGETS
classDef targets fill:#FEF3C7,stroke:#D97706,color:#78350F
classDef mon fill:#EEF2FF,stroke:#425CC7,color:#1E1B4B
classDef scale fill:#FCE7F3,stroke:#BE185D,color:#831843
class DB,CACHE,LLM_API,MCP_SRV targets
class SCHED,EWMA,STATE,CB_ON,CB_OFF mon
class DECIDE,UP,DOWN,HOLD scaleWhat's new in this release
Eight work-streams from the improvement plan are implemented as additive, self-contained modules. No existing call-site in mcp_agent.* was modified (except for one upstream-bug fix — see audit.md §2). The new code is isolated under src/mcp_agent/enhancements/:
ID | Work-stream | Module | Headline class / function |
P1.1 | Protocol negotiation & compatibility |
|
|
P1.2 | Agent-to-Agent (A2A) protocol |
|
|
P2.1 | Adaptive streaming with backpressure |
|
|
P2.2 | Connection pooling & circuit breaker |
|
|
P3.1 | Hot-reload plugin architecture |
|
|
P3.2 | Custom workflow pattern registry |
|
|
P4.1 | Resilient execution & state recovery |
|
|
P4.2 | Health monitoring & autoscaling |
|
|
A complete engineering record — including the why and how for each module, design alternatives considered, and reproduction instructions for every number claimed above — lives in audit.md (595 lines). A short consumer-facing manifest lives in ENHANCEMENTS.md.
Capability surface
from mcp_agent.enhancements import (
# P1.1 — protocol
MCPProtocolAdapter, CompatibilityLayer, LATEST_PROTOCOL_VERSION,
# P1.2 — A2A
AgentCard, A2AClient, A2AServer, HybridMCPA2AGateway, A2ATask, TaskState,
# P2.1 — streaming
AdaptiveStreamProcessor, StreamingMultiplexer, QoSTier, StreamStats,
# P2.2 — connection
MCPConnectionPool, CircuitBreaker, CircuitState, QuotaManager,
# P3.1 — plugin
Plugin, PluginManager, load_plugin,
# P3.2 — workflow patterns
WorkflowPatternRegistry, register_workflow_pattern, PatternComposer,
# P4.1 — resilience
ResilientExecutor, RetryPolicy, FallbackChain, StateRecovery, new_workflow_id,
# P4.2 — health
HealthMonitor, HealthCheck, HealthStatus, AutoScaler, ScaleDecision,
)Every symbol above is covered by unit tests; see tests/enhancements/ for 109 working examples.
Quickstart
Install
# clone
git clone <this-repo> mcp-agent && cd mcp-agent
# install with uv (recommended)
uv sync
# or with pip + venv
python -m venv .venv && source .venv/bin/activate
pip install -e ".[anthropic,openai]"
pip install -e ".[dev]"Run a basic agent (upstream runtime, unchanged)
import asyncio
from mcp_agent.app import MCPApp
from mcp_agent.agents.agent import Agent
from mcp_agent.workflows.llm.augmented_llm_openai import OpenAIAugmentedLLM
app = MCPApp(name="hello")
async def main() -> None:
async with app.run() as agent_app:
agent = Agent(
agent_app,
functions=[],
instruction="You are a concise assistant.",
)
llm = await agent.attach_llm(OpenAIAugmentedLLM)
print(await llm.generate_str("Say hello in one sentence."))
asyncio.run(main())Use the new reliability layer
import asyncio
from mcp_agent.enhancements import (
AdaptiveStreamProcessor, QoSTier,
CircuitBreaker, CircuitState,
HealthMonitor, HealthCheck, HealthStatus,
)
async def main():
# adaptive streaming with 3 QoS tiers + backpressure
proc = AdaptiveStreamProcessor(maxsize=1024)
async def producer():
for i in range(10_000):
await proc.put(i, qos=QoSTier.BEST_EFFORT)
await proc.close()
async def consumer():
async for item in proc.process():
...
await asyncio.gather(producer(), consumer())
print(proc.stats) # StreamStats(items_in=10_000, items_out=10_000, ...)
# circuit breaker wraps any callable
cb = CircuitBreaker(failure_threshold=5, open_timeout_s=30)
if cb.before_call():
try:
... # do work
cb.record_success()
except Exception:
cb.record_failure()
# health monitor with EWMA + autoscaler hook
monitor = HealthMonitor()
monitor.register("db", HealthCheck(check=lambda: (HealthStatus.HEALTHY, "ok")))
await monitor.check_once()
asyncio.run(main())See examples/ (upstream) and src/mcp_agent/enhancements/examples/ (new bundled demos) for more.
Architecture deep-dive
P1.1 — Protocol negotiation (enhancements/protocol/)
Five MCP protocol versions are now first-class: 1.0, 1.20, 2.0, 2.1, plus A2A v0.3. MCPProtocolAdapter selects the highest mutually-supported version between client and server, normalises capabilities into a NegotiatedCapabilities object, and surfaces DEPRECATED_IN_V2 / V2_ONLY_FEATURES lists. CompatibilityLayer wraps a session and:
emits
DeprecationWarningwhen a v1-only call (roots/list,resources/list) is made on a v2 session, so consumers get a soft migration signal;raises
ProtocolFeatureUnavailablewhen a v2-only call is attempted on a v1 session, so callers fail fast instead of producing silent no-ops.
This module is pure-python and has no I/O — it can be unit-tested without a live MCP server.
P1.2 — Agent-to-Agent protocol (enhancements/a2a/)
Implements the A2A v0.3 spec (agent discovery via /.well-known/agent.json, task lifecycle with submitted → working → input-required → completed/canceled/failed, in-proc and HTTP transports). The headline class is HybridMCPA2AGateway, which bridges any A2A agent into an MCP tool surface — remote A2A agents appear as a2a__<agent_name> MCP tools. This lets a single MCP client orchestrate a fleet of A2A peers without changing its calling code.
A2AClient supports both transport="http" (httpx-backed, production use) and transport="inproc" (for tests and side-effect-free composition). send_task_and_wait() polls the task lifecycle with backoff until it reaches a terminal state.
P2.1 — Adaptive streaming with backpressure (enhancements/streaming/)
AdaptiveStreamProcessor is a bounded asyncio.Queue with three QoS tiers:
Tier | Behaviour when full |
| drop the oldest item; bump |
| block the producer (classic backpressure) |
| block + invoke |
StreamStats exposes items_in, items_out, dropped, backpressure_events, backpressure_ms, throughput_per_sec. StreamingMultiplexer is a weighted round-robin fan-in over multiple named sources — useful for merging telemetry streams from N agents.
P2.2 — Connection pooling, circuit breaker, quota (enhancements/connection/)
MCPConnectionPool maintains per-target bounded pools with a global cap (semaphore). Idle connections are reused; broken ones are reaped via the cleanup callback. Each target has its own CircuitBreaker.
CircuitBreaker is a 3-state (CLOSED / OPEN / HALF_OPEN) breaker with exponential backoff on recovery (backoff_base_s * 2^(trips-1), capped at backoff_max_s). State transitions emit on_trip async callbacks so the health monitor can react.
QuotaManager provides per-key semaphores + a token-bucket rate-limiter + a max-total counter — useful for protecting an upstream LLM API from being melted by a misbehaving workflow.
P3.1 — Hot-reload plugin architecture (enhancements/plugin/)
Plugin is a minimal base class with async setup(app) and async teardown(). PluginManager loads plugins from a dotted path (pkg.mod:Class) or a filesystem path (./my_plugin.py), supports unload() with graceful teardown, and hot-reloads changed plugins without restarting the process.
Hot-reload uses watchdog if available (with a 250 ms debounce handler), and falls back to a content-hash based polling loop otherwise — the polling path matters because some filesystems don't deliver watchdog events reliably.
P3.2 — Workflow pattern registry & composer (enhancements/workflow_patterns/)
WorkflowPatternRegistry is a first-registration-wins registry of named patterns. The @register_workflow_pattern("name") class decorator lets downstream code declare new patterns idiomatically:
@register_workflow_pattern("my_pipeline")
class MyPipeline(WorkflowPattern):
async def execute(self, input):
...PatternComposer chains patterns sequentially, forwarding each output as the next input. None outputs are skipped — this lets optional steps fall out of a chain cleanly.
P4.1 — Resilient executor & state recovery (enhancements/resilience/)
ResilientExecutor wraps an async callable with retry → fallback → state-recovery semantics:
RetryPolicy — exponential backoff (
base_delay * multiplier^attempt, capped atmax_delay) + jitter;is_retriable(exc)filter;FallbackChain — ordered
predicate → fnpairs; the first matching predicate wins, otherwise returns(False, None);StateRecovery —
save(workflow_id, step, state)/load(workflow_id)in-memory snapshot store (subclassable for Redis/DB); on retry, the executor resumes from the latest snapshot instead of redoing completed steps.
ExecutionStats reports attempts, successes, failures, fallbacks used, total delay spent in backoff, and the last error.
P4.2 — Health monitoring & autoscaling (enhancements/health/)
HealthCheck wraps an async check() → (HealthStatus, detail) callable. Internally it tracks EWMA latency and EWMA error rate over the last 20 invocations, and degrades the reported status based on configurable thresholds (latency_warn_ms, latency_unhealthy_ms, error_rate_threshold). The "predictive" bit: if the EWMA error rate exceeds the threshold, the check is marked UNHEALTHY even if the most recent call succeeded — this catches slow-burn degradation that point-in-time thresholds miss.
HealthMonitor runs all registered checks on a schedule and fires on_unhealthy / on_recovered callbacks on transitions (not on every check) to avoid alert storms. AutoScaler subscribes to the monitor: UNHEALTHY → SCALE_UP, HEALTHY + cooldown → SCALE_DOWN. Per-component cooldowns prevent thrash.
Benchmark methodology
Benchmarks live under scripts/enhancements_benchmarks/:
capture_baseline.py— runs the upstream test suite, computes pass rate, coverage, capability probes, and a naive streaming throughput upper-bound (no work per item).capture_enhanced.py— runs the enhanced suite, computes pass rate ontests/enhancements/, coverage onsrc/mcp_agent/enhancements/, and the adaptive streaming throughput (real per-item work).bench_streaming.py— direct throughput comparison of naiveasyncio.QueuevsAdaptiveStreamProcessoracross QoS tiers.
All benchmarks use asyncio-native timing (no time.time() jitter), warm up for 1 000 iterations, then measure 10 000 iterations. Throughput numbers are reported as items / wall_time_s. Coverage is measured with pytest-cov configured via the Makefile (CLI excluded to match upstream's coverage scope).
Run them yourself:
make coverage # upstream-style coverage (CLI excluded)
python scripts/enhancements_benchmarks/capture_baseline.py
python scripts/enhancements_benchmarks/capture_enhanced.py
python scripts/enhancements_benchmarks/bench_streaming.pyEvaluation methodology
Evaluation has three tiers:
Unit tests — every public class has its own module under
tests/enhancements/. 109 tests, 0 regressions, 82 % coverage on the new package.End-to-end scenarios —
tests/enhancements/test_end_to_end.pyruns 4 cross-cutting scenarios that compose multiple subsystems (e.g. protocol negotiation → connection pool → resilient executor with A2A fallback → adaptive streaming → health monitor + autoscaler) to prove the modules interoperate, not just pass in isolation.Performance regression tests —
tests/enhancements/test_perf_regression.pyasserts the adaptive streaming throughput stays within ±30 % of the recorded baseline and well above the plan's "100 msg/s" floor. These tests fail loudly if a refactor regresses throughput.
All three tiers run in CI via pytest tests/enhancements/ and via the Makefile's tests target.
Fault injection
Faults are injected in-test, not via a separate chaos engineering tool — this keeps the test suite self-contained and reproducible without external dependencies.
Fault | Where it's injected | What it proves |
Slow consumer (backpressure) |
| Producer is blocked, no items dropped |
Overload on |
| Oldest items dropped, throughput preserved |
Repeated downstream failure |
| Breaker opens after N failures, fast-fails subsequent calls |
Breaker recovery |
| Half-open → closed transition on success |
Rate-limit exceeded |
| Quota semaphore blocks, releases on release |
Plugin file change |
| Old instance torn down, new instance set up, counter persists |
Retry-then-success |
| Exponential backoff between attempts, success on attempt N |
Fallback chain |
| First matching predicate wins, downstream fallbacks skipped |
State recovery |
| Resumes from snapshot, doesn't redo completed step |
Health degradation |
| EWMA error rate degrades status even on intermittent success |
Autoscaler signal |
|
|
Observability
Three layers of observability are wired into the platform:
Structured logging — the upstream
mcp_agent.loggingpackage (Rich-based) is unchanged; the new modules emit structured log records via the same logger so downstream collectors see a unified stream.OpenTelemetry tracing — the upstream
mcp_agent.tracingpackage (OTLP exporter, semconv, token counter) is unchanged; new modules emit spans with stable names (enhancements.streaming.process,enhancements.connection.acquire,enhancements.resilience.execute_with_resilience, etc.) so dashboards work out of the box.Health & autoscaling signals —
HealthMonitorexposesHealthCheckResultobjects with EWMA latency, EWMA error rate, and currentHealthStatus.AutoScalerexposesScaleSignalevents. Both can be fed to Prometheus via a thin exporter (left as an integration exercise — seeaudit.md§6 for what is deliberately out of scope).
Measured results
All numbers below are reproducible from the repository — see Reproducibility for exact commands.
Test pass rate
Suite | Pass | Fail | Error | Pass rate |
Upstream baseline (cloned at | 1292 | 100 | 107 | 85.9 % |
After upstream-bug fix (no new code) | 1494 | 5 | 4 | 99.4 % |
After enhancements (this fork) | 1602 | 6 | 4 | 99.4 % |
The 85.9 % → 99.4 % jump on the upstream baseline comes from fixing the @abstractmethod generate_stream regression (see audit.md §2). The 1494 → 1602 jump comes from 109 new enhancement tests with zero regressions.
The remaining 6 failures are pre-existing environmental drift (mimetypes lib mismatch, boto3 stub mismatch, asyncio loop policy on the test host) — none are caused by the new code.
Coverage
Scope | Line coverage |
| 55 % |
| 82 % |
Coverage is configured via the Makefile (coverage run --omit="src/mcp_agent/cli/**" -m pytest tests -m "not integration").
Streaming throughput
Implementation | Throughput (msg/s) | Notes |
Naive | 352 000 | Baseline |
| 470 000 | +34 % over baseline |
Plan's stated baseline | 100 | 4 700× over the floor |
The adaptive processor is faster than the naive baseline because it batches StreamStats updates and uses a tier-aware enqueue path that avoids unnecessary asyncio.sleep(0) yields.
Connection pool
Metric | Naive (new connection per call) | Pooled |
Factory calls for 1 000 acquire/release cycles | 1 000 | ~250 (4× reduction) |
Avg acquire latency (μs) | ~820 | ~210 |
Capability surface
Capability | Before | After |
Multi-version protocol negotiation | ❌ | ✅ (5 versions) |
Agent-to-Agent discovery | ❌ | ✅ (A2A v0.3) |
Adaptive streaming with QoS | ❌ | ✅ (3 tiers) |
Connection pooling | ❌ | ✅ |
Circuit breaker | ❌ | ✅ (3-state, EWMA) |
Hot-reload plugins | ❌ | ✅ (watchdog + polling) |
Workflow pattern registry | ❌ | ✅ (decorator-based) |
Resilient executor with state recovery | ❌ | ✅ |
Health monitor with autoscaler | ❌ | ✅ |
Reproducibility
Every number above is reproducible from a clean checkout:
# 1. install
uv sync
uv pip install -e ".[dev]"
# 2. test pass rate (whole suite)
pytest tests/ -q
# 3. coverage on the new package
make coverage
# or: pytest tests/enhancements/ --cov=src/mcp_agent/enhancements --cov-report=term
# 4. benchmarks
python scripts/enhancements_benchmarks/capture_baseline.py
python scripts/enhancements_benchmarks/capture_enhanced.py
python scripts/enhancements_benchmarks/bench_streaming.py
# 5. per-subsystem tests
pytest tests/enhancements/test_protocol_adapter.py -v
pytest tests/enhancements/test_a2a.py -v
pytest tests/enhancements/test_streaming.py -v
pytest tests/enhancements/test_connection_pool.py -v
pytest tests/enhancements/test_plugin_manager.py -v
pytest tests/enhancements/test_workflow_patterns.py -v
pytest tests/enhancements/test_resilience.py -v
pytest tests/enhancements/test_health_monitor.py -v
pytest tests/enhancements/test_end_to_end.py -v
pytest tests/enhancements/test_perf_regression.py -vExpected wall-clock for the full enhancement suite on a modern laptop: ~25 s. Expected wall-clock for the full repository suite: ~3 min.
Project layout
.
├── LICENSE # Apache-2.0 (with attribution)
├── NOTICE # third-party attribution
├── README.md # this file
├── audit.md # engineering record of every change
├── ENHANCEMENTS.md # short consumer-facing manifest
├── CONTRIBUTING.md
├── SECURITY.md
├── pyproject.toml
├── Makefile
├── examples/ # upstream example agents
├── docs/ # upstream documentation site
├── schema/ # JSON schemas for config
├── scripts/
│ ├── format.py lint.py gen_schema.py promptify.py
│ └── enhancements_benchmarks/ # new — benchmark scripts
│ ├── capture_baseline.py
│ ├── capture_enhanced.py
│ └── bench_streaming.py
├── src/mcp_agent/
│ ├── app.py config.py console.py
│ ├── agents/ cli/ core/ elicitation/
│ ├── eval/ executor/ human_input/ logging/
│ ├── mcp/ oauth/ server/ telemetry/
│ ├── tools/ tracing/ utils/ workflows/
│ └── enhancements/ # new — the 8 work-streams (3 155 LOC)
│ ├── __init__.py
│ ├── protocol/ # P1.1
│ ├── a2a/ # P1.2
│ ├── streaming/ # P2.1
│ ├── connection/ # P2.2
│ ├── plugin/ # P3.1
│ ├── workflow_patterns/ # P3.2
│ ├── resilience/ # P4.1
│ ├── health/ # P4.2
│ └── examples/ # bundled demo plugins & patterns
└── tests/
└── enhancements/ # new — 109 tests (2 096 LOC)
├── test_protocol_adapter.py
├── test_a2a.py
├── test_streaming.py
├── test_connection_pool.py
├── test_plugin_manager.py
├── test_workflow_patterns.py
├── test_resilience.py
├── test_health_monitor.py
├── test_end_to_end.py
└── test_perf_regression.pyTesting
# upstream suite (sanity check — should match the numbers in audit.md)
pytest tests/ -q
# enhancement suite only
pytest tests/enhancements/ -v
# with coverage
make coverageThe enhancement suite is hermetic — it does not require a live LLM API key, MCP server, or A2A peer. The inproc A2A transport and the polling-based plugin watcher mean every test runs in-process and finishes in milliseconds.
For the upstream suite, some integration tests require API keys; they are marked @pytest.mark.integration and skipped by default.
Attribution
This project extends lastmile-ai/mcp-agent, licensed under the Apache License, Version 2.0. The reliability, benchmarking, evaluation, observability, fault-injection, testing, infrastructure and reporting layers in this repository are independently developed extensions.
Specifically, the following are independently developed under this project:
src/mcp_agent/enhancements/(the entire package)tests/enhancements/(the entire test directory)scripts/enhancements_benchmarks/audit.md,ENHANCEMENTS.md,NOTICE, and this README
The upstream lastmile-ai/mcp-agent source remains under its original Apache-2.0 license; modifications to upstream files are limited to a single bug fix documented in audit.md §2 and are clearly marked.
License
Licensed under the Apache License, Version 2.0 — see LICENSE for the full text. Third-party attributions are listed in NOTICE.
This server cannot be installed
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Remote MCP for A2A failure replay MCP, structured receipts, audit logs, and reviewer-ready evidence.
Workflow diagnostics, capability routing, and x402 settlement for MCP-compatible agents.
Control plane for autonomous software labor. Agents claim objectives over MCP with audit trail.
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/Akgithub2028/Agent-Reliability-and-Evaluation-Lab'
If you have feedback or need assistance with the MCP directory API, please join our Discord server