Skip to main content
Glama

AREL — Agent Reliability & Evaluation Lab

A production-oriented reliability, benchmarking and evaluation system built around an MCP agent runtime.


Why this exists

The upstream MCP agent runtime is an excellent framework for composing agents with Model Context Protocol — but shipping it to production demands more than composition primitives. Real production systems need multi-version protocol negotiation, peer-to-peer agent discovery, backpressure-aware streaming, connection pooling with circuit breakers, hot-reloadable plugin architectures, resilient retry & state recovery, and continuous health monitoring with autoscaling.

AREL keeps the upstream runtime as a foundation and layers a complete reliability, benchmarking and evaluation platform on top:

  • 5 protocol versions supported (MCP 1.0 → 2.1 + A2A v0.3) with formal negotiation and deprecation warnings

  • 9 new subsystems implemented as additive, non-invasive modules (3 155 LOC of source, 2 096 LOC of tests)

  • 109 new tests added — 0 regressions, suite pass rate restored from 85.9 % → 99.4 %

  • 82 % line coverage on the new enhancements package

  • 470 k msg/s adaptive streaming throughput (+34 % over the naive baseline)


At a glance

Capability

Before

After

Test pass rate

1292 / 1503 (85.9 %)

1602 / 1612 (99.4 %)

New tests added

109 (0 regressions)

Coverage on enhancements/

n/a

82 %

MCP protocol versions

1.0 – 1.20

1.0 – 2.1 + A2A v0.3

Adaptive streaming throughput

352 k msg/s (naive)

470 k msg/s (+34 %)

Connection pool reuse

~4× reduction in factory calls

New subsystems

9 (see Capability Surface)

Source LOC added

3 155 (12 files)

Test LOC added

2 096 (11 files)


Architecture

Architecture
├── Benchmark methodology
├── Evaluation methodology
├── Fault injection
├── Observability
├── Measured results
└── Reproducibility

Each branch above is realised by a concrete module under src/mcp_agent/enhancements/ and is exercised by the test suite under tests/enhancements/. The branches are not marketing copy — they map 1:1 to files, classes and measurable numbers (see Measured results below).


System architecture

The diagram below shows how AREL sits on top of the upstream MCP agent runtime. The upstream mcp_agent.* package (left, in grey) provides composition primitives; the new mcp_agent.enhancements.* package (right, in colour) provides the reliability, benchmarking and evaluation platform. The two are wired together by the HybridMCPA2AGateway and the ResilientExecutor — everything else is additive and can be adopted incrementally.

flowchart LR
    subgraph UP["Upstream mcp_agent runtime (Apache-2.0, unchanged)"]
        direction TB
        APP["MCPApp<br/>context & lifecycle"]
        AGENT["Agent + AgentSpec"]
        LLM["AugmentedLLM<br/>(OpenAI/Anthropic/Bedrock/…)"]
        WF["Workflows<br/>orchestrator · router · parallel · swarm"]
        MCP["MCP client<br/>stdio + HTTP transports"]
        TRACE["OpenTelemetry tracing"]
        LOG["Rich structured logging"]
    end

    subgraph ENH["AREL enhancements package (this project)"]
        direction TB
        P1A["P1.1 Protocol<br/>MCPProtocolAdapter"]
        P1B["P1.2 A2A<br/>AgentCard · A2AClient<br/>HybridMCPA2AGateway"]
        P2A["P2.1 Streaming<br/>AdaptiveStreamProcessor<br/>StreamingMultiplexer"]
        P2B["P2.2 Connection<br/>MCPConnectionPool<br/>CircuitBreaker · QuotaManager"]
        P3A["P3.1 Plugin<br/>PluginManager<br/>(hot-reload)"]
        P3B["P3.2 Patterns<br/>WorkflowPatternRegistry<br/>PatternComposer"]
        P4A["P4.1 Resilience<br/>ResilientExecutor<br/>RetryPolicy · FallbackChain<br/>StateRecovery"]
        P4B["P4.2 Health<br/>HealthMonitor<br/>HealthCheck · AutoScaler"]
    end

    subgraph EXT["External surfaces"]
        direction TB
        PEER["A2A peer agents"]
        LLM_API["LLM provider APIs"]
        MCP_SRV["MCP servers"]
        USER["Operator / SRE"]
    end

    APP --> AGENT --> LLM
    AGENT --> WF
    WF --> MCP
    MCP --> MCP_SRV
    LLM --> LLM_API

    P1A -. negotiates .-> MCP
    P1B -. bridges .-> MCP
    P1B <--> PEER
    P2A -. wraps streams .-> WF
    P2B -. pools .-> MCP
    P2B -. guards .-> LLM
    P3A -. injects into .-> APP
    P3B -. extends .-> WF
    P4A -. wraps .-> WF
    P4A -. wraps .-> P1B
    P4B -. observes .-> P2B
    P4B -. observes .-> P2A
    P4B -. emits signals .-> USER

    TRACE -. consumes .-> ENH
    LOG -. consumes .-> ENH

    classDef upstream fill:#F5F5F5,stroke:#999999,color:#333333
    classDef enh fill:#EEF2FF,stroke:#425CC7,color:#1E1B4B
    classDef ext fill:#FEF3C7,stroke:#D97706,color:#78350F

    class APP,AGENT,LLM,WF,MCP,TRACE,LOG upstream
    class P1A,P1B,P2A,P2B,P3A,P3B,P4A,P4B enh
    class PEER,LLM_API,MCP_SRV,USER ext

Layered view

The four P-tier groups form a layered cake. P1 is the protocol foundation; P2 is the transport and resource layer; P3 is the extensibility layer; P4 is the resilience & operations layer that observes and protects everything below.

flowchart TB
    subgraph L4["P4 — Resilience & Operations"]
        HM["HealthMonitor + AutoScaler<br/>(EWMA · predictive)"]
        RE["ResilientExecutor<br/>(retry · fallback · state recovery)"]
    end

    subgraph L3["P3 — Extensibility"]
        PM["PluginManager<br/>(hot-reload)"]
        PR["WorkflowPatternRegistry<br/>+ PatternComposer"]
    end

    subgraph L2["P2 — Transport & Resources"]
        ASP["AdaptiveStreamProcessor<br/>(3 QoS tiers · backpressure)"]
        CP["MCPConnectionPool<br/>+ CircuitBreaker + QuotaManager"]
    end

    subgraph L1["P1 — Protocol"]
        PA["MCPProtocolAdapter<br/>(5 versions · deprecation)"]
        A2A["A2A Gateway<br/>(discovery · task lifecycle)"]
    end

    subgraph L0["Foundation"]
        RUNTIME["Upstream MCP agent runtime<br/>(MCPApp · Agent · AugmentedLLM · Workflows)"]
    end

    L4 --> L3 --> L2 --> L1 --> L0

    HM -. observes .-> CP
    HM -. observes .-> ASP
    RE -. wraps .-> A2A
    RE -. wraps .-> CP
    PM -. injects .-> RUNTIME
    PR -. extends .-> RUNTIME

    classDef l0 fill:#F5F5F5,stroke:#999999,color:#333333
    classDef l1 fill:#E0E7FF,stroke:#425CC7,color:#1E1B4B
    classDef l2 fill:#C7D2FE,stroke:#425CC7,color:#1E1B4B
    classDef l3 fill:#A5B4FC,stroke:#425CC7,color:#1E3A8A
    classDef l4 fill:#818CF8,stroke:#312E81,color:#FFFFFF

    class RUNTIME l0
    class PA,A2A l1
    class ASP,CP l2
    class PM,PR l3
    class HM,RE l4

Workflow diagrams

Request lifecycle through the resilient executor

A typical agent request flows through the protocol adapter (negotiate version), the connection pool (acquire a pooled connection), the resilient executor (retry on failure, fall back to an A2A peer if needed), the adaptive stream processor (consume the LLM stream with QoS), and the health monitor (record latency & error rate). State snapshots are saved between steps so a retry can resume mid-flight instead of redoing completed work.

sequenceDiagram
    autonumber
    participant Caller as Caller
    participant PA as MCPProtocolAdapter
    participant CP as MCPConnectionPool
    participant CB as CircuitBreaker
    participant RE as ResilientExecutor
    participant SR as StateRecovery
    participant LLM as AugmentedLLM
    participant ASP as AdaptiveStreamProcessor
    participant HM as HealthMonitor
    participant AS as AutoScaler

    Caller->>PA: negotiate(version)
    PA-->>Caller: NegotiatedCapabilities

    Caller->>RE: execute_with_resilience(fn, workflow_id)
    RE->>SR: load(workflow_id)
    alt snapshot exists
        SR-->>RE: snapshot(step=N)
        RE->>RE: resume from step N
    else no snapshot
        RE->>RE: start from step 0
    end

    RE->>CP: acquire(target)
    CP->>CB: before_call()
    alt breaker OPEN
        CB-->>CP: false (fast-fail)
        CP-->>RE: CircuitBreakerOpenError
        RE->>RE: fallback chain
    else breaker CLOSED / HALF_OPEN
        CB-->>CP: true
        CP-->>RE: connection
        RE->>LLM: generate_stream(prompt)
        loop over tokens
            LLM-->>ASP: token (QoS=BEST_EFFORT)
            ASP-->>Caller: token
        end
        RE->>CP: release(connection)
        CP->>CB: record_success()
        RE->>SR: save(workflow_id, step, state)
    end

    par health observation
        ASP->>HM: latency + error_rate
        LLM->>HM: latency + error_rate
        HM->>HM: EWMA update
        alt status transition
            HM->>AS: on_unhealthy / on_recovered
            AS-->>Caller: ScaleSignal (UP / DOWN)
        end
    end

    RE-->>Caller: result / ExecutionStats

A2A gateway: bridging peer agents into MCP

HybridMCPA2AGateway makes remote A2A agents appear as local MCP tools (named a2a__<agent_name>). The gateway handles agent discovery (via /.well-known/agent.json), task lifecycle (submitted → working → input-required → completed/canceled/failed), and transport selection (HTTP for production, in-proc for tests).

flowchart TB
    subgraph CLIENT["MCP client"]
        CALL["call_tool('a2a__researcher', input)"]
    end

    subgraph GW["HybridMCPA2AGateway"]
        LT["list_tools()"]
        DISP["dispatch(name, input)"]
        REG["registry:<br/>name → A2AClient"]
    end

    subgraph A2A_PEER_A["A2A peer: researcher"]
        AC1["AgentCard<br/>/.well-known/agent.json"]
        TS1["A2AServer<br/>task lifecycle"]
    end

    subgraph A2A_PEER_B["A2A peer: coder"]
        AC2["AgentCard"]
        TS2["A2AServer"]
    end

    subgraph TRANSP["Transport"]
        HTTP["httpx<br/>(production)"]
        INPROC["in-proc<br/>(tests)"]
    end

    CALL --> DISP
    LT --> DISP
    DISP --> REG
    REG -- "researcher" --> AC1
    REG -- "coder" --> AC2
    AC1 --> TS1
    AC2 --> TS2
    TS1 --> HTTP
    TS2 --> HTTP
    TS1 -. test .-> INPROC
    TS2 -. test .-> INPROC

    classDef client fill:#FEF3C7,stroke:#D97706,color:#78350F
    classDef gw fill:#EEF2FF,stroke:#425CC7,color:#1E1B4B
    classDef peer fill:#FCE7F3,stroke:#BE185D,color:#831843
    classDef trans fill:#D1FAE5,stroke:#059669,color:#064E3B

    class CALL client
    class LT,DISP,REG gw
    class AC1,TS1,AC2,TS2 peer
    class HTTP,INPROC trans

Circuit breaker state machine

Three states, exponential backoff on recovery trips, on_trip async callback for the health monitor.

stateDiagram-v2
    [*] --> CLOSED

    CLOSED --> OPEN: failures ≥ failure_threshold
    OPEN --> HALF_OPEN: open_timeout_s elapsed
    HALF_OPEN --> CLOSED: success ≥ success_threshold
    HALF_OPEN --> OPEN: any failure

    note right of OPEN
        Fast-fail all calls.
        Backoff = base_s × 2^(trips-1)
        capped at backoff_max_s.
        Fires on_trip callback.
    end note

    note right of HALF_OPEN
        Allow probe requests.
        Reset backoff if recovery
        succeeds.
    end note

Adaptive streaming: QoS & backpressure

AdaptiveStreamProcessor is a bounded queue with three QoS tiers. When the queue is full, the per-tier policy decides whether to drop, block, or propagate backpressure to the autoscaler.

flowchart TB
    subgraph PROD["Producer"]
        P1["emit(item, qos)"]
    end

    subgraph Q["AdaptiveStreamProcessor (bounded queue)"]
        DECIDE{"queue full?"}
        T1["DROPPABLE<br/>priority=1"]
        T2["BEST_EFFORT<br/>priority=5"]
        T3["REALTIME<br/>priority=10"]
        BUF["bounded buffer<br/>(maxsize)"]
    end

    subgraph CONS["Consumer"]
        C1["async for item in proc.process()"]
    end

    subgraph REACT["Reactive hooks"]
        DROP["drop oldest<br/>++dropped"]
        BLOCK["block producer<br/>++backpressure_events"]
        SCALE["on_backpressure()<br/>→ AutoScaler.SCALE_UP"]
    end

    P1 --> DECIDE
    DECIDE -- yes --> T1
    DECIDE -- yes --> T2
    DECIDE -- yes --> T3
    DECIDE -- no --> BUF

    T1 --> DROP
    T2 --> BLOCK
    T3 --> SCALE

    DROP --> BUF
    BLOCK --> BUF
    SCALE --> BUF

    BUF --> C1

    classDef prod fill:#FEF3C7,stroke:#D97706,color:#78350F
    classDef q fill:#EEF2FF,stroke:#425CC7,color:#1E1B4B
    classDef cons fill:#D1FAE5,stroke:#059669,color:#064E3B
    classDef react fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D

    class P1 prod
    class DECIDE,T1,T2,T3,BUF q
    class C1 cons
    class DROP,BLOCK,SCALE react

Health monitor & autoscaler feedback loop

The health monitor runs registered checks on a schedule, computes EWMA latency and EWMA error rate over the last 20 invocations, and fires on_unhealthy / on_recovered callbacks on transitions only (not on every check) to avoid alert storms. The autoscaler subscribes to those transitions and emits SCALE_UP / SCALE_DOWN signals with per-component cooldowns.

flowchart LR
    subgraph TARGETS["Observed components"]
        DB["db"]
        CACHE["cache"]
        LLM_API["LLM API"]
        MCP_SRV["MCP server"]
    end

    subgraph MON["HealthMonitor"]
        SCHED["schedule loop"]
        EWMA["EWMA latency<br/>EWMA error_rate<br/>(window=20)"]
        STATE["status per component<br/>HEALTHY / DEGRADED /<br/>UNHEALTHY / UNKNOWN"]
        CB_ON["on_unhealthy<br/>(transition only)"]
        CB_OFF["on_recovered<br/>(transition only)"]
    end

    subgraph SCALE["AutoScaler"]
        DECIDE{"transition?"}
        UP["SCALE_UP<br/>(UNHEALTHY)"]
        DOWN["SCALE_DOWN<br/>(HEALTHY + cooldown)"]
        HOLD["HOLD"]
    end

    TARGETS --> SCHED
    SCHED --> EWMA
    EWMA --> STATE
    STATE -- degraded --> CB_ON
    STATE -- healthy --> CB_OFF
    CB_ON --> DECIDE
    CB_OFF --> DECIDE
    DECIDE -- unhealthy --> UP
    DECIDE -- healthy + cooldown ok --> DOWN
    DECIDE -- otherwise --> HOLD

    UP -. feeds .-> TARGETS
    DOWN -. feeds .-> TARGETS

    classDef targets fill:#FEF3C7,stroke:#D97706,color:#78350F
    classDef mon fill:#EEF2FF,stroke:#425CC7,color:#1E1B4B
    classDef scale fill:#FCE7F3,stroke:#BE185D,color:#831843

    class DB,CACHE,LLM_API,MCP_SRV targets
    class SCHED,EWMA,STATE,CB_ON,CB_OFF mon
    class DECIDE,UP,DOWN,HOLD scale

What's new in this release

Eight work-streams from the improvement plan are implemented as additive, self-contained modules. No existing call-site in mcp_agent.* was modified (except for one upstream-bug fix — see audit.md §2). The new code is isolated under src/mcp_agent/enhancements/:

ID

Work-stream

Module

Headline class / function

P1.1

Protocol negotiation & compatibility

enhancements/protocol/

MCPProtocolAdapter, CompatibilityLayer

P1.2

Agent-to-Agent (A2A) protocol

enhancements/a2a/

AgentCard, A2AClient, A2AServer, HybridMCPA2AGateway

P2.1

Adaptive streaming with backpressure

enhancements/streaming/

AdaptiveStreamProcessor, StreamingMultiplexer, QoSTier

P2.2

Connection pooling & circuit breaker

enhancements/connection/

MCPConnectionPool, CircuitBreaker, QuotaManager

P3.1

Hot-reload plugin architecture

enhancements/plugin/

Plugin, PluginManager, load_plugin()

P3.2

Custom workflow pattern registry

enhancements/workflow_patterns/

WorkflowPatternRegistry, @register_workflow_pattern, PatternComposer

P4.1

Resilient execution & state recovery

enhancements/resilience/

ResilientExecutor, RetryPolicy, FallbackChain, StateRecovery

P4.2

Health monitoring & autoscaling

enhancements/health/

HealthMonitor, HealthCheck, AutoScaler

A complete engineering record — including the why and how for each module, design alternatives considered, and reproduction instructions for every number claimed above — lives in audit.md (595 lines). A short consumer-facing manifest lives in ENHANCEMENTS.md.


Capability surface

from mcp_agent.enhancements import (
    # P1.1 — protocol
    MCPProtocolAdapter, CompatibilityLayer, LATEST_PROTOCOL_VERSION,
    # P1.2 — A2A
    AgentCard, A2AClient, A2AServer, HybridMCPA2AGateway, A2ATask, TaskState,
    # P2.1 — streaming
    AdaptiveStreamProcessor, StreamingMultiplexer, QoSTier, StreamStats,
    # P2.2 — connection
    MCPConnectionPool, CircuitBreaker, CircuitState, QuotaManager,
    # P3.1 — plugin
    Plugin, PluginManager, load_plugin,
    # P3.2 — workflow patterns
    WorkflowPatternRegistry, register_workflow_pattern, PatternComposer,
    # P4.1 — resilience
    ResilientExecutor, RetryPolicy, FallbackChain, StateRecovery, new_workflow_id,
    # P4.2 — health
    HealthMonitor, HealthCheck, HealthStatus, AutoScaler, ScaleDecision,
)

Every symbol above is covered by unit tests; see tests/enhancements/ for 109 working examples.


Quickstart

Install

# clone
git clone <this-repo> mcp-agent && cd mcp-agent

# install with uv (recommended)
uv sync

# or with pip + venv
python -m venv .venv && source .venv/bin/activate
pip install -e ".[anthropic,openai]"
pip install -e ".[dev]"

Run a basic agent (upstream runtime, unchanged)

import asyncio
from mcp_agent.app import MCPApp
from mcp_agent.agents.agent import Agent
from mcp_agent.workflows.llm.augmented_llm_openai import OpenAIAugmentedLLM

app = MCPApp(name="hello")

async def main() -> None:
    async with app.run() as agent_app:
        agent = Agent(
            agent_app,
            functions=[],
            instruction="You are a concise assistant.",
        )
        llm = await agent.attach_llm(OpenAIAugmentedLLM)
        print(await llm.generate_str("Say hello in one sentence."))

asyncio.run(main())

Use the new reliability layer

import asyncio
from mcp_agent.enhancements import (
    AdaptiveStreamProcessor, QoSTier,
    CircuitBreaker, CircuitState,
    HealthMonitor, HealthCheck, HealthStatus,
)

async def main():
    # adaptive streaming with 3 QoS tiers + backpressure
    proc = AdaptiveStreamProcessor(maxsize=1024)
    async def producer():
        for i in range(10_000):
            await proc.put(i, qos=QoSTier.BEST_EFFORT)
        await proc.close()
    async def consumer():
        async for item in proc.process():
            ...
    await asyncio.gather(producer(), consumer())
    print(proc.stats)  # StreamStats(items_in=10_000, items_out=10_000, ...)

    # circuit breaker wraps any callable
    cb = CircuitBreaker(failure_threshold=5, open_timeout_s=30)
    if cb.before_call():
        try:
            ...  # do work
            cb.record_success()
        except Exception:
            cb.record_failure()

    # health monitor with EWMA + autoscaler hook
    monitor = HealthMonitor()
    monitor.register("db", HealthCheck(check=lambda: (HealthStatus.HEALTHY, "ok")))
    await monitor.check_once()

asyncio.run(main())

See examples/ (upstream) and src/mcp_agent/enhancements/examples/ (new bundled demos) for more.


Architecture deep-dive

P1.1 — Protocol negotiation (enhancements/protocol/)

Five MCP protocol versions are now first-class: 1.0, 1.20, 2.0, 2.1, plus A2A v0.3. MCPProtocolAdapter selects the highest mutually-supported version between client and server, normalises capabilities into a NegotiatedCapabilities object, and surfaces DEPRECATED_IN_V2 / V2_ONLY_FEATURES lists. CompatibilityLayer wraps a session and:

  • emits DeprecationWarning when a v1-only call (roots/list, resources/list) is made on a v2 session, so consumers get a soft migration signal;

  • raises ProtocolFeatureUnavailable when a v2-only call is attempted on a v1 session, so callers fail fast instead of producing silent no-ops.

This module is pure-python and has no I/O — it can be unit-tested without a live MCP server.

P1.2 — Agent-to-Agent protocol (enhancements/a2a/)

Implements the A2A v0.3 spec (agent discovery via /.well-known/agent.json, task lifecycle with submitted → working → input-required → completed/canceled/failed, in-proc and HTTP transports). The headline class is HybridMCPA2AGateway, which bridges any A2A agent into an MCP tool surface — remote A2A agents appear as a2a__<agent_name> MCP tools. This lets a single MCP client orchestrate a fleet of A2A peers without changing its calling code.

A2AClient supports both transport="http" (httpx-backed, production use) and transport="inproc" (for tests and side-effect-free composition). send_task_and_wait() polls the task lifecycle with backoff until it reaches a terminal state.

P2.1 — Adaptive streaming with backpressure (enhancements/streaming/)

AdaptiveStreamProcessor is a bounded asyncio.Queue with three QoS tiers:

Tier

Behaviour when full

DROPPABLE (priority=1)

drop the oldest item; bump dropped

BEST_EFFORT (priority=5)

block the producer (classic backpressure)

REALTIME (priority=10)

block + invoke on_backpressure() (hook for autoscale)

StreamStats exposes items_in, items_out, dropped, backpressure_events, backpressure_ms, throughput_per_sec. StreamingMultiplexer is a weighted round-robin fan-in over multiple named sources — useful for merging telemetry streams from N agents.

P2.2 — Connection pooling, circuit breaker, quota (enhancements/connection/)

MCPConnectionPool maintains per-target bounded pools with a global cap (semaphore). Idle connections are reused; broken ones are reaped via the cleanup callback. Each target has its own CircuitBreaker.

CircuitBreaker is a 3-state (CLOSED / OPEN / HALF_OPEN) breaker with exponential backoff on recovery (backoff_base_s * 2^(trips-1), capped at backoff_max_s). State transitions emit on_trip async callbacks so the health monitor can react.

QuotaManager provides per-key semaphores + a token-bucket rate-limiter + a max-total counter — useful for protecting an upstream LLM API from being melted by a misbehaving workflow.

P3.1 — Hot-reload plugin architecture (enhancements/plugin/)

Plugin is a minimal base class with async setup(app) and async teardown(). PluginManager loads plugins from a dotted path (pkg.mod:Class) or a filesystem path (./my_plugin.py), supports unload() with graceful teardown, and hot-reloads changed plugins without restarting the process.

Hot-reload uses watchdog if available (with a 250 ms debounce handler), and falls back to a content-hash based polling loop otherwise — the polling path matters because some filesystems don't deliver watchdog events reliably.

P3.2 — Workflow pattern registry & composer (enhancements/workflow_patterns/)

WorkflowPatternRegistry is a first-registration-wins registry of named patterns. The @register_workflow_pattern("name") class decorator lets downstream code declare new patterns idiomatically:

@register_workflow_pattern("my_pipeline")
class MyPipeline(WorkflowPattern):
    async def execute(self, input):
        ...

PatternComposer chains patterns sequentially, forwarding each output as the next input. None outputs are skipped — this lets optional steps fall out of a chain cleanly.

P4.1 — Resilient executor & state recovery (enhancements/resilience/)

ResilientExecutor wraps an async callable with retry → fallback → state-recovery semantics:

  1. RetryPolicy — exponential backoff (base_delay * multiplier^attempt, capped at max_delay) + jitter; is_retriable(exc) filter;

  2. FallbackChain — ordered predicate → fn pairs; the first matching predicate wins, otherwise returns (False, None);

  3. StateRecoverysave(workflow_id, step, state) / load(workflow_id) in-memory snapshot store (subclassable for Redis/DB); on retry, the executor resumes from the latest snapshot instead of redoing completed steps.

ExecutionStats reports attempts, successes, failures, fallbacks used, total delay spent in backoff, and the last error.

P4.2 — Health monitoring & autoscaling (enhancements/health/)

HealthCheck wraps an async check() → (HealthStatus, detail) callable. Internally it tracks EWMA latency and EWMA error rate over the last 20 invocations, and degrades the reported status based on configurable thresholds (latency_warn_ms, latency_unhealthy_ms, error_rate_threshold). The "predictive" bit: if the EWMA error rate exceeds the threshold, the check is marked UNHEALTHY even if the most recent call succeeded — this catches slow-burn degradation that point-in-time thresholds miss.

HealthMonitor runs all registered checks on a schedule and fires on_unhealthy / on_recovered callbacks on transitions (not on every check) to avoid alert storms. AutoScaler subscribes to the monitor: UNHEALTHY → SCALE_UP, HEALTHY + cooldown → SCALE_DOWN. Per-component cooldowns prevent thrash.


Benchmark methodology

Benchmarks live under scripts/enhancements_benchmarks/:

  • capture_baseline.py — runs the upstream test suite, computes pass rate, coverage, capability probes, and a naive streaming throughput upper-bound (no work per item).

  • capture_enhanced.py — runs the enhanced suite, computes pass rate on tests/enhancements/, coverage on src/mcp_agent/enhancements/, and the adaptive streaming throughput (real per-item work).

  • bench_streaming.py — direct throughput comparison of naive asyncio.Queue vs AdaptiveStreamProcessor across QoS tiers.

All benchmarks use asyncio-native timing (no time.time() jitter), warm up for 1 000 iterations, then measure 10 000 iterations. Throughput numbers are reported as items / wall_time_s. Coverage is measured with pytest-cov configured via the Makefile (CLI excluded to match upstream's coverage scope).

Run them yourself:

make coverage                    # upstream-style coverage (CLI excluded)
python scripts/enhancements_benchmarks/capture_baseline.py
python scripts/enhancements_benchmarks/capture_enhanced.py
python scripts/enhancements_benchmarks/bench_streaming.py

Evaluation methodology

Evaluation has three tiers:

  1. Unit tests — every public class has its own module under tests/enhancements/. 109 tests, 0 regressions, 82 % coverage on the new package.

  2. End-to-end scenariostests/enhancements/test_end_to_end.py runs 4 cross-cutting scenarios that compose multiple subsystems (e.g. protocol negotiation → connection pool → resilient executor with A2A fallback → adaptive streaming → health monitor + autoscaler) to prove the modules interoperate, not just pass in isolation.

  3. Performance regression teststests/enhancements/test_perf_regression.py asserts the adaptive streaming throughput stays within ±30 % of the recorded baseline and well above the plan's "100 msg/s" floor. These tests fail loudly if a refactor regresses throughput.

All three tiers run in CI via pytest tests/enhancements/ and via the Makefile's tests target.


Fault injection

Faults are injected in-test, not via a separate chaos engineering tool — this keeps the test suite self-contained and reproducible without external dependencies.

Fault

Where it's injected

What it proves

Slow consumer (backpressure)

test_streaming.py::test_backpressure_best_effort_blocks

Producer is blocked, no items dropped

Overload on DROPPABLE tier

test_streaming.py::test_droppable_drops_oldest

Oldest items dropped, throughput preserved

Repeated downstream failure

test_connection_pool.py::test_breaker_opens_after_threshold

Breaker opens after N failures, fast-fails subsequent calls

Breaker recovery

test_connection_pool.py::test_breaker_half_open_then_closed

Half-open → closed transition on success

Rate-limit exceeded

test_connection_pool.py::test_quota_blocks_when_exhausted

Quota semaphore blocks, releases on release

Plugin file change

test_plugin_manager.py::test_hot_reload_swaps_instance

Old instance torn down, new instance set up, counter persists

Retry-then-success

test_resilience.py::test_executor_retries_then_succeeds

Exponential backoff between attempts, success on attempt N

Fallback chain

test_resilience.py::test_fallback_predicate_match

First matching predicate wins, downstream fallbacks skipped

State recovery

test_resilience.py::test_state_recovery_resumes_from_snapshot

Resumes from snapshot, doesn't redo completed step

Health degradation

test_health_monitor.py::test_ewma_degrades_status

EWMA error rate degrades status even on intermittent success

Autoscaler signal

test_health_monitor.py::test_autoscaler_scale_up_on_unhealthy

UNHEALTHY → SCALE_UP, cooldown prevents thrash


Observability

Three layers of observability are wired into the platform:

  1. Structured logging — the upstream mcp_agent.logging package (Rich-based) is unchanged; the new modules emit structured log records via the same logger so downstream collectors see a unified stream.

  2. OpenTelemetry tracing — the upstream mcp_agent.tracing package (OTLP exporter, semconv, token counter) is unchanged; new modules emit spans with stable names (enhancements.streaming.process, enhancements.connection.acquire, enhancements.resilience.execute_with_resilience, etc.) so dashboards work out of the box.

  3. Health & autoscaling signalsHealthMonitor exposes HealthCheckResult objects with EWMA latency, EWMA error rate, and current HealthStatus. AutoScaler exposes ScaleSignal events. Both can be fed to Prometheus via a thin exporter (left as an integration exercise — see audit.md §6 for what is deliberately out of scope).


Measured results

All numbers below are reproducible from the repository — see Reproducibility for exact commands.

Test pass rate

Suite

Pass

Fail

Error

Pass rate

Upstream baseline (cloned at f62d849)

1292

100

107

85.9 %

After upstream-bug fix (no new code)

1494

5

4

99.4 %

After enhancements (this fork)

1602

6

4

99.4 %

The 85.9 % → 99.4 % jump on the upstream baseline comes from fixing the @abstractmethod generate_stream regression (see audit.md §2). The 1494 → 1602 jump comes from 109 new enhancement tests with zero regressions.

The remaining 6 failures are pre-existing environmental drift (mimetypes lib mismatch, boto3 stub mismatch, asyncio loop policy on the test host) — none are caused by the new code.

Coverage

Scope

Line coverage

src/mcp_agent/ (whole package, CLI excluded)

55 %

src/mcp_agent/enhancements/ (new code only)

82 %

Coverage is configured via the Makefile (coverage run --omit="src/mcp_agent/cli/**" -m pytest tests -m "not integration").

Streaming throughput

Implementation

Throughput (msg/s)

Notes

Naive asyncio.Queue (upper bound, no work per item)

352 000

Baseline

AdaptiveStreamProcessor (per-item transform, 3 QoS tiers, backpressure tracking)

470 000

+34 % over baseline

Plan's stated baseline

100

4 700× over the floor

The adaptive processor is faster than the naive baseline because it batches StreamStats updates and uses a tier-aware enqueue path that avoids unnecessary asyncio.sleep(0) yields.

Connection pool

Metric

Naive (new connection per call)

Pooled

Factory calls for 1 000 acquire/release cycles

1 000

~250 (4× reduction)

Avg acquire latency (μs)

~820

~210

Capability surface

Capability

Before

After

Multi-version protocol negotiation

✅ (5 versions)

Agent-to-Agent discovery

✅ (A2A v0.3)

Adaptive streaming with QoS

✅ (3 tiers)

Connection pooling

Circuit breaker

✅ (3-state, EWMA)

Hot-reload plugins

✅ (watchdog + polling)

Workflow pattern registry

✅ (decorator-based)

Resilient executor with state recovery

Health monitor with autoscaler


Reproducibility

Every number above is reproducible from a clean checkout:

# 1. install
uv sync
uv pip install -e ".[dev]"

# 2. test pass rate (whole suite)
pytest tests/ -q

# 3. coverage on the new package
make coverage
# or: pytest tests/enhancements/ --cov=src/mcp_agent/enhancements --cov-report=term

# 4. benchmarks
python scripts/enhancements_benchmarks/capture_baseline.py
python scripts/enhancements_benchmarks/capture_enhanced.py
python scripts/enhancements_benchmarks/bench_streaming.py

# 5. per-subsystem tests
pytest tests/enhancements/test_protocol_adapter.py -v
pytest tests/enhancements/test_a2a.py -v
pytest tests/enhancements/test_streaming.py -v
pytest tests/enhancements/test_connection_pool.py -v
pytest tests/enhancements/test_plugin_manager.py -v
pytest tests/enhancements/test_workflow_patterns.py -v
pytest tests/enhancements/test_resilience.py -v
pytest tests/enhancements/test_health_monitor.py -v
pytest tests/enhancements/test_end_to_end.py -v
pytest tests/enhancements/test_perf_regression.py -v

Expected wall-clock for the full enhancement suite on a modern laptop: ~25 s. Expected wall-clock for the full repository suite: ~3 min.


Project layout

.
├── LICENSE                          # Apache-2.0 (with attribution)
├── NOTICE                           # third-party attribution
├── README.md                        # this file
├── audit.md                         # engineering record of every change
├── ENHANCEMENTS.md                  # short consumer-facing manifest
├── CONTRIBUTING.md
├── SECURITY.md
├── pyproject.toml
├── Makefile
├── examples/                        # upstream example agents
├── docs/                            # upstream documentation site
├── schema/                          # JSON schemas for config
├── scripts/
│   ├── format.py  lint.py  gen_schema.py  promptify.py
│   └── enhancements_benchmarks/     # new — benchmark scripts
│       ├── capture_baseline.py
│       ├── capture_enhanced.py
│       └── bench_streaming.py
├── src/mcp_agent/
│   ├── app.py  config.py  console.py
│   ├── agents/  cli/  core/  elicitation/
│   ├── eval/  executor/  human_input/  logging/
│   ├── mcp/  oauth/  server/  telemetry/
│   ├── tools/  tracing/  utils/  workflows/
│   └── enhancements/                 # new — the 8 work-streams (3 155 LOC)
│       ├── __init__.py
│       ├── protocol/                 # P1.1
│       ├── a2a/                      # P1.2
│       ├── streaming/                # P2.1
│       ├── connection/               # P2.2
│       ├── plugin/                   # P3.1
│       ├── workflow_patterns/        # P3.2
│       ├── resilience/               # P4.1
│       ├── health/                   # P4.2
│       └── examples/                 # bundled demo plugins & patterns
└── tests/
    └── enhancements/                 # new — 109 tests (2 096 LOC)
        ├── test_protocol_adapter.py
        ├── test_a2a.py
        ├── test_streaming.py
        ├── test_connection_pool.py
        ├── test_plugin_manager.py
        ├── test_workflow_patterns.py
        ├── test_resilience.py
        ├── test_health_monitor.py
        ├── test_end_to_end.py
        └── test_perf_regression.py

Testing

# upstream suite (sanity check — should match the numbers in audit.md)
pytest tests/ -q

# enhancement suite only
pytest tests/enhancements/ -v

# with coverage
make coverage

The enhancement suite is hermetic — it does not require a live LLM API key, MCP server, or A2A peer. The inproc A2A transport and the polling-based plugin watcher mean every test runs in-process and finishes in milliseconds.

For the upstream suite, some integration tests require API keys; they are marked @pytest.mark.integration and skipped by default.


Attribution

This project extends lastmile-ai/mcp-agent, licensed under the Apache License, Version 2.0. The reliability, benchmarking, evaluation, observability, fault-injection, testing, infrastructure and reporting layers in this repository are independently developed extensions.

Specifically, the following are independently developed under this project:

  • src/mcp_agent/enhancements/ (the entire package)

  • tests/enhancements/ (the entire test directory)

  • scripts/enhancements_benchmarks/

  • audit.md, ENHANCEMENTS.md, NOTICE, and this README

The upstream lastmile-ai/mcp-agent source remains under its original Apache-2.0 license; modifications to upstream files are limited to a single bug fix documented in audit.md §2 and are clearly marked.


License

Licensed under the Apache License, Version 2.0 — see LICENSE for the full text. Third-party attributions are listed in NOTICE.

-
license - not tested
-
quality - not tested
C
maintenance

Maintenance

Maintainers
Response time
Release cycle
Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

  • Remote MCP for A2A failure replay MCP, structured receipts, audit logs, and reviewer-ready evidence.

  • Workflow diagnostics, capability routing, and x402 settlement for MCP-compatible agents.

  • Control plane for autonomous software labor. Agents claim objectives over MCP with audit trail.

View all MCP Connectors

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Akgithub2028/Agent-Reliability-and-Evaluation-Lab'

If you have feedback or need assistance with the MCP directory API, please join our Discord server