# Observability Guide
This guide covers foundry-mcp's observability stack, including OpenTelemetry tracing and Prometheus metrics.
## Design Philosophy
foundry-mcp implements **graceful degradation** for all observability features:
- When optional dependencies are not installed, all observability features become no-ops
- Zero performance overhead when observability is disabled
- No import errors or runtime failures from missing dependencies
- Features can be enabled incrementally as needed
This means you can:
1. Run foundry-mcp without any observability dependencies
2. Add Prometheus metrics without OpenTelemetry
3. Add OpenTelemetry tracing without Prometheus
4. Enable full observability with both
## Quick Start
### Installing Dependencies
```bash
# Prometheus metrics only
pip install foundry-mcp[metrics]
# OpenTelemetry tracing only
pip install foundry-mcp[tracing]
# Full observability stack
pip install foundry-mcp[observability]
```
### Enabling Prometheus Metrics
Add to your `foundry-mcp.toml`:
```toml
[observability]
enabled = true
prometheus_enabled = true
prometheus_port = 9090 # Expose /metrics on this port
```
Verify metrics are available:
```bash
curl http://localhost:9090/metrics
```
### Enabling OpenTelemetry Tracing
Add to your `foundry-mcp.toml`:
```toml
[observability]
enabled = true
otel_enabled = true
otel_endpoint = "localhost:4317" # OTLP gRPC endpoint
otel_service_name = "foundry-mcp"
otel_sample_rate = 1.0 # Sample all traces (reduce in production)
```
### Full Stack Configuration
```toml
[observability]
enabled = true
# OpenTelemetry
otel_enabled = true
otel_endpoint = "otel-collector:4317"
otel_service_name = "foundry-mcp"
otel_sample_rate = 0.1 # Sample 10% in production
# Prometheus
prometheus_enabled = true
prometheus_port = 9090
prometheus_host = "0.0.0.0"
prometheus_namespace = "foundry_mcp"
```
## Configuration Reference
### TOML Configuration
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `observability.enabled` | bool | `false` | Master switch for all observability |
| `observability.otel_enabled` | bool | `false` | Enable OpenTelemetry tracing |
| `observability.otel_endpoint` | string | `localhost:4317` | OTLP gRPC endpoint |
| `observability.otel_service_name` | string | `foundry-mcp` | Service name in traces |
| `observability.otel_sample_rate` | float | `1.0` | Trace sampling rate (0.0-1.0) |
| `observability.prometheus_enabled` | bool | `false` | Enable Prometheus metrics |
| `observability.prometheus_port` | int | `0` | HTTP port for /metrics (0=disabled) |
| `observability.prometheus_host` | string | `0.0.0.0` | HTTP server bind address |
| `observability.prometheus_namespace` | string | `foundry_mcp` | Metric name prefix |
### Environment Variables
All TOML settings can be overridden via environment variables:
| Environment Variable | TOML Equivalent |
|---------------------|-----------------|
| `FOUNDRY_MCP_OBSERVABILITY_ENABLED` | `observability.enabled` |
| `FOUNDRY_MCP_OTEL_ENABLED` | `observability.otel_enabled` |
| `FOUNDRY_MCP_OTEL_ENDPOINT` | `observability.otel_endpoint` |
| `FOUNDRY_MCP_OTEL_SERVICE_NAME` | `observability.otel_service_name` |
| `FOUNDRY_MCP_OTEL_SAMPLE_RATE` | `observability.otel_sample_rate` |
| `FOUNDRY_MCP_PROMETHEUS_ENABLED` | `observability.prometheus_enabled` |
| `FOUNDRY_MCP_PROMETHEUS_PORT` | `observability.prometheus_port` |
| `FOUNDRY_MCP_PROMETHEUS_HOST` | `observability.prometheus_host` |
| `FOUNDRY_MCP_PROMETHEUS_NAMESPACE` | `observability.prometheus_namespace` |
## Metrics Catalog
foundry-mcp emits the following Prometheus metrics:
### Tool Metrics
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `foundry_mcp_tool_invocations_total` | Counter | tool, status | Total tool invocations |
| `foundry_mcp_tool_duration_seconds` | Histogram | tool | Tool execution duration |
| `foundry_mcp_tool_errors_total` | Counter | tool, error_type | Total tool errors |
| `foundry_mcp_active_operations` | Gauge | operation_type | Currently active operations |
### Resource Metrics
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `foundry_mcp_resource_access_total` | Counter | resource_type, action | Total resource accesses |
For the full catalog with examples, see `src/foundry_mcp/core/metrics_registry.py`.
## Deep Research Audit Artifacts
Deep research workflows emit a per-run JSONL audit artifact when enabled.
Each line is a structured event covering phase starts/completions, provider
results, LLM prompts/responses, and summary stats.
**Location (default)**:
- `specs/.research/deep_research/<research_id>.audit.jsonl`
**Config toggle**:
- `research.deep_research_audit_artifacts = true`
**Example event**:
```json
{"timestamp":"2026-01-01T12:00:00Z","event_id":"...","event_type":"analysis_result","level":"info","research_id":"deepres-abc123","phase":"analysis","iteration":1,"data":{"provider_id":"gemini","tokens_used":1234,"parse_success":true,"finding_count":4}}
```
## Troubleshooting
### Observability Not Working
1. **Check dependencies are installed:**
```bash
pip show prometheus-client opentelemetry-sdk
```
2. **Verify configuration:**
```python
from foundry_mcp.core.observability import get_observability_status
print(get_observability_status())
```
3. **Check logs for initialization errors:**
Look for "Failed to initialize OpenTelemetry" or "Failed to initialize Prometheus" messages.
### Metrics Not Appearing
1. **Verify Prometheus port is exposed:**
```bash
curl http://localhost:9090/metrics
```
2. **Check `prometheus_port` is non-zero** in configuration.
3. **Verify `observability.enabled = true`** - this is the master switch.
### Traces Not Appearing in Jaeger/Zipkin
1. **Verify OTLP endpoint is reachable:**
```bash
nc -zv localhost 4317
```
2. **Check `otel_sample_rate`** - set to `1.0` for debugging.
3. **Ensure traces are flushed** - the server calls `shutdown()` on exit to flush pending spans.
### High Memory Usage
If observability is causing memory issues:
1. Reduce `otel_sample_rate` to sample fewer traces
2. Ensure the OTLP endpoint is healthy (spans queue when endpoint is down)
3. Set `prometheus_port = 0` if not using Prometheus HTTP server
## Programmatic Access
For advanced use cases, access the observability manager directly:
```python
from foundry_mcp.core.observability import get_observability_manager
manager = get_observability_manager()
# Check what's enabled
print(f"Tracing: {manager.is_tracing_enabled()}")
print(f"Metrics: {manager.is_metrics_enabled()}")
# Get a tracer for custom spans
tracer = manager.get_tracer("my-module")
with tracer.start_as_current_span("my-operation"):
# ... do work
pass
# Get Prometheus exporter for custom metrics
exporter = manager.get_prometheus_exporter()
exporter.record_tool_invocation("custom-tool", success=True, duration_ms=42.5)
```
