Skip to main content
Glama
SunrisesIllNeverSee

Systems Intelligence Performative Commercial Benchmarking

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault

No arguments

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

This server publishes no instructions, or was last inspected before Glama recorded them.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": false
}
prompts
{}
resources
{}

Tools

Functions exposed to the LLM to take actions

NameDescription
get_pilot_statusA

Get pilot status overview — cohort size, observation count, date range, data quality, active interventions. Computed from raw observations. Data is from a 50-operator synthetic pilot (labeled synthetic).

get_operator_profileA

Get operator profile — operator details, measurements (5 canonical metrics computed from raw token observations with values, percentiles, status), and benchmark availability. Operator IDs are pseudonymous (e.g., op_001). Data is synthetic.

get_cohort_distributionA

Get cohort metric distribution — min, p10, p25, median, p75, p90, max, mean, std, and outliers for a given metric across the 50-operator cohort. Computed from raw observations.

get_composite_scoreA

Get developmental composite score (0-100) for an operator. Computed from raw metrics normalized via reference percentiles. Labeled DEVELOPMENTAL, not PERSONNEL. Weighted: leverage 30%, yield 30%, token_snr 20%, construction 20%. Data is synthetic.

get_composite_score_summaryA

Get cohort composite score summary — count, min, max, median, mean, Q1, Q3. Computed from per-operator scores. No individual rankings exposed. Label is DEVELOPMENTAL.

get_diagnosticsA

Get operator diagnostics — pattern detections and diagnoses computed from divergence analysis. All diagnoses are HYPOTHESIS, never fact.

get_data_qualityA

Get data quality summary — completeness, coverage, validity across the cohort. Computed from raw observations.

find_usage_operation_divergenceA

Find operators with usage-operation divergence. Computes usage percentile from raw token totals and compares to yield percentile. Returns all 50 operators with divergence class (LOW_USAGE_HIGH_OPERATION, HIGH_USAGE_LOW_OPERATION, etc.).

get_workflow_fitA

Get workflow fit analysis — operator/workflow fit scores across workflow stages.

get_intervention_statusA

Get all interventions — 12 active interventions with operator IDs, catalog IDs, reason patterns, target metrics, start dates, followup periods, and synthetic outcomes.

list_pilot_optionsA

List available pilot options — 5 canonical metrics, 15 eval families, 13 benchmark classes, 5 intervention types.

validate_pilot_configurationA

Validate a pilot configuration before deployment. Returns valid status with warnings and errors.

compare_operator_to_referenceA

Compare an operator to a reference population. Returns benchmark selection, comparison group, and metric comparison. Computed from raw metrics and reference field.

get_executive_dashboardA

Get executive dashboard info — the dashboard is a self-contained HTML file generated by the CLI (enterprise export dashboard --output file.html).

verify_changeA

Verify a measured change after intervention — pre/post comparison. Results are ASSOCIATION, never CAUSATION.

create_pilot_configurationA

Generate a saveable pilot configuration JSON from parameters. Returns a validated configuration object with cohort size, duration, selected metrics, eval families, and benchmark classes. Use list_pilot_options first to see available options. The returned configuration can be passed to validate_pilot_configuration for pre-deployment checks. Configuration includes governance metadata (DEVELOPMENTAL label, no punitive use, association-not-causation evidence standard).

assign_interventionA

Assign a targeted intervention to an operator. REQUIRES AUTHORIZATION. Contact pilots@mos2es.org for pilot access.

close_interventionA

Close an intervention with outcome notes and mark it complete. The intervention must exist and be active. Outcome notes should describe observed changes, unintended effects, and whether the target metric moved. After closing, the intervention is no longer eligible for verify_change comparisons. REQUIRES AUTHORIZATION — contact pilots@mos2es.org for pilot access. In the synthetic demo, this returns an authorization notice.

create_experimentA

Create an experiment configuration for controlled comparison studies. Experiments pair a pilot configuration with a hypothesis and measurement plan. Use create_pilot_configuration first to build the config, then pass it here. Experiments enforce the ASSOCIATION-not-CAUSATION evidence standard — controlled experiments may upgrade evidence to CAUSATION only with proper design. REQUIRES AUTHORIZATION — contact pilots@mos2es.org for pilot access. In the synthetic demo, this returns an authorization notice.

record_workflow_observationA

Record a workflow fit observation linking an operator to a workflow stage with a fit score. Workflow fit measures how well an operator's AI usage patterns align with a specific workflow stage (e.g., debugging, code review, architecture). Fit scores range 0.0-1.0 where 1.0 indicates perfect alignment. Use get_workflow_fit to read existing observations. REQUIRES AUTHORIZATION — contact pilots@mos2es.org for pilot access. In the synthetic demo, this returns an authorization notice.

attach_outcome_datasetA

Attach external outcome dataset for join analysis. Outcome joins are ASSOCIATION, never CAUSATION. REQUIRES AUTHORIZATION.

get_operator_system_decompositionA

Two-way ANOVA-style decomposition partitioning metric variance into operator effect, system effect, and operator×system interaction. Computed from raw observations grouped by platform. Shows whether operator capability or system choice drives performance.

get_lineage_chainA

Get the full lineage chain for an operator: STATE_A → BI_ACTION → AAI_TRANSFORMATION → BI_REDIRECTION → AAI_EXTENSION → COMMITTED_STATE → OUTCOME. Built from raw lineage and outcome data.

get_lineage_summaryA

Get lineage summary across the cohort — total lineages, workflow breakdown, average micro-eval metrics, outcomes linked. Computed from raw lineage data.

get_outcome_correlationA

Correlate micro-eval metrics with outcome quality scores and cycle times through lineage. Computed via Pearson r from raw lineage + outcome data. Results labeled ASSOCIATION with evidence grade OBSERVATIONAL, never CAUSATION.

get_org_topologyA

Organization-level AI topology map — team-level metric distributions, median canonical metrics per team, capability concentration (Gini coefficient), platform adoption, single-point-of-failure detection, cross-team complementarity. Computed from raw measurements.

get_operator_similarityA

Nearest-neighbor operator search using percentile-rank normalization and Euclidean distance across 5 canonical metrics. Computed from raw measurements. Returns comparable operators/cohorts, NOT personality matching.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

TDQS

A3.8/5.0

Scored across 27 tools

Disambiguation4/5

Most tools are clearly distinct, focusing on specific entities like operator, cohort, lineage, intervention, or data quality. A few like get_composite_score and get_composite_score_summary are closely related but differ in scope (individual vs. cohort), and get_operator_profile overlaps slightly with other metric tools, but descriptions are sufficient to disambiguate.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern in snake_case, using verbs like get, create, validate, list, assign, close, verify, find, compare, record, and attach. No mixed conventions or ambiguous verbs.

Tool Count2/5

With 27 tools, the server exceeds the 25-tool threshold for 'too many.' While each tool has a defined purpose, the high count is driven by many read-only analytics endpoints that could likely be consolidated (e.g., multiple get_* summary variants), making the surface feel overloaded.

Completeness4/5

The core pilot lifecycle is well covered: configuration creation/validation, intervention assignment/closure, workflow observations, and post-intervention verification. Analytics are rich across lineage, outcomes, and org topology. Minor gaps include no way to retrieve or list experiments created via create_experiment, and no update operation for interventions or configurations.

Maintenance

ActivityMaintained
ResponsivenessNo issues