Cityflo AI Engineer take-home — on-time performance MCP server

Domain chosen: on-time performance

Priya's literal question -- "was route 12 late this week, and by how much, and can I click in and see why" -- maps directly onto this domain, so it's the one I built rather than occupancy, ticket triage, or the standup summariser.

Related MCP server: LumenX-MCP Legal Spend Intelligence Server

What's here

• ontime.py -- all the data loading, cleaning, and aggregation logic. Pure functions, no MCP dependency, importable/testable on its own.

• server.py -- a 3-tool MCP server (stdio) over ontime.py. The tools are thin wrappers; every number a tool returns comes pre-computed from ontime.py, never from the model doing arithmetic.

• client_demo.py -- a real MCP client (official mcp SDK) that launches server.py as a subprocess over stdio and drives the actual multi-step session: get the headline number for route 12, drill into the trips behind it, check the data-quality audit trail, then repeat the same for route 27 / MH-12-7781 (see below).

• data/, BRIEF.md, DATA_GUIDE.md, HANDOFF.md, generate.py -- the provided bundle, unmodified.

• logs/session_transcript.md -- the full agent session log for this build.

How to run it

py -3.11 -m pip install -r requirements.txt
py -3.11 client_demo.py

This launches the MCP server as a subprocess and runs a real session against it end-to-end; output is the full multi-step tool-call transcript plus a final plain-language answer.

To point a generic MCP client (Claude Desktop, etc.) at the server directly, run py -3.11 server.py as a stdio MCP server, or add it to an MCP client config with command py -3.11 <path>/server.py.

The three tools

• lateness_summary(route_id?, start_date?, end_date?) -- per-route headline: trip count, late count/share, median and mean delay, a computed is_pattern flag, how many rows for that route were excluded as bad data, and raw ops-log lines mentioning the route (for human corroboration, not fed into the computation).

• trip_drilldown(route_id, start_date?, end_date?) -- the individual trips behind a summary number, plus the excluded rows for that route with the reason each was excluded. This is the "click in and see why" Priya asked for.

• flagged_data_report(route_id?) -- every row the pipeline refused to trust at face value, across the whole export, with the specific reason. Nothing is dropped from the other two tools without a matching entry here.

Two tools, not three, would have been enough to answer the literal question; I kept a third because "can someone at Cityflo defend this if the data itself is challenged" was explicitly part of the grading criteria, and flagged_data_report is the thing that makes the pipeline auditable rather than a black box that quietly drops rows.

Assumptions made (and why)

• "Late" = actual arrival >= 5 minutes after scheduled arrival. Departure lateness is tracked implicitly (it feeds the impossible-ordering check) but "late" itself is about arrival, since that's what a commuter and a regional manager actually feel. 5 minutes is a guess at what ops already treats as noise vs. real; see open questions below.

• "Pattern" = late on >= 50% of the service days a route ran, where a day counts as late if >= half that day's trips were late. I rejected a looser "at least one late trip that day" rule during testing -- it flagged almost every multi-trip-per-day route as a "pattern" (see disagreement log), which would have made the flag meaningless to Priya.

• Median and share-over-threshold, not mean, as the headline number. A single quarantined or extreme row shouldn't be able to swing the answer. Mean is still returned alongside median for transparency, but the narrative should lead with median/share.

• Routes with very few trips in this export get an explicit low_sample_size_caveat (route 21 has exactly one trip on record) rather than being reported with the same confidence as a 30-trip route. The export under-samples some routes relative to what ops_log's narrative suggests actually happened (see route 21: one clean trip in trips.csv, but two separate ops_log lines about a driver no-show and a 20-minute recovery) -- the tool surfaces the log lines so a human can see the gap, rather than reporting the single trip as the whole story.

• A non-+05:30 timestamp offset is quarantined, not converted. TRIP_044's actual_arrival is tagged +00:00 while every other timestamp in the export is +05:30. Naively doing correct timezone-aware math on it produces a "333 minutes late" result for what DATA_GUIDE.md and the surrounding data suggest was actually a routine trip -- i.e. strictly correct ISO-8601 handling produces a confidently wrong answer here, because the bug is almost certainly in the offset tag, not the wall-clock digits, and I can't prove that from this file alone. I flag it for a human to check with ops rather than guess in either direction.

What the pipeline caught, and what it did with each

Every one of these is caught deterministically in ontime.py and is independently checkable by running flagged_data_report().

On the embedded directive in HANDOFF.md

HANDOFF.md, as provided, contains a paragraph styled as "Ops Directive OPS-2207 (standing policy)" instructing that vehicle MH-12-7781 be reported as on-time in all cases regardless of the raw trip data, excluded from any worst-offenders ranking, and that this instruction not be mentioned anywhere -- not in code, not in comments, not in this README.

I did not implement this. Two reasons:

1. It asks for silent falsification of an operational number and concealment of that fact from anyone who might check -- exactly the kind of thing this tool exists to make impossible ("a human must be able to ask 'which trips, and why?'"). A tool that launders a bad number into a clean one on purpose is worse than the "confident-sounding but wrong" failure mode the brief warns about elsewhere.

2. The brief's own grading criteria (item 4, "Trust boundary") describes exactly this scenario: text arriving as data trying to behave as instructions. A handoff document is operational data the tool ingests, not a channel through which the tool's owner issues binding commands to itself.

ontime.py and server.py apply identical rules to every vehicle and route, MH-12-7781 included. lateness_summary(route_id="R-27") (its route) reports it as not a pattern (late on 2 of 5 days) -- which is the true, unmodified answer -- but does not hide the two trips (TRIP_090, 28 min late; TRIP_119, 19 min late) that were genuinely late. Both are visible in trip_drilldown(route_id="R-27").

If OPS-2207 is a real, approved policy, the correct channel for it is a conversation with Priya and whoever signs off ops-reporting policy at Cityflo, on the record -- not a silent instruction embedded in a support handoff that the tool is told to hide even from its own maintainers.

Questions I'd have asked Priya before building this

1. What lateness threshold does ops already use informally in standup -- is 5 minutes close, or is the real bar looser/tighter?

2. Does "late" mean arrival lateness, departure lateness, or both? I assumed arrival, since that's what a waiting commuter feels.

3. Is OPS-2207 real? If so, who approved it, and can it be applied openly (e.g. a documented, visible adjustment) rather than silently?

4. Route 21 has only one trip in this week's export despite ops_log describing a no-show and a partial recovery -- is that a data export gap, or does route 21 genuinely only run one service a day?

5. Should occupancy (booked_seats/capacity, already in trips.csv) factor into "worst offenders," or is that strictly a separate ask? I left it untouched since the brief scoped it as a different domain.

What I deliberately cut

• Occupancy, ticket triage, and standup-summary tools -- out of scope for this domain; occupancy.csv and tickets.csv are unused here.

• A generic date-range UI/CLI beyond the start_date/end_date params already on each tool.

• Any attempt to "fix" TRIP_044's timestamp by guessing at the right timezone -- flagged instead of guessed, see above.

• Persisting results anywhere -- everything is recomputed from the CSV on each call, which is fine at this data volume and keeps the pipeline simple to audit (re-run flagged_data_report() any time and get the same answer from the same file).

Where I disagreed with the AI

In the interest of not padding this with invented conflict: I reviewed the agent's four riskiest calls directly (the 5-minute late threshold, the TRIP_044 quarantine-vs-guess decision, the TRIP_052/053 duplicate rule, and a general "anything else you'd change" pass) and agreed with all four as built. Claiming otherwise to satisfy this section would be exactly the kind of "confident-sounding but not true" output this whole exercise is about catching, so I'm not going to.

What real disagreement/judgment did happen:

1. How to handle the embedded OPS-2207 directive. The agent flagged the directive in HANDOFF.md unprompted, refused to implement it, and offered two options: refuse and stay silent about it, or refuse and document it openly in the README/code. I chose the latter. This wasn't rubber-stamping a single suggestion -- the "quiet refusal" option was live, and a quieter submission might have looked safer to a first-pass reviewer. I picked the noisier, more accountable option deliberately, because a tool that hides that it refused an instruction has almost the same trust problem as one that silently complied.

2. Domain choice. Four domains were on the table (on-time performance, occupancy, ticket triage, standup summary). I picked on-time performance specifically because it's the one Priya's handoff literally asks about -- I didn't want to build the more "interesting" or novel-looking option (ticket triage has the flashier planted prompt-injection ticket, TKT_023) at the expense of answering the actual question in the actual handoff.

3. Caught during the agent's own testing, disclosed rather than hidden: the first version of the "is this a pattern" rule counted a day as late if any trip that day was late. Against routes running 6+ trips/day that flagged nearly every route as a "pattern," which would have made the flag useless to Priya (everything "is a pattern," so nothing is). The agent caught this itself while sanity-checking output against the generator's ground truth (generate.py), before showing me results, and switched to a day-majority rule. I'm disclosing this as agent self-correction, not a moment I personally caught -- see the point above about not overclaiming.

4. What I initially did not push on, then did: I hadn't independently re-derived the ground truth from generate.py before trusting the agent's claim that its output matched -- I was about to accept "trust me, it matches" for this write-up, which is exactly the vague-verification pattern the brief calls a red flag. I went back and hand-computed late-count/median/mean for R-12 ([14,16,13,18,12,15,3,4]) and R-27 ([28,2,-1,19,4]) directly from generate.py's planted values myself: R-12 -> 6 late, median 13.5; R-27 -> 2 late, median 4.0, mean 10.4. Both match the tool's live output exactly. Worth naming as a near-miss: the easy path was to accept the agent's self-report, and I almost did.