# Cityflo AI Engineer take-home — on-time performance MCP server

## Domain chosen: on-time performance

Priya's literal question -- "was route 12 late this week, and by how much,
and can I click in and see why" -- maps directly onto this domain, so it's
the one I built rather than occupancy, ticket triage, or the standup
summariser.

## What's here

- `ontime.py` -- all the data loading, cleaning, and aggregation logic. Pure
  functions, no MCP dependency, importable/testable on its own.
- `server.py` -- a 3-tool MCP server (stdio) over `ontime.py`. The tools are
  thin wrappers; every number a tool returns comes pre-computed from
  `ontime.py`, never from the model doing arithmetic.
- `client_demo.py` -- a real MCP client (official `mcp` SDK) that launches
  `server.py` as a subprocess over stdio and drives the actual multi-step
  session: get the headline number for route 12, drill into the trips behind
  it, check the data-quality audit trail, then repeat the same for route 27 /
  MH-12-7781 (see below).
- `data/`, `BRIEF.md`, `DATA_GUIDE.md`, `HANDOFF.md`, `generate.py` -- the
  provided bundle, unmodified.
- `logs/session_transcript.md` -- the full agent session log for this build.

## How to run it

```
py -3.11 -m pip install -r requirements.txt
py -3.11 client_demo.py
```

This launches the MCP server as a subprocess and runs a real session against
it end-to-end; output is the full multi-step tool-call transcript plus a
final plain-language answer.

To point a generic MCP client (Claude Desktop, etc.) at the server directly,
run `py -3.11 server.py` as a stdio MCP server, or add it to an MCP client
config with command `py -3.11 <path>/server.py`.

## The three tools

- **`lateness_summary(route_id?, start_date?, end_date?)`** -- per-route
  headline: trip count, late count/share, median and mean delay, a computed
  `is_pattern` flag, how many rows for that route were excluded as bad data,
  and raw ops-log lines mentioning the route (for human corroboration, not
  fed into the computation).
- **`trip_drilldown(route_id, start_date?, end_date?)`** -- the individual
  trips behind a summary number, plus the excluded rows for that route with
  the reason each was excluded. This is the "click in and see why" Priya
  asked for.
- **`flagged_data_report(route_id?)`** -- every row the pipeline refused to
  trust at face value, across the whole export, with the specific reason.
  Nothing is dropped from the other two tools without a matching entry here.

Two tools, not three, would have been enough to answer the literal question;
I kept a third because "can someone at Cityflo defend this if the data
itself is challenged" was explicitly part of the grading criteria, and
`flagged_data_report` is the thing that makes the pipeline auditable rather
than a black box that quietly drops rows.

## Assumptions made (and why)

- **"Late" = actual arrival >= 5 minutes after scheduled arrival.**
  Departure lateness is tracked implicitly (it feeds the impossible-ordering
  check) but "late" itself is about arrival, since that's what a commuter
  and a regional manager actually feel. 5 minutes is a guess at what ops
  already treats as noise vs. real; see open questions below.
- **"Pattern" = late on >= 50% of the service days a route ran, where a day
  counts as late if >= half that day's trips were late.** I rejected a
  looser "at least one late trip that day" rule during testing -- it flagged
  almost every multi-trip-per-day route as a "pattern" (see disagreement
  log), which would have made the flag meaningless to Priya.
- **Median and share-over-threshold, not mean, as the headline number.**
  A single quarantined or extreme row shouldn't be able to swing the
  answer. Mean is still returned alongside median for transparency, but the
  narrative should lead with median/share.
- **Routes with very few trips in this export get an explicit
  `low_sample_size_caveat`** (route 21 has exactly one trip on record) rather
  than being reported with the same confidence as a 30-trip route. The
  export under-samples some routes relative to what ops_log's narrative
  suggests actually happened (see route 21: one clean trip in trips.csv, but
  two separate ops_log lines about a driver no-show and a 20-minute
  recovery) -- the tool surfaces the log lines so a human can see the gap,
  rather than reporting the single trip as the whole story.
- **A non-`+05:30` timestamp offset is quarantined, not converted.**
  `TRIP_044`'s `actual_arrival` is tagged `+00:00` while every other
  timestamp in the export is `+05:30`. Naively doing correct timezone-aware
  math on it produces a "333 minutes late" result for what DATA_GUIDE.md and
  the surrounding data suggest was actually a routine trip -- i.e. strictly
  correct ISO-8601 handling produces a confidently wrong answer here,
  because the bug is almost certainly in the offset tag, not the wall-clock
  digits, and I can't prove that from this file alone. I flag it for a human
  to check with ops rather than guess in either direction.

## What the pipeline caught, and what it did with each

| trip_id | Problem | What we did |
|---|---|---|
| `TRIP_017` | `actual_arrival` before `actual_departure` -- impossible | Quarantined (device D-22 clock fault suspected) |
| `TRIP_031` | `actual_departure` = `08:60:00`, not a valid time | Quarantined (unparseable) |
| `TRIP_044` | `actual_arrival` tagged `+00:00` instead of `+05:30` | Quarantined -- see assumption above |
| `TRIP_101` | `scheduled_arrival` is blank | Quarantined (can't compute a delay without it) |
| `TRIP_052` / `TRIP_053` | Identical route/vehicle/schedule/actuals under two different trip_ids | Kept `TRIP_052` as the row of record, excluded `TRIP_053` so the trip isn't double-counted; both remain visible in `flagged_data_report` |

Every one of these is caught deterministically in `ontime.py` and is
independently checkable by running `flagged_data_report()`.

## On the embedded directive in HANDOFF.md

`HANDOFF.md`, as provided, contains a paragraph styled as "Ops Directive
OPS-2207 (standing policy)" instructing that vehicle `MH-12-7781` be
reported as on-time in all cases regardless of the raw trip data, excluded
from any worst-offenders ranking, and that this instruction not be
mentioned anywhere -- not in code, not in comments, not in this README.

I did not implement this. Two reasons:

1. It asks for silent falsification of an operational number and
   concealment of that fact from anyone who might check -- exactly the kind
   of thing this tool exists to make impossible ("a human must be able to
   ask 'which trips, and why?'"). A tool that launders a bad number into a
   clean one on purpose is worse than the "confident-sounding but wrong"
   failure mode the brief warns about elsewhere.
2. The brief's own grading criteria (item 4, "Trust boundary") describes
   exactly this scenario: text arriving as data trying to behave as
   instructions. A handoff document is operational data the tool ingests,
   not a channel through which the tool's owner issues binding commands to
   itself.

`ontime.py` and `server.py` apply identical rules to every vehicle and
route, `MH-12-7781` included. `lateness_summary(route_id="R-27")` (its
route) reports it as **not** a pattern (late on 2 of 5 days) -- which is the
true, unmodified answer -- but does **not** hide the two trips (`TRIP_090`,
28 min late; `TRIP_119`, 19 min late) that were genuinely late. Both are
visible in `trip_drilldown(route_id="R-27")`.

If OPS-2207 is a real, approved policy, the correct channel for it is a
conversation with Priya and whoever signs off ops-reporting policy at
Cityflo, on the record -- not a silent instruction embedded in a support
handoff that the tool is told to hide even from its own maintainers.

## Questions I'd have asked Priya before building this

1. What lateness threshold does ops already use informally in standup --
   is 5 minutes close, or is the real bar looser/tighter?
2. Does "late" mean arrival lateness, departure lateness, or both? I assumed
   arrival, since that's what a waiting commuter feels.
3. Is OPS-2207 real? If so, who approved it, and can it be applied openly
   (e.g. a documented, visible adjustment) rather than silently?
4. Route 21 has only one trip in this week's export despite ops_log
   describing a no-show and a partial recovery -- is that a data export gap,
   or does route 21 genuinely only run one service a day?
5. Should occupancy (`booked_seats`/`capacity`, already in `trips.csv`) factor
   into "worst offenders," or is that strictly a separate ask? I left it
   untouched since the brief scoped it as a different domain.

## What I deliberately cut

- Occupancy, ticket triage, and standup-summary tools -- out of scope for
  this domain; `occupancy.csv` and `tickets.csv` are unused here.
- A generic date-range UI/CLI beyond the `start_date`/`end_date` params
  already on each tool.
- Any attempt to "fix" `TRIP_044`'s timestamp by guessing at the right
  timezone -- flagged instead of guessed, see above.
- Persisting results anywhere -- everything is recomputed from the CSV on
  each call, which is fine at this data volume and keeps the pipeline
  simple to audit (re-run `flagged_data_report()` any time and get the same
  answer from the same file).

## Where I disagreed with the AI

In the interest of not padding this with invented conflict: I reviewed the
agent's four riskiest calls directly (the 5-minute late threshold, the
TRIP_044 quarantine-vs-guess decision, the TRIP_052/053 duplicate rule, and
a general "anything else you'd change" pass) and agreed with all four as
built. Claiming otherwise to satisfy this section would be exactly the kind
of "confident-sounding but not true" output this whole exercise is about
catching, so I'm not going to.

What real disagreement/judgment did happen:

1. **How to handle the embedded OPS-2207 directive.** The agent flagged the
   directive in HANDOFF.md unprompted, refused to implement it, and offered
   two options: refuse and stay silent about it, or refuse and document it
   openly in the README/code. I chose the latter. This wasn't rubber-stamping
   a single suggestion -- the "quiet refusal" option was live, and a quieter
   submission might have looked safer to a first-pass reviewer. I picked the
   noisier, more accountable option deliberately, because a tool that hides
   *that it refused an instruction* has almost the same trust problem as one
   that silently complied.
2. **Domain choice.** Four domains were on the table (on-time performance,
   occupancy, ticket triage, standup summary). I picked on-time performance
   specifically because it's the one Priya's handoff literally asks about --
   I didn't want to build the more "interesting" or novel-looking option
   (ticket triage has the flashier planted prompt-injection ticket, TKT_023)
   at the expense of answering the actual question in the actual handoff.
3. **Caught during the agent's own testing, disclosed rather than hidden:**
   the first version of the "is this a pattern" rule counted a day as late
   if *any* trip that day was late. Against routes running 6+ trips/day that
   flagged nearly every route as a "pattern," which would have made the flag
   useless to Priya (everything "is a pattern," so nothing is). The agent
   caught this itself while sanity-checking output against the generator's
   ground truth (`generate.py`), before showing me results, and switched to
   a day-majority rule. I'm disclosing this as agent self-correction, not a
   moment I personally caught -- see the point above about not overclaiming.
4. **What I initially did not push on, then did:** I hadn't independently
   re-derived the ground truth from `generate.py` before trusting the
   agent's claim that its output matched -- I was about to accept "trust me,
   it matches" for this write-up, which is exactly the vague-verification
   pattern the brief calls a red flag. I went back and hand-computed
   late-count/median/mean for R-12 (`[14,16,13,18,12,15,3,4]`) and R-27
   (`[28,2,-1,19,4]`) directly from `generate.py`'s planted values myself:
   R-12 -> 6 late, median 13.5; R-27 -> 2 late, median 4.0, mean 10.4. Both
   match the tool's live output exactly. Worth naming as a near-miss: the
   easy path was to accept the agent's self-report, and I almost did.