run_standard_setting
Aggregate standard-setting panel ratings into a defensible cut score with standard error, panel agreement, per-panelist flags, and items needing review.
Instructions
Aggregate standard-setting panel judgements into a cut score with its defensibility record.
Methods: "angoff" (ratings are probabilities that a minimally competent candidate answers each item correctly), "yes_no_angoff" (0/1 judgements), "bookmark" (page placements on an Ordered Item Booklet).
Panel ratings may be passed inline as a panelists-by-items array, or read
from a CSV via ratings_file (one row per panelist, one column per item, no
header). Inline is acceptable here because panel ratings are aggregate expert
judgements, not candidate records.
THE CUT SCORE IS NOT THE OUTPUT. The output is the cut plus its standard error, the panel's ICC agreement, per-panelist severity and noise flags, items with no panel consensus, and round-over-round spread. Report the standard error whenever you report the cut: differences smaller than about two standard errors are inside sampling noise, and panels routinely argue over gaps well inside that.
SUPPLY empirical_p_values WHENEVER YOU HAVE THEM. Content experts
systematically overestimate how easy items are, because they know the answer.
The reality check catches a wholesale miscalibration that would otherwise
produce a cut implying an unusable pass rate, and identifies the specific
items to re-discuss.
bookmark requires ordered_item_thetas from an IRT calibration; this
version of the package does not fit IRT models, and classical item difficulty
is not a substitute for them.
REFUSES below 3 panelists (the cut's standard error is not interpretable) and warns below 8. Also refuses ratings outside [0, 1] rather than guessing that they are percentages.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| sem | No | Standard error of measurement of the form, in raw score points, as returned by `compute_classical_stats`. Enables the conventional cut minus/plus 1 SEM adjustments to be reported. Do not confuse it with the panel standard error also in the result: SEM is uncertainty about measuring a candidate, the panel SE is uncertainty about what a different panel would have recommended. Non-negative: it is used as a distance either side of the cut, so a negative value would swap the two adjustments while leaving both labelled correctly. | |
| method | No | 'angoff' -- ratings are probabilities in [0, 1] that a minimally competent candidate answers each item correctly. 'yes_no_angoff' -- ratings are 0/1 judgements of whether such a candidate answers each item correctly. 'bookmark' -- ratings are page placements in an Ordered Item Booklet, and `ordered_item_thetas` is then required. | angoff |
| item_ids | No | Item identifiers in rating order, so that per-item output names the items a panel has to re-discuss instead of giving positions. Supply these whenever the result will go back to a panel. Must be exactly one per item: a short list is refused, not padded, because the per-item review list is built by walking this and a short one drops the surplus items from the output silently. | |
| ratings_file | No | Path to a CSV of panel ratings: one row per panelist, one column per item, no header row. An alternative to `panel_ratings`, for panels large enough that inlining them is unwieldy. | |
| rp_criterion | No | Bookmark method only: the response probability defining mastery of a page, conventionally 0.67. It is an arbitrary convention that moves the cut, so record the value used with the resulting standard. Note this value is RECORDED, not applied -- the RP criterion is already built into `ordered_item_thetas` -- so it must match the RP those thetas were computed at, or the cut carries a standard it was not set against. Strictly between 0 and 1. | |
| panel_ratings | No | Panel judgements as a panelists-by-items array: one inner list per panelist, one entry per item, in a consistent item order. Inline is acceptable here, unlike candidate responses, because these are aggregate expert judgements rather than individual records. Takes precedence over `ratings_file` if both are given. Angoff ratings are probabilities in [0, 1]; values outside that are refused rather than rescaled, because guessing percentages wrong moves the cut by a factor of 100. | |
| bookmark_pages | No | Bookmark method only, and required for it: each panelist's page placement in the Ordered Item Booklet, one number per panelist. Note this is a FLAT list, unlike `panel_ratings` -- a panelist places one bookmark for the whole form, not one judgement per item. Angoff ratings cannot be used here; if that is what you have, the method is 'angoff' or 'yes_no_angoff'. Pages are 1-based and must fall within the booklet length. | |
| options_per_item | No | Number of answer options for each item, in rating order. Used to flag items whose mean rating falls below the chance level (1/options): a panel saying a minimally competent candidate does worse than guessing is a misunderstanding of the rating task, not a low probability. At least 2 per item. For a multiple-response item count the selectable options, not the correct ones. | |
| empirical_p_values | No | Observed proportion correct for each item, in rating order, from a real administration. SUPPLY THESE WHENEVER THEY EXIST: they enable the reality check that catches the standard failure of Angoff panels, which is systematically overestimating how easy items are because the panel knows the answers. Without them a wholesale miscalibration passes through silently. Proportions, not percentages, and one per item. | |
| score_distribution | No | Observed number-correct scores from a real cohort, used to report the pass rate implied by the cut and by each rounding and SEM adjustment. That set of pass rates is shown together on purpose: the adjustments are policy choices, and choosing among them by which pass rate it produces is the thing this output exists to make visible. Scores only -- no identifiers, and not a response matrix. Raw number-correct, so each value is between 0 and the form length; percentages are refused rather than rescaled. | |
| ordered_item_thetas | No | Bookmark method only, and required for it: the theta at which each Ordered Item Booklet page reaches the response-probability criterion, from an IRT calibration. This package does not fit IRT models, and classical item difficulty is not a substitute -- it is sample-dependent and on a different scale. One per booklet page, in booklet order, and non-decreasing if the booklet is really ordered. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |