flag_dif
Screen every item for differential item functioning (DIF) between subgroups using three methods, flagging uniform and non-uniform DIF for content review.
Instructions
Screen every item for differential item functioning between subgroups.
Runs three methods per item, deliberately not redundant:
Mantel-Haenszel with ETS A/B/C classification -- the operational standard, but it detects UNIFORM DIF ONLY.
Standardised p-difference (Dorans & Kulick) -- same finding on the proportion-correct metric, which panels read without translation.
Logistic regression -- adds the group-by-ability interaction, so it catches NON-UNIFORM DIF that Mantel-Haenszel is structurally blind to. Items flagged by logistic but category A under MH are called out separately; an MH-only screen would have missed them.
Matching is on observed number-correct score. purify=True (default) removes
flagged items from the matching criterion and retests; note that difR in R
defaults to no purification, so numbers differ from a naive difR comparison.
REFUSES when the smaller group is under 50, and warns under 200 because the ETS A/B/C bands were calibrated above that.
CRITICAL FOR REPORTING: a DIF flag is not a finding of bias. It means the item behaves differently for two groups of equal overall proficiency, which routes the item to human content review to decide whether there is a construct-irrelevant reason. Never describe flagged items as biased items, and never report the flag count as a count of biased items. Use the Benjamini-Hochberg adjusted p-values, not the raw ones -- screening a 65-item form is 65 simultaneous tests.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| layout | No | 'wide' -- one row per examinee, one column per item. 'long' -- one row per examinee-item response, which requires `id_column` and expects `item_id` and `response` columns. | wide |
| purify | No | Re-run the analysis with flagged items removed from the matching score, to a fixed point. On by default because a DIF item left in the criterion contaminates the ability match used to test every other item. difR in R defaults to no purification, so leave this on for defensibility and turn it off only to reproduce a difR run. | |
| scored | No | True if the cells already hold 0/1 item scores. False if they hold the option each examinee selected (e.g. 'A', 'C'), which requires `answer_key`. This is never inferred: a raw-option file read as scored produces a complete, plausible-looking analysis of nothing. Confirm it with `describe_dataset` before trusting any statistic. | |
| id_column | No | Name of the examinee identifier column. Naming it excludes it from the item columns; leaving it unnamed makes it an extra 'item'. Identifiers are used only for exclusion and are never returned. | |
| answer_key | No | Correct option for each item, in the same order as the item columns. Required when `scored=False`; ignored otherwise. Its length must equal the item count -- a key that is off by one mis-scores every item after the offset, and the result looks like a form-wide item-quality problem. One entry per item, holding the option itself -- not its position, and not the whole key as a single string. | |
| focal_group | No | Value in `group_column` whose examinees are tested against the reference group. Omit to run every non-reference level in turn; note that the false-discovery correction is applied within each comparison, so multiplicity compounds across them. A value found IN the column, not the column name. | |
| linear_form | No | True if every examinee saw the items in column order, which is what makes a trailing run of blanks readable as not-reached. Set False for adaptive or randomised-order delivery: position is then unknown, so all blanks are treated as omits and a warning says so. | |
| group_column | Yes | Name of the column holding the subgroup membership to test. DIF is a between-group comparison, so there is no default: check the column name and the subgroup sizes with `describe_dataset` first, since the smaller group's size determines whether this analysis can run at all. | |
| response_file | Yes | Path to the response file on disk, read locally. Candidate responses are passed as a path and never inline, so that a multi-thousand-examinee file does not enter the conversation transcript. | |
| missing_policy | No | How blank responses are scored. 'omit_incorrect_notreached_missing' (default) scores a blank mid-form as incorrect -- the examinee had the opportunity -- and excludes a trailing run of blanks as not-reached, because scoring those wrong confounds item difficulty with speededness. 'all_incorrect' scores every blank wrong, which makes end-of-form items look harder and less discriminating in proportion to how speeded the form is; use it only if the scoring rules genuinely penalise unreached items. 'all_missing' excludes every blank, which conditions p-values on having attempted the item and turns items examinees skip when unsure into easy-looking ones. The two non-default policies each raise a warning. | omit_incorrect_notreached_missing |
| reference_group | No | Value in `group_column` to use as the reference (comparison) group. Defaults to the largest group other than the focal one. Which group is reference flips the sign of every DIF statistic, so set it explicitly whenever the direction of the finding will be reported. A value found IN the column, not the column name. | |
| pool_thin_strata | No | Merge adjacent score strata too thin to contribute to the Mantel-Haenszel table. Recovers information at the score extremes on small samples, at the cost of matching examinees less exactly on ability -- the exact match is the assumption the method rests on, so this is off by default. Affects Mantel-Haenszel and the standardised p-difference only; logistic DIF does not stratify. The choice is recorded in `method_notes.matching_criterion`. | |
| effect_size_bands | No | Effect-size thresholds for logistic DIF on the Nagelkerke delta-R-squared scale. 'jodoin_gierl' (default, 0.035/0.070) is recalibrated to align with the ETS A/B/C categories. 'zumbo_thomas' (0.13/0.26) is far more permissive and will flag materially fewer items. Report which one was used -- the choice changes the flag count, which is why it is a parameter rather than a buried constant. | jodoin_gierl |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |