Introduction to aiDIF

1. Overview

Classical differential item functioning (DIF) asks whether an item behaves differently across groups within a single scoring condition. When responses are scored by an AI system rather than by human raters, a second question appears: does moving from human to AI scoring change the group contrast?

aiDIF separates three quantities, using the robust scaling framework of Halpin (2024):

An item can carry substantial DIF under both conditions and have a DASB of zero: the AI reproduced whatever the human raters were doing. Conversely, an item with no DIF under either condition in isolation can still show a nonzero DASB. The three quantities answer different questions.

2. Who should use aiDIF

aiDIF is for psychometricians, assessment researchers, and quantitative analysts who already have item calibrations from both scoring conditions and want an integrated workflow for robust DIF estimation, cross-scoring comparison, diagnostics, and visualisation.

It assumes comfort with IRT calibration output: slope and intercept estimates and their asymptotic covariance matrices. aiDIF operates after calibration. It never sees respondent-level item responses, which keeps it independent of any particular estimation package but means the quality of its input is your responsibility.

3. The example data

make_aidif_eg() returns item parameter estimates for six items in two groups under both scoring conditions, with a known structure:

Item 1 is the interesting contrast. It has real DIF, but the AI reproduces it exactly, so its DASB should be near zero.

eg <- make_aidif_eg()
str(eg, max.level = 2)
#> List of 2
#>  $ human:List of 3
#>   ..$ par.names:List of 2
#>   ..$ est      :List of 2
#>   ..$ var.cov  :List of 2
#>  $ ai   :List of 3
#>   ..$ par.names:List of 2
#>   ..$ est      :List of 2
#>   ..$ var.cov  :List of 2

4. Calibration and identification

This section matters more than its length suggests.

DASB is a difference of differences across four calibrations: two groups times two scoring conditions. If each of those calibrations carries its own arbitrary metric origin \(c_{gs}\), then the constants contribute

\[(c_{FA} - c_{FH} - c_{RA} + c_{RH})\]

to every item’s DASB. Raw DASB is therefore identified only up to an item-common additive offset, and that offset is perfectly confounded with a uniform DASB affecting all items equally.

There are two ways to proceed.

Common metric. If linking was established upstream, so that the four calibrations are demonstrably on the same metric, the offset is zero and raw DASB is interpretable as it stands. This is the default.

fit_common <- fit_aidif(eg$human, eg$ai, metric = "common")

Linked. Otherwise, estimate the offset from the data. aiDIF applies the same bi-square M-estimator it already uses for within-condition robust scaling to the DASB vector, and reports DASB relative to the estimated offset. This assumes most items have no scoring-condition-by-group effect — the same majority assumption that underlies robust DIF scaling generally.

fit_linked <- fit_aidif(eg$human, eg$ai, metric = "linked")
attr(fit_linked$scoring_bias, "offset")
#> [1] -0.005274666

Do not skip this choice. Subtracting intercepts from separately normalised latent scales without a defensible linking strategy produces numbers that look fine and mean nothing.

5. Independent and paired scoring designs

The variance of DASB depends on how the two conditions were produced.

Independent (the default) sums the four marginal variances. It is correct when the human and AI calibrations come from disjoint respondent samples.

Paired — the same responses scored twice — requires the cross-condition covariance within each group. Groups remain independent of each other, since they consist of disjoint respondents:

\[\mathrm{Var}(\mathrm{DASB}_i) = \sum_{g,s}\mathrm{Var}(d_{igs}) - 2\mathrm{Cov}(d_{iFH}, d_{iFA}) - 2\mathrm{Cov}(d_{iRH}, d_{iRA}).\]

That covariance is normally positive, so the independence variance is an upper bound for paired data. Using design = "independent" on paired data gives conservative p-values, not anti-conservative ones. If you do not have the cross-condition covariance, the default is the safe choice; say so in your write-up rather than pretending the design was independent.

# With cross-condition covariance available:
fit_paired <- fit_aidif(eg$human, eg$ai,
                        design    = "paired",
                        cross_cov = my_cross_cov)

6. Fitting

mod <- fit_aidif(eg$human, eg$ai, metric = "linked", adjust = "BH")
print(mod)
#> AI-DIF Analysis
#> ----------------------------------------------
#> Human scoring  - robust scale est: -0.5776  (SE: 0.0747)
#>                - DIF items flagged: 3 / 6
#> AI scoring     - robust scale est: -0.5921  (SE: 0.0748)
#>                - DIF items flagged: 3 / 6
#> DASB test      - items with differential AI bias: 1 / 6 (BH-adjusted)
#>                - design: independent   metric: linked

7. Conventional DIF in each condition

round(mod$dif_human, 4)
#>         delta     se       z  p_val
#> item1  0.5693 0.0759  7.4995 0.0000
#> item2  0.0366 0.1060  0.3448 0.7303
#> item3  0.2302 0.0623  3.6953 0.0002
#> item4  0.0163 0.0931  0.1756 0.8606
#> item5  0.2700 0.0693  3.8947 0.0001
#> item6 -0.1181 0.1232 -0.9584 0.3379
round(mod$dif_ai, 4)
#>         delta     se       z  p_val
#> item1  0.5756 0.0761  7.5596 0.0000
#> item2  0.0466 0.1046  0.4458 0.6557
#> item3  0.5499 0.0619  8.8820 0.0000
#> item4  0.0046 0.0926  0.0495 0.9605
#> item5  0.3308 0.0695  4.7559 0.0000
#> item6 -0.1455 0.1240 -1.1737 0.2405

8. DASB, intervals, and multiplicity

Six items means six tests. adjust applies a multiplicity correction across the items in this analysis; the family is the item set, not the union of the human-DIF, AI-DIF, and DASB tests.

mod$scoring_bias
#> Differential AI Scoring Bias
#>   Design: independent   Metric: linked   Adjustment: BH
#>   Estimated item-common metric offset: -0.0053
#>   Note: independence variance. Conservative if the same responses
#>         were scored under both conditions.
#> 
#>   shift_g1 shift_g2    DASB     se   lower  upper       z  p_val  p_adj
#> 1     0.13     0.12 -0.0047 0.1243 -0.2484 0.2390 -0.0380 0.9697 0.9697
#> 2     0.08     0.07 -0.0047 0.1243 -0.2484 0.2390 -0.0380 0.9697 0.9697
#> 3     0.11     0.54  0.4353 0.1534  0.1346 0.7359  2.8378 0.0045 0.0273
#> 4     0.12     0.09 -0.0247 0.1248 -0.2694 0.2200 -0.1981 0.8430 0.9697
#> 5     0.07     0.13  0.0653 0.1278 -0.1853 0.3158  0.5106 0.6096 0.9697
#> 6     0.11     0.08 -0.0247 0.1248 -0.2694 0.2200 -0.1981 0.8430 0.9697

Item 3 carries the planted scoring-condition-by-group effect, so its DASB should depart from zero. Item 1 shows that conventional DIF can persist across scoring conditions without the AI having introduced any additional differential functioning.

9. Describing the pattern of change

ai_effect_summary(mod)
#>       human_delta     ai_delta human_flag ai_flag       status
#> item1  0.56933231  0.575603316       TRUE    TRUE   stable_dif
#> item2  0.03656025  0.046648397      FALSE   FALSE stable_clean
#> item3  0.23015316  0.549884606       TRUE    TRUE   stable_dif
#> item4  0.01634948  0.004582135      FALSE   FALSE stable_clean
#> item5  0.26998185  0.330766728       TRUE    TRUE   stable_dif
#> item6 -0.11810882 -0.145530542      FALSE   FALSE stable_clean

These labels are descriptive. They compare significance flags between the two conditions, and a difference in significance is not itself a significant difference (Gelman and Stern 2006). The DASB table in the previous section is the test of whether the contrast changed; this table describes the pattern once you have that answer.

10. Anchor diagnostics

Items downweighted to near zero were effectively excluded from the robust scale estimate, which flags them as likely DIF-contaminated.

anchor_weights(mod)
#>       human_weight ai_weight
#> item1    0.0000000 0.0000000
#> item2    0.9455790 0.9100670
#> item3    0.0000000 0.0000000
#> item4    0.9861794 0.9988997
#> item5    0.0000000 0.0000000
#> item6    0.6093599 0.4496091

11. Visualisation

plot(mod, type = "dif_forest")

Forest plot of item-level DIF estimates with confidence intervals under human and AI scoring.

plot(mod, type = "dasb")

Bar chart of DASB estimates per item with confidence intervals.

plot(mod, type = "weights")

Dot plot of bi-square anchor weights per item in each scoring condition.

plot(mod, type = "rho")

Bi-square objective function against candidate location values, with the robust estimate marked.

12. Full report

summary(mod)
#> ============================================================== 
#>  AI Differential Item Functioning Analysis (aiDIF)
#> ============================================================== 
#> 
#> Items: 6   Scaling: pooled   Alpha: 0.05
#> DASB design: independent   Metric: linked   Adjustment: BH
#> 
#> --- Human scoring DIF ---------------------------------------
#>   Robust scale estimate: -0.5776  (SE: 0.0747)
#>   Iterations: 10   Multiple solutions: no
#> 
#>   Wald DIF tests:
#>         delta     se       z  p_val
#> item1  0.5693 0.0759  7.4995 0.0000
#> item2  0.0366 0.1060  0.3448 0.7303
#> item3  0.2302 0.0623  3.6953 0.0002
#> item4  0.0163 0.0931  0.1756 0.8606
#> item5  0.2700 0.0693  3.8947 0.0001
#> item6 -0.1181 0.1232 -0.9584 0.3379
#> 
#> --- AI scoring DIF ------------------------------------------
#>   Robust scale estimate: -0.5921  (SE: 0.0748)
#>   Iterations: 11   Multiple solutions: no
#> 
#>   Wald DIF tests:
#>         delta     se       z  p_val
#> item1  0.5756 0.0761  7.5596 0.0000
#> item2  0.0466 0.1046  0.4458 0.6557
#> item3  0.5499 0.0619  8.8820 0.0000
#> item4  0.0046 0.0926  0.0495 0.9605
#> item5  0.3308 0.0695  4.7559 0.0000
#> item6 -0.1455 0.1240 -1.1737 0.2405
#> 
#> --- Differential AI Scoring Bias (DASB) ----------------------
#>   H0: the human-to-AI scoring shift is the same in both groups
#>   (Positive DASB => AI scoring disadvantages the focal group)
#> 
#> Differential AI Scoring Bias
#>   Design: independent   Metric: linked   Adjustment: BH
#>   Estimated item-common metric offset: -0.0053
#>   Note: independence variance. Conservative if the same responses
#>         were scored under both conditions.
#> 
#>   shift_g1 shift_g2    DASB     se   lower  upper       z  p_val  p_adj
#> 1     0.13     0.12 -0.0047 0.1243 -0.2484 0.2390 -0.0380 0.9697 0.9697
#> 2     0.08     0.07 -0.0047 0.1243 -0.2484 0.2390 -0.0380 0.9697 0.9697
#> 3     0.11     0.54  0.4353 0.1534  0.1346 0.7359  2.8378 0.0045 0.0273
#> 4     0.12     0.09 -0.0247 0.1248 -0.2694 0.2200 -0.1981 0.8430 0.9697
#> 5     0.07     0.13  0.0653 0.1278 -0.1853 0.3158  0.5106 0.6096 0.9697
#> 6     0.11     0.08 -0.0247 0.1248 -0.2694 0.2200 -0.1981 0.8430 0.9697
#> 
#> --- AI-effect classification ---------------------------------
#>   Descriptive only. It compares significance flags between the two
#>   conditions; a change in flag status is not itself evidence of a
#>   change in effect (Gelman & Stern, 2006). The DASB table above is
#>   the test of whether the group contrast actually changed.
#> 
#>       human_delta     ai_delta human_flag ai_flag       status
#> item1  0.56933231  0.575603316       TRUE    TRUE   stable_dif
#> item2  0.03656025  0.046648397      FALSE   FALSE stable_clean
#> item3  0.23015316  0.549884606       TRUE    TRUE   stable_dif
#> item4  0.01634948  0.004582135      FALSE   FALSE stable_clean
#> item5  0.26998185  0.330766728       TRUE    TRUE   stable_dif
#> item6 -0.11810882 -0.145530542      FALSE   FALSE stable_clean
#> 
#>   Status counts:
#> 
#> stable_clean   stable_dif 
#>            3            3

summary() returns an object, so components can be extracted rather than scraped from printed output:

s <- summary(mod, adjust = "holm")
s$dasb$p_adj
#> [1] 1.00000000 1.00000000 0.02725664 1.00000000 1.00000000 1.00000000

13. Using aiDIF after calibration with mirt

as_aidif() converts a fitted multiple-group model directly into the structure aiDIF expects, replacing the manual parameter and covariance extraction earlier versions required. mirt is in Suggests, so this section runs only when it is installed.

set.seed(2026)
n <- 600
J <- 8

a <- runif(J, 0.8, 1.6)
d <- rnorm(J, 0, 0.6)

sim_group <- function(n, a, d, mu = 0, d_shift = rep(0, length(d))) {
  th <- rnorm(n, mu, 1)
  p  <- t(vapply(th, function(t) plogis(a * t + d + d_shift), numeric(length(d))))
  matrix(rbinom(length(p), 1, p), nrow = n)
}

# Human scoring: item 1 has DIF in the focal group.
dif_h <- c(0.5, rep(0, J - 1))
# AI scoring adds a uniform +0.1 drift, plus DASB at item 3.
drift <- rep(0.1, J)
dasb  <- c(0, 0, 0.4, rep(0, J - 5), 0, 0)

human <- rbind(sim_group(n, a, d),
               sim_group(n, a, d + dif_h, mu = 0.5))
ai    <- rbind(sim_group(n, a, d + drift),
               sim_group(n, a, d + dif_h + drift + dasb, mu = 0.5))
grp   <- rep(c("reference", "focal"), each = n)

colnames(human) <- colnames(ai) <- paste0("item", seq_len(J))

# No invariance constraints: each group is calibrated on its own metric with
# the latent mean fixed at 0 and variance at 1. That is exactly the input
# robust scaling expects -- it estimates the linking constant itself, so
# constraining anchors here would pre-empt the method.
human_mod <- mirt::multipleGroup(as.data.frame(human), 1, group = grp,
                                 itemtype = "2PL", SE = TRUE, verbose = FALSE)
ai_mod    <- mirt::multipleGroup(as.data.frame(ai), 1, group = grp,
                                 itemtype = "2PL", SE = TRUE, verbose = FALSE)

human_in <- as_aidif(human_mod, groups = c("reference", "focal"))
ai_in    <- as_aidif(ai_mod,    groups = c("reference", "focal"))

fit_mirt <- fit_aidif(human_in, ai_in, metric = "linked", adjust = "BH")
print(fit_mirt)
#> AI-DIF Analysis
#> ----------------------------------------------
#> Human scoring  - robust scale est: 0.2777  (SE: 0.0812)
#>                - DIF items flagged: 2 / 8
#> AI scoring     - robust scale est: 0.3746  (SE: 0.1077)
#>                - DIF items flagged: 3 / 8
#> DASB test      - items with differential AI bias: 1 / 8 (BH-adjusted)
#>                - design: independent   metric: linked
fit_mirt$scoring_bias
#> Differential AI Scoring Bias
#>   Design: independent   Metric: linked   Adjustment: BH
#>   Estimated item-common metric offset: 0.0595
#>   Note: independence variance. Conservative if the same responses
#>         were scored under both conditions.
#> 
#>   shift_g1 shift_g2    DASB     se   lower  upper       z  p_val  p_adj
#> 1  -0.0545   0.2439  0.2389 0.2313 -0.2144 0.6922  1.0328 0.3017 0.7535
#> 2   0.3266   0.2694 -0.1168 0.2648 -0.6358 0.4023 -0.4409 0.6593 0.7535
#> 3  -0.0707   0.6101  0.6213 0.2122  0.2054 1.0371  2.9283 0.0034 0.0273
#> 4   0.1515   0.0991 -0.1119 0.1789 -0.4624 0.2387 -0.6254 0.5317 0.7535
#> 5  -0.0007   0.2136  0.1548 0.2201 -0.2765 0.5861  0.7033 0.4819 0.7535
#> 6   0.1598   0.1210 -0.0983 0.1653 -0.4222 0.2257 -0.5945 0.5522 0.7535
#> 7   0.2592   0.4245  0.1059 0.1953 -0.2768 0.4886  0.5422 0.5877 0.7535
#> 8   0.0966   0.1374 -0.0187 0.2083 -0.4270 0.3895 -0.0900 0.9283 0.9283

14. Simulating item-parameter inputs

simulate_aidif_data() does not simulate respondent-level item responses and refit IRT models. It directly generates item-parameter estimates and their asymptotic covariance matrices, consistent with a 2PL fitted to n_obs observations per group. It is intended for package demonstration and method benchmarking, not as a substitute for a full response-level simulation study. For the latter, use the mirt workflow in the previous section.

dat <- simulate_aidif_data(n_items = 8, n_obs = 600,
                           dif_items = c(1, 2), dif_mag = 0.5,
                           dasb_items = 5, dasb_mag = 0.4, seed = 123)
sim_mod <- fit_aidif(dat$human, dat$ai, metric = "linked", adjust = "BH")
print(sim_mod)
#> AI-DIF Analysis
#> ----------------------------------------------
#> Human scoring  - robust scale est: -0.2670  (SE: 0.0322)
#>                - DIF items flagged: 4 / 8
#> AI scoring     - robust scale est: 0.0536  (SE: 0.0363)
#>                - DIF items flagged: 5 / 8
#> DASB test      - items with differential AI bias: 1 / 8 (BH-adjusted)
#>                - design: independent   metric: linked

15. Limitations

References

Gelman, Andrew, and Hal Stern. 2006. “The Difference Between ‘Significant’ and ‘Not Significant’ Is Not Itself Statistically Significant.” The American Statistician 60 (4): 328–31. https://doi.org/10.1198/000313006X152649.
Halpin, Peter F. 2024. “Differential Item Functioning via Robust Scaling.” Psychometrika 89 (3): 796–821. https://doi.org/10.1007/s11336-024-09957-6.