| Title: | What Works Clearinghouse Standards for Education Impact Evaluations |
|---|---|
| Description: | Applies the group-design determinations of the What Works Clearinghouse (WWC) to education impact studies. Computes WWC effect sizes (Hedges' g with the small-sample correction, and the Cox index) and classifies baseline equivalence; classifies overall and differential attrition against the WWC attrition boundary; returns the group-design study rating; and reports the robustness of the baseline-equivalence verdict as a multiverse over the computation choices an analyst could defensibly make differently. Provides report-ready tables and Love plots. |
| Authors: | Yuxia Liang [aut, cre] (ORCID: <https://orcid.org/0009-0008-6489-3836>) |
| Maintainer: | Yuxia Liang <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.6.0.9000 |
| Built: | 2026-10-30 13:00:58 UTC |
| Source: | https://github.com/zl1212-ship-it/baselinr |
Computes overall and differential sample attrition for a two-group design, the inputs to the What Works Clearinghouse (WWC) attrition standard. Differential attrition is the absolute difference between the treatment and comparison attrition rates.
attrition(treatment, retained, na.rm = TRUE)attrition(treatment, retained, na.rm = TRUE)
treatment |
Vector identifying group membership; exactly two unique
non-missing values (the larger is treated as the treatment group, as in
|
retained |
Logical (or |
na.rm |
Logical; drop rows where |
This function reports the attrition rates; it does not classify them. Compare the overall and differential rates against the WWC attrition boundary for your chosen response assumption (cautious or optimistic) in the Procedures Handbook.
A one-row data frame with columns attrition_overall,
attrition_treatment, attrition_comparison, and
differential_attrition (all proportions).
What Works Clearinghouse (2022). Procedures and Standards Handbook (Version 5.0). U.S. Department of Education.
set.seed(1) g <- rep(c(1, 0), each = 100) kept <- rbinom(200, 1, ifelse(g == 1, 0.9, 0.8)) attrition(g, kept)set.seed(1) g <- rep(c(1, 0), each = 100) kept <- rbinom(200, 1, ifelse(g == 1, 0.9, 0.8)) attrition(g, kept)
Given a study's overall and differential attrition, classifies it as low or
high attrition against the What Works Clearinghouse (WWC) attrition boundary
(Standards Handbook Version 4.1, Table II.1). This is the classification that
attrition() deliberately leaves to the user: attrition() reports the
rates, attrition_boundary() applies the standard.
attrition_boundary( overall, differential, assumption = c("cautious", "optimistic") )attrition_boundary( overall, differential, assumption = c("cautious", "optimistic") )
overall |
Overall attrition, as a proportion in |
differential |
Differential attrition, as a proportion (the absolute
difference in group attrition rates, as returned by |
assumption |
Which boundary to apply: |
The WWC uses one of two boundaries. The cautious boundary is applied when the intervention could plausibly affect attrition (for example, a dropout prevention program); the optimistic boundary when it is unlikely to (for example, a first-grade reading program). The applicable boundary is set by the review protocol, not chosen post hoc; the default here is the more conservative cautious boundary.
A data frame, one row per input, with columns overall,
differential, assumption, max_differential (the highest differential
attrition still counted as low, as a proportion; NA where the overall rate
is beyond the boundary), and attrition ("low" or "high").
What Works Clearinghouse (2020). Standards Handbook, Version 4.1, Table II.1. U.S. Department of Education.
# A study with 8% overall and 3 percentage-point differential attrition: attrition_boundary(0.08, 0.03) # low under the cautious boundary # 30% overall, 6-point differential: high if cautious, low if optimistic attrition_boundary(0.30, 0.06, "cautious") attrition_boundary(0.30, 0.06, "optimistic") # Chained from attrition(): set.seed(1) g <- rep(c(1, 0), each = 100) kept <- rbinom(200, 1, ifelse(g == 1, 0.9, 0.82)) a <- attrition(g, kept) attrition_boundary(a$attrition_overall, a$differential_attrition)# A study with 8% overall and 3 percentage-point differential attrition: attrition_boundary(0.08, 0.03) # low under the cautious boundary # 30% overall, 6-point differential: high if cautious, low if optimistic attrition_boundary(0.30, 0.06, "cautious") attrition_boundary(0.30, 0.06, "optimistic") # Chained from attrition(): set.seed(1) g <- rep(c(1, 0), each = 100) kept <- rbinom(200, 1, ifelse(g == 1, 0.9, 0.82)) a <- attrition(g, kept) attrition_boundary(a$attrition_overall, a$differential_attrition)
Builds a report-ready baseline-equivalence table for a set of covariates, reporting group sample sizes, summaries, the appropriate standardized effect size, and the corresponding What Works Clearinghouse (WWC) equivalence category for each covariate. Continuous covariates use Hedges' g; binary covariates use the Cox index.
baseline_equivalence( data, treatment, covariates = NULL, missing = c("pairwise", "complete"), report_missing = FALSE )baseline_equivalence( data, treatment, covariates = NULL, missing = c("pairwise", "complete"), report_missing = FALSE )
data |
A data frame. |
treatment |
String naming the column in |
covariates |
Character vector of column names to evaluate. Defaults to
all numeric, logical, and factor columns in |
missing |
How to handle missing covariate values. |
report_missing |
Logical; if |
A covariate with exactly two unique non-missing values is treated as binary;
any other numeric covariate is treated as continuous. A non-numeric covariate
with more than two categories is not supported and raises an error. The
binary-versus-continuous decision uses the sample actually contributing to
the row, so it can differ between missing = "pairwise" and
missing = "complete" when deletion removes all cases at one value.
Under WWC review, baseline equivalence must be established on the analytic
sample, the cases actually contributing to the impact estimate. This
function sees only the covariates, never the outcome, so neither option
identifies the analytic sample by itself; to establish equivalence on the
analytic sample, pass that subset of cases as data. Within the data
supplied, missing = "complete" keeps every row on one common set of
cases, as a complete-case analysis would, while "pairwise" uses all
available information per covariate. report_missing = TRUE documents how
much is missing in each group, complementing the sample-loss reporting in
attrition().
A data frame with one row per covariate and the columns:
covariate; type ("continuous" or "binary"); n_treatment,
n_comparison; mean_treatment, mean_comparison (group means for
continuous covariates, event proportions for binary ones); sd_treatment,
sd_comparison; effect_size (Hedges' g or Cox index, per type); and
wwc_category. If report_missing = TRUE, the columns
missing_treatment and missing_comparison appear after the
sample-size columns.
What Works Clearinghouse (2022). Procedures and Standards Handbook (Version 5.0). U.S. Department of Education.
df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) baseline_equivalence(df, treatment = "treat")df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) baseline_equivalence(df, treatment = "treat")
Corrects the statistical significance of a finding for clustering, following the What Works Clearinghouse (WWC) procedure based on Hedges (2007). The correction applies when assignment was at the cluster level (classrooms or schools) but the analysis was at the student level, a "mismatch" that leaves the reported standard errors too small. It computes the t statistic implied by the effect size, then corrects both the t statistic and its degrees of freedom for clustering, and returns the clustering-corrected two-tailed p value and significance decision.
cluster_correction( g, n_treatment, n_comparison, m_treatment, m_comparison, icc = NULL, outcome = c("achievement", "other") )cluster_correction( g, n_treatment, n_comparison, m_treatment, m_comparison, icc = NULL, outcome = c("achievement", "other") )
g |
Effect size (Hedges' g) ignoring clustering. |
n_treatment, n_comparison
|
Student-level sample sizes. |
m_treatment, m_comparison
|
Number of clusters in each group. |
icc |
Intraclass correlation. If |
outcome |
|
Because the correction can only reduce significance, the WWC applies it only to findings the study authors reported as statistically significant; a finding that was not significant stays not significant. This function computes the corrected values regardless; apply them where the WWC rules call for it.
A data frame, one row per input, with columns t (ignoring
clustering), t_corrected, df (corrected degrees of freedom), p_value
(clustering-corrected, two-tailed), and significant (p_value < 0.05).
What Works Clearinghouse (2020). Procedures Handbook, Version 4.1, Appendix F. Hedges, L. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341-370.
# A finding with a moderate effect from a clustered design: cluster_correction( g = 0.30, n_treatment = 200, n_comparison = 200, m_treatment = 10, m_comparison = 10, outcome = "achievement" )# A finding with a moderate effect from a clustered design: cluster_correction( g = 0.30, n_treatment = 200, n_comparison = 200, m_treatment = 10, m_comparison = 10, outcome = "achievement" )
Computes the What Works Clearinghouse (WWC) Cox index, a standardized effect size for a binary (dichotomous) covariate. The Cox index places the difference between two proportions on a scale comparable to Hedges' g, so it can be classified with the same baseline-equivalence thresholds.
cox_index(x, treatment, na.rm = TRUE)cox_index(x, treatment, na.rm = TRUE)
x |
A binary covariate (numeric |
treatment |
Vector the same length as |
na.rm |
Logical; drop rows where |
The index is , where and are the proportions in the "event"
category for the treatment and comparison groups.
A single numeric value: the Cox index. Returns NA (with a warning)
when a group proportion is exactly 0 or 1, where the index is undefined.
What Works Clearinghouse (2022). Procedures and Standards Handbook (Version 5.0). U.S. Department of Education.
x <- c(1, 1, 1, 1, 0, 1, 0, 0) g <- c(1, 1, 1, 1, 0, 0, 0, 0) cox_index(x, g)x <- c(1, 1, 1, 1, 0, 1, 0, 0) g <- c(1, 1, 1, 1, 0, 0, 0, 0) cox_index(x, g)
Renders the result of baseline_equivalence() as a formatted gt table
with rounded statistics and readable column labels. Requires the gt
package.
gt_baseline(equivalence, decimals = 2)gt_baseline(equivalence, decimals = 2)
equivalence |
A data frame returned by |
decimals |
Number of decimal places for the numeric columns. Default 2. |
A gt_tbl object.
if (requireNamespace("gt", quietly = TRUE)) { df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) tbl <- gt_baseline(baseline_equivalence(df, "treat")) }if (requireNamespace("gt", quietly = TRUE)) { df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) tbl <- gt_baseline(baseline_equivalence(df, "treat")) }
Computes the standardized mean difference (Hedges' g) between a treatment and a comparison group for a single numeric covariate, using the pooled within-group standard deviation and the small-sample correction factor used by the What Works Clearinghouse (WWC).
hedges_g(x, treatment, na.rm = TRUE)hedges_g(x, treatment, na.rm = TRUE)
x |
Numeric vector of covariate values. |
treatment |
Vector the same length as |
na.rm |
Logical; drop rows where |
The correction factor is , where
.
A single numeric value: Hedges' g. Positive when the treatment group mean exceeds the comparison group mean.
What Works Clearinghouse (2022). Procedures and Standards Handbook (Version 5.0). U.S. Department of Education.
x <- c(5, 6, 7, 4, 5, 6) g <- c(1, 1, 1, 0, 0, 0) hedges_g(x, g) # 0.8x <- c(5, 6, 7, 4, 5, 6) g <- c(1, 1, 1, 0, 0, 0) hedges_g(x, g) # 0.8
Plots the standardized effect size for each covariate from
baseline_equivalence(), with reference lines at the What Works
Clearinghouse (WWC) thresholds (0.05 and 0.25) and points coloured by WWC
category. Requires the ggplot2 package.
love_plot(equivalence, signed = FALSE)love_plot(equivalence, signed = FALSE)
equivalence |
A data frame returned by |
signed |
Logical. If |
A ggplot object.
if (requireNamespace("ggplot2", quietly = TRUE)) { df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) love_plot(baseline_equivalence(df, "treat")) }if (requireNamespace("ggplot2", quietly = TRUE)) { df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) love_plot(baseline_equivalence(df, "treat")) }
A small, simulated (not real) dataset for demonstrating baseline equivalence assessment in a quasi-experimental education evaluation. It represents 400 students: 200 who received a tutoring program and 200 comparison students who did not, with baseline covariates measured before the program and an outcome measured after. The treatment group is mildly positively selected, so the covariates span all three What Works Clearinghouse equivalence categories.
tutoringtutoring
A data frame with 400 rows and 8 variables:
Treatment indicator: 1 = received tutoring, 0 = comparison.
Baseline reading score (continuous).
Baseline attendance rate, 0-1 (continuous).
Age in years at baseline (continuous).
1 = female, 0 = not (binary).
Eligible for free or reduced-price lunch: 1 = yes (binary).
English language learner: 1 = yes (binary).
Reading score after the program (continuous outcome).
Simulated for package examples with data-raw/tutoring.R; not real
student data.
Maps standardized effect sizes to the three What Works Clearinghouse baseline-equivalence categories. Sign is ignored; classification uses the absolute value of the effect size.
wwc_classify(es)wwc_classify(es)
es |
Numeric vector of standardized effect sizes (e.g. values returned
by |
A character vector the same length as es:
"satisfied" when |es| <= 0.05 (no adjustment needed),
"satisfied_with_adjustment" when 0.05 < |es| <= 0.25
(equivalence holds only if the covariate is adjusted for in the
impact model),
"not_satisfied" when |es| > 0.25.
NA inputs return NA.
What Works Clearinghouse (2022). Procedures and Standards Handbook (Version 5.0). U.S. Department of Education.
wwc_classify(c(0.03, 0.12, 0.80))wwc_classify(c(0.03, 0.12, 0.80))
Applies the What Works Clearinghouse (WWC) group-design rating logic to the two determinations this package supports, sample attrition and baseline equivalence, and returns the study's rating. This encodes the main decision path of the WWC Standards Handbook (Version 4.1, Section II): a randomized controlled trial (RCT) with low attrition can meet standards without reservations; an RCT with high attrition is held to the same baseline requirement as a quasi-experimental design (QED); and a QED, or a high attrition RCT, meets standards with reservations only if baseline equivalence is established, and otherwise does not meet standards.
wwc_rating(design, baseline, attrition = NULL)wwc_rating(design, baseline, attrition = NULL)
design |
|
baseline |
The overall baseline-equivalence verdict, one of
|
attrition |
For an RCT, |
This function covers the attrition-and-equivalence path only. A real WWC review also checks that random assignment was not compromised, that there are no confounding factors, and that the required baseline measures were used; those judgments are the reviewer's and are assumed satisfied here.
A length-one character string: one of
"Meets WWC Group Design Standards Without Reservations",
"Meets WWC Group Design Standards With Reservations", or
"Does Not Meet WWC Group Design Standards". The reasoning is attached as
attr(x, "basis").
What Works Clearinghouse (2020). Standards Handbook, Version 4.1, Section II. U.S. Department of Education.
# Low-attrition RCT: meets without reservations regardless of baseline. wwc_rating("rct", baseline = "not_satisfied", attrition = "low") # High-attrition RCT hinges on baseline equivalence, like a QED. wwc_rating("rct", baseline = "satisfied_with_adjustment", attrition = "high") # QED that fails baseline equivalence does not meet standards. wwc_rating("qed", baseline = "not_satisfied")# Low-attrition RCT: meets without reservations regardless of baseline. wwc_rating("rct", baseline = "not_satisfied", attrition = "low") # High-attrition RCT hinges on baseline equivalence, like a QED. wwc_rating("rct", baseline = "satisfied_with_adjustment", attrition = "high") # QED that fails baseline equivalence does not meet standards. wwc_rating("qed", baseline = "not_satisfied")
Reports how stable a baseline-equivalence verdict is to the computation choices a careful analyst might defensibly make differently. For each continuous covariate it recomputes the standardized difference under the cross of two choices, standardizing by the pooled versus the comparison-group standard deviation and applying the What Works Clearinghouse (WWC) small-sample correction or not, and records whether the covariate's WWC category changes. Binary covariates use the Cox index, which does not depend on these choices. It also reports whether the overall verdict changes.
wwc_robustness(data, treatment, covariates = NULL)wwc_robustness(data, treatment, covariates = NULL)
data |
A data frame. |
treatment |
String naming the treatment-indicator column (see
|
covariates |
Character vector of covariate columns. Defaults to all
eligible columns other than |
This is a multiverse, or specification-curve, view of a single WWC determination: it shows whether the verdict depends on which defensible choice is made.
A data frame with one row per covariate and the columns covariate,
type, category_default (the category under baselinr's default), the set
of categories the covariate takes across the defensible choices, flips
(whether that set has more than one category), and abs_es_min /
abs_es_max (the range of the absolute effect size across choices). The
overall verdict under each choice is attached as attr(x, "overall"), and
attr(x, "overall_stable") is TRUE when the overall verdict is invariant.
What Works Clearinghouse (2022). Procedures and Standards Handbook (Version 5.0). U.S. Department of Education. Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702-712.
df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) r <- wwc_robustness(df, treatment = "treat") r attr(r, "overall")df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) r <- wwc_robustness(df, treatment = "treat") r attr(r, "overall")
Summarizes a baseline_equivalence() table into a one-row overall
assessment: how many covariates fall in each What Works Clearinghouse (WWC)
category, the largest absolute effect size, and an overall verdict.
wwc_summary(equivalence)wwc_summary(equivalence)
equivalence |
A data frame returned by |
The overall verdict follows the logic of the categories: if any covariate is
"not_satisfied", baseline equivalence cannot be established
("not_satisfied"); otherwise, if any covariate requires adjustment, the
verdict is "satisfied_with_adjustment" (equivalence holds only if those
covariates are adjusted for in the impact model); otherwise "satisfied".
A one-row data frame with columns n_covariates, n_satisfied,
n_satisfied_with_adjustment, n_not_satisfied, max_abs_effect, and
overall.
What Works Clearinghouse (2022). Procedures and Standards Handbook (Version 5.0). U.S. Department of Education.
df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) wwc_summary(baseline_equivalence(df, "treat"))df <- data.frame( treat = c(1, 1, 1, 0, 0, 0), pretest = c(5, 6, 7, 4, 5, 6), female = c(1, 0, 1, 0, 0, 1) ) wwc_summary(baseline_equivalence(df, "treat"))