Generalizability Path Example: Characterizing Underrepresented Populations

Overview

A common challenge in translating evidence from randomized controlled trials (RCTs) to real-world practice is that trial participants may not reflect the broader target population. Following Parikh et al. (2025), a subgroup is “underrepresented” or “insufficiently represented” when the trial contains too little information about it to estimate its treatment effect precisely. This happens mainly when the subgroup is both poorly represented in the trial and has a treatment effect that differs from the rest of the population. If such subgroups are present, estimates of the Target Average Treatment Effect (TATE) can be imprecise when the trial results are transported to the target population. The Sample Target Average Treatment Effect (STATE), the average treatment effect over the target sample, is a finite sample equivalent version of the TATE.

ROOT identifies these subgroups with an interpretable decision tree and reports the Weighted Target Average Treatment Effect (WTATE): the average treatment effect in the sufficiently represented subpopulation, which can be estimated with lower variance than the unweighted STATE.

This vignette walks through a complete generalizability analysis using the built-in diabetes_data dataset.

The diabetes_data Dataset

diabetes_data is a simulated dataset that mimics a randomized trial of a glucose-lowering drug (S = 1) and a broader target population of adults with type 2 diabetes (S = 0). The outcome Y is the reduction in HbA1c (percentage points) at 6 months; treatment and outcome are observed only in the trial.

library(ROOT)

data(diabetes_data, package = "ROOT")
str(diabetes_data)
#> 'data.frame':    10000 obs. of  13 variables:
#>  $ Age         : num  64 45 60 57 50 30 49 46 59 52 ...
#>  $ Sex_Male    : int  0 0 1 1 1 0 0 0 1 0 ...
#>  $ Race_Black  : int  0 1 0 0 0 0 1 0 0 0 ...
#>  $ Rural       : int  0 0 0 1 0 0 0 0 0 0 ...
#>  $ BMI         : num  30.6 30.9 34.6 27.3 31.2 33 21.4 25.5 37.4 31.1 ...
#>  $ HbA1c       : num  6.5 9 8.1 8 7.8 8 10.1 8 7.5 6.5 ...
#>  $ Insulin     : int  1 1 0 0 0 0 1 0 0 0 ...
#>  $ CKD         : int  0 0 0 1 0 0 0 0 0 0 ...
#>  $ HeartFailure: int  0 0 0 0 0 0 0 0 0 0 ...
#>  $ DietYes     : int  0 0 1 0 1 0 0 0 0 0 ...
#>  $ S           : int  0 0 1 0 1 0 0 0 0 0 ...
#>  $ Tr          : int  NA NA 1 NA 1 NA NA NA NA NA ...
#>  $ Y           : num  NA NA 1.07 NA 1.73 ...

The columns are:

# How many trial vs target population units?
table(S = diabetes_data$S)
#> S
#>    0    1 
#> 7942 2058

# Treatment breakdown within the trial
table(Tr = diabetes_data$Tr[diabetes_data$S == 1])
#> Tr
#>    0    1 
#> 1037 1021

Checking Covariate Overlap

Before running ROOT, it is good practice to check whether trial participants differ from the target population on key covariates.

covariate_cols <- c("Age", "Sex_Male", "Race_Black", "Rural", "BMI",
                    "HbA1c", "Insulin", "CKD", "HeartFailure", "DietYes")

overlap <- sapply(covariate_cols, function(v) {
  tapply(diabetes_data[[v]], diabetes_data$S, mean, na.rm = TRUE)
})

knitr::kable(
  t(overlap),
  digits  = 3,
  caption = "Covariate means by sample membership (S = 1: trial, S = 0: target)"
)

Chronic kidney disease, heart failure, insulin use, Black race, rural residence, and older age are all less common in the trial than in the target population. A comparison of covariate means cannot tell us which of these differences matter for transporting the treatment effect: that depends on which covariates also modify the effect. ROOT answers this question directly.

Fitting ROOT in Generalizability Mode

We use characterizing_underrep(), which is the high-level wrapper around ROOT() for generalizability/transportability analyses. It expects data to contain Y, Tr, and S, and internally:

1. Estimates transportability scores using logistic regression models (default) for \(P(S = 1 \mid X)\) and \(P(\text{Tr} = 1 \mid X, S = 1)\).
2. Constructs inverse propensity weighting (IPW) transported scores \(v_i\).
3. Grows a forest of randomized trees that minimize the variance of the weighted estimator \(\widehat{\text{WTATE}}\).
4. Selects a Rashomon set of the top-\(k\) trees and aggregates their weight assignments by vote: by default, a unit is kept if more than two-thirds of the Rashomon-set trees keep it.
5. Fits a single summary tree characterizing the final \(w_{\text{opt}}\) assignments.

Because Age, BMI, and HbA1c are continuous, we use split_strategy = "random_quantile", which proposes split points at random quantiles of each covariate rather than always at the midpoint of its range. This lets the trees search over many candidate thresholds. We also grow more trees than the default to explore the larger covariate space, and lower leaf_proba from its default of 0.25 to 0.1 so that individual trees can grow deep enough to isolate several subgroups. Because the final rule excludes four separate subgroups, we also set summary_max_depth = 4 so that the summary tree is deep enough to describe it (the default is 3). By default, ROOT also requires every candidate rule to keep at least 10% of the trial participants (min_kept = 0.1). This guards against a degenerate solution: the estimated standard error can be made almost zero by keeping only a handful of participants whose scores happen to be similar, which a thorough search would otherwise find.

gen_fit <- characterizing_underrep(
  data                  = diabetes_data,
  generalizability_path = TRUE,
  num_trees             = 500,
  top_k_trees           = TRUE,
  k                     = 10,
  split_strategy        = "random_quantile",
  leaf_proba            = 0.1,
  summary_max_depth     = 4,
  seed                  = 123
)

Inspecting the Results

print(gen_fit)
#> characterizing_underrep object
#>   --- ROOT brief summary ---
#> ROOT object
#>   Generalizability mode: TRUE 
#> 
#> Summary classifier (f):
#> n= 2058 
#> 
#> node), split, n, loss, yval, (yprob)
#>       * denotes terminal node
#> 
#>  1) root 2058 689 1 (0.33479106 0.66520894)  
#>    2) Insulin>=0.5 365   0 0 (1.00000000 0.00000000) *
#>    3) Insulin< 0.5 1693 324 1 (0.19137626 0.80862374)  
#>      6) Age>=66.5 222   0 0 (1.00000000 0.00000000) *
#>      7) Age< 66.5 1471 102 1 (0.06934058 0.93065942)  
#>       14) HeartFailure>=0.5 82   1 0 (0.98780488 0.01219512) *
#>       15) HeartFailure< 0.5 1389  21 1 (0.01511879 0.98488121) *
#> 
#> Estimand summary (generalizability mode):
#>   Unweighted STATE = 1.186458, SE = 0.08287386
#>   Weighted   WTATE = 0.8839887, SE = 0.04516677

summary(gen_fit)
#> characterizing_underrep object
#>   --- ROOT summary ---
#> ROOT object
#>   Generalizability mode: TRUE 
#> 
#> Summary classifier (f):
#> n= 2058 
#> 
#> node), split, n, loss, yval, (yprob)
#>       * denotes terminal node
#> 
#>  1) root 2058 689 1 (0.33479106 0.66520894)  
#>    2) Insulin>=0.5 365   0 0 (1.00000000 0.00000000) *
#>    3) Insulin< 0.5 1693 324 1 (0.19137626 0.80862374)  
#>      6) Age>=66.5 222   0 0 (1.00000000 0.00000000) *
#>      7) Age< 66.5 1471 102 1 (0.06934058 0.93065942)  
#>       14) HeartFailure>=0.5 82   1 0 (0.98780488 0.01219512) *
#>       15) HeartFailure< 0.5 1389  21 1 (0.01511879 0.98488121) *
#> 
#> Global objective function:
#>   User-supplied: No (default objective used)
#> 
#> Estimand summary (generalizability mode):
#>   Unweighted STATE = 1.186458, SE = 0.08287386
#>   Weighted   WTATE = 0.8839887, SE = 0.04516677
#> 
#> Diagnostics:
#>   Number of trees grown: 500
#>   Rashomon set size: 10
#>   Split strategy: random_quantile
#>   % observations with w_opt == 1: 66.5%
#> 
#>   Leaf summary:    4 terminal nodes
#>  leaf_id                                                rule predicted_w    n
#>        2                                 root & Insulin>=0.5           0  365
#>        6                     root & Insulin< 0.5 & Age>=66.5           0  222
#>       14 root & Insulin< 0.5 & Age< 66.5 & HeartFailure>=0.5           0   82
#>       15 root & Insulin< 0.5 & Age< 66.5 & HeartFailure< 0.5           1 1389
#>   pct                           label
#>  17.7 Under-represented (drop, w = 0)
#>  10.8 Under-represented (drop, w = 0)
#>   4.0 Under-represented (drop, w = 0)
#>  67.5       Represented (keep, w = 1)

gen_fit$leaf_summary
#>   leaf_id                                                rule predicted_w    n
#> 1       2                                 root & Insulin>=0.5           0  365
#> 2       6                     root & Insulin< 0.5 & Age>=66.5           0  222
#> 3      14 root & Insulin< 0.5 & Age< 66.5 & HeartFailure>=0.5           0   82
#> 4      15 root & Insulin< 0.5 & Age< 66.5 & HeartFailure< 0.5           1 1389
#>    pct                           label
#> 1 17.7 Under-represented (drop, w = 0)
#> 2 10.8 Under-represented (drop, w = 0)
#> 3  4.0 Under-represented (drop, w = 0)
#> 4 67.5       Represented (keep, w = 1)

Visualizing the Characterization Tree

plot() renders the final characterized tree from the Rashomon set. Blue leaves (\(w = 1\)) denote well-represented subgroups; orange leaves (\(w = 0\)) denote underrepresented subgroups. The percentage shown in each leaf is the share of trial units falling into that node.

plot(gen_fit)

The tree reads top-down as a decision rule. Follow the branches from the root (all trial units) to each leaf to read the complete inclusion/exclusion rule for that subgroup.

Interpreting the Output

From the characterized tree and leaf summary, we can describe the underrepresented subgroups in plain language:

• Participants who use insulin are flagged as underrepresented (\(w = 0\)), regardless of their other characteristics.
• Participants aged 67 or older who do not use insulin are flagged as underrepresented.
• Participants with heart failure among the remaining (younger, non-insulin) participants are flagged as underrepresented.
• The remaining participants, those younger than 67 who use neither insulin nor have heart failure, form the well-represented subpopulation (\(w = 1\)) for whom the WTATE is estimated.

These flags match three of the four effect modifiers in the simulation. Insulin use, older age, and heart failure are all under-sampled in the trial and strongly modify the treatment effect.

The age cut-off ROOT selects (67) is a few years below the true threshold of 70. Trial participation declines steadily with age, so participants aged 67 to 69 are also somewhat under-sampled, and excluding them costs little precision because their treatment effect is the same as that of younger participants. A cut-off a few years early is therefore a conservative choice rather than an error.

The fourth effect modifier, chronic kidney disease, is not flagged consistently. Only about 45 trial participants fall in the retained region and have CKD, and the Rashomon-set vote excludes only some of them; they are spread across the large retained leaf, so the depth-limited summary tree does not split on CKD at all. ROOT is a randomized search without an optimality guarantee, and a small subgroup like this one is the hardest to isolate: each candidate tree must find it in addition to the three larger subgroups. Growing more trees, or trying other seeds, makes it more likely that the Rashomon set excludes it. The consequence for the estimand is visible in the comparison with the truth below.

Equally important is what the tree leaves out:

• Black participants and rural residents are kept, even though both groups are clearly under-sampled in the trial. Their treatment effect is no different from that of other participants, so excluding them would shrink the target population without improving precision.
• Baseline HbA1c, a mild effect modifier that is well represented in the trial, does not appear in the tree.
• Sex_Male, BMI, and DietYes do not appear in the tree.

An approach that flagged subgroups by their trial participation probability alone would treat Black and rural participants the same way as participants with CKD or heart failure. ROOT distinguishes them because it accounts for both representation and treatment-effect heterogeneity, which is the central idea of Parikh et al. (2025).

trial_data <- subset(diabetes_data, S == 1)
w_opt  <- gen_fit$root$D_rash$w_opt
w_tree <- as.integer(as.character(
  predict(gen_fit$root$f, newdata = trial_data, type = "class")
))

c(share_kept_w_opt = mean(w_opt),
  share_kept_tree  = mean(w_tree),
  agreement        = mean(w_opt == w_tree))
#> share_kept_w_opt  share_kept_tree        agreement 
#>        0.6652089        0.6749271        0.9893100

true_cate <- function(d) {
  with(d, 1 + 0.1 * (HbA1c - 8) - 2.0 * CKD - 2.4 * HeartFailure +
         1.5 * Insulin + 2.0 * (Age >= 70))
}

target_data <- subset(diabetes_data, S == 0)
tau_target  <- true_cate(target_data)
w_target    <- as.integer(as.character(
  predict(gen_fit$root$f, newdata = target_data, type = "class")
))

est <- gen_fit$root$estimate
data.frame(
  estimand = c("STATE", "WTATE"),
  true     = c(mean(tau_target), mean(tau_target[w_target == 1])),
  estimate = c(est$value_unweighted, est$value_weighted),
  SE       = c(est$se_unweighted, est$se_weighted)
)
#>   estimand      true  estimate         SE
#> 1    STATE 1.1176379 1.1864577 0.08287386
#> 2    WTATE 0.7827431 0.8839887 0.04516677

Key Parameters

Reference

Parikh, H., Ross, R. K., Stuart, E., & Rudolph, K. E. (2025). Who Are We Missing?: A Principled Approach to Characterizing the Underrepresented Population. Journal of the American Statistical Association, 120(551), 1414–1423. https://doi.org/10.1080/01621459.2025.2495319