Docs/CUPED

CUPED and Covariate Adjustment

Name: Zetyra
Price: 99 USD
Rating: 4.9 (47 reviews)
Author: Zetyra

Technical documentation for variance reduction using pre-experiment data. This page covers the mathematical foundations, statistical assumptions, and regulatory considerations for implementing CUPED in clinical trials and A/B testing.

1. Theoretical Foundation

CUPED (Controlled-experiment Using Pre-Experiment Data) is a specific application of ANCOVA-based covariate adjustment. While popularized in tech by Deng et al. (2013)^[1], its roots lie in classical experimental design (Fisher, 1935)^[7]. Zetyra implements this to reduce variance by leveraging pre-randomization measures to account for baseline subject heterogeneity.

Mathematical Formulation

The adjustment implemented in Zetyra:

Y_{adj} = Y - \theta(X - \mu_X)

is mathematically equivalent to the OLS regression^[1]:

Y_i = \alpha + \tau Z_i + \beta X_i + \epsilon_i

Where:

• $Z_i$ is the treatment assignment (0 or 1)
• $X_i$ is the centered baseline covariate
• $\tau$ represents the treatment effect
• $\theta = \text{Cov}(X,Y) / \text{Var}(X)$ is the optimal adjustment coefficient

Why These Are Equivalent

Zetyra uses the adjustment formula for computational efficiency, but the interpretation is identical to including $X$ as a covariate in your ANCOVA model.

2. Assumptions & Diagnostics

For the variance reduction estimates to be valid and unbiased, the following conditions must be met:

Linearity

The relationship between the covariate ( $X$ ) and the outcome ( $Y$ ) must be approximately linear.

How to check: Plot $X$ vs $Y$ . If relationship is curved, consider:

Log-transforming the covariate
Using polynomial terms ( $X^2$ )
Stratified CUPED (separate $\theta$ by subgroup)

Independence

The covariate must be measured prior to randomization to ensure it is independent of treatment assignment ( $Z \perp X$ ), preserving the Type I error rate.

How to verify: Check that $X$ was measured before randomization in your data collection timestamps. Post-randomization covariates can introduce bias.

Homoscedasticity

Residual variance should be constant across the range of the covariate.

How to check: Plot residuals vs fitted values. If variance increases with $X$ , consider robust standard errors or variance-stabilizing transformations.

Missing Data

Zetyra's current implementation assumes covariates are available for all subjects.

What to do: For trials with missing baseline data, use mean imputation or exclude subjects only if the “Missing at Random” (MAR) assumption holds. Document your approach in the SAP.

What happens when assumptions are violated? Mild violations typically result in underestimated variance reduction (conservative). Severe violations can lead to biased treatment effect estimates. Yang & Tsiatis (2001) showed that in RCTs, linear adjustment remains robust even with mild misspecification^[3].

3. Binary Endpoints: Logistic CUPED

For binary outcomes $Y \in \{0,1\}$ , linear CUPED can produce adjusted proportions outside $[0,1]$ . Zetyra uses logistic regression with G-computation to maintain valid probability bounds.

Implementation Steps

Fit logistic regression

$\text{logit}(P(Y=1)) = \alpha + \tau Z + \beta X$

Predict counterfactual outcomes

For each subject, predict outcomes under both treatment and control

Average predictions (G-computation)

Compute adjusted proportions: $\hat{P}_{adj} = \frac{1}{n}\sum_i \hat{P}(Y_i=1|Z,X_i)$

Compute effect estimate

Adjusted risk difference, relative risk, or odds ratio

Why G-computation? Unlike simple logistic coefficient interpretation, G-computation provides marginal (population-averaged) effect estimates that are directly interpretable as adjusted proportions^[5].

4. Regulatory Considerations (FDA & ICH)

CUPED adjustments in Zetyra are designed to align with FDA Guidance on Covariate Adjustment (2023) and ICH E9(R1).

Pre-specification Requirement

To satisfy regulatory scrutiny, the use of CUPED and the specific choice of covariate ( $X$ ) must be documented in the Statistical Analysis Plan (SAP) prior to unblinding^[2].

Type I Error Preservation

In randomized controlled trials (RCTs), linear adjustment for baseline covariates is generally robust and does not inflate Type I error, even if the model is slightly misspecified (Yang & Tsiatis, 2001)^[3].

Documentation for SAP

Your SAP should include:

The specific covariate(s) to be used
Justification for covariate selection
The adjustment method (linear CUPED vs. logistic)
Handling of missing covariate data

5. Covariate Selection Guidelines

Choosing the right covariate is critical for maximizing variance reduction. The correlation $\rho$ between $X$ and $Y$ determines effectiveness.

Best Practice Hierarchy

1st

Same metric, pre-period

Use baseline revenue to adjust trial revenue. Typically $\rho = 0.5\text{-}0.8$ .

2nd

Closely related metrics

Use baseline engagement to adjust conversion. Typically $\rho = 0.3\text{-}0.5$ .

3rd

Stratification variables

Use demographic covariates when pre-period data unavailable. Lower $\rho$ .

How Many Covariates?

Start with one strong covariate. Adding multiple covariates yields diminishing returns and risks overfitting. If using multiple covariates, consider ridge regression or pre-specified variable selection^[6].

Estimating $\rho$ from Historical Data

Use data from 2+ weeks before trial start
Calculate correlation between pre/post periods
Conservative planning: Use $\rho - 0.1$ for sample size calculations

6. Sample Size Reduction

With CUPED, required sample size scales by $(1 - \rho^2)$ . This determines how many fewer subjects you need compared to an unadjusted analysis.

Correlation ( $\rho$ )	Variance Reduction	Subjects Saved^*	Typical Use Case
0.3	9%	~10%	Demographic covariates
0.5	25%	~33%	Related metrics
0.7	49%	~50%	Same metric, pre-period
0.9	81%	~80%	Highly stable outcomes

^* Subjects saved = $1 - (1 - \rho^2)$ . For example, 25% variance reduction means you need $1/(1-0.25) = 33\%$ fewer subjects.

Planning Recommendation: Estimate $\rho$ from pilot data or historical trials, then use Zetyra's calculator with conservative ( $\rho - 0.1$ ) assumption to account for uncertainty.

7. Limitations & When Not to Use CUPED

CUPED provides minimal benefit in certain scenarios. Consider alternative methods when these conditions apply:

New Users

No historical data exists (e.g., onboarding experiments, first-time visitors). Alternative: Stratified randomization by acquisition channel.

Rare Events

Binary outcomes with <5% base rate have limited pre-period signal. Alternative: Post-stratification or Bayesian methods.

Short Time Windows

Less than 7 days of pre-period data yields unstable $\rho$ estimates. Alternative: Extend pre-period or use different covariates.

Low Correlation

$\rho < 0.3$ provides <9% variance reduction—often not worth the complexity. Alternative: Standard analysis with larger sample size.

Scenario	CUPED Benefit	Recommended Alternative
New user experiments	None	Stratified randomization
Rare events (<5%)	Minimal	Bayesian methods
<7 days pre-period	Unreliable	Extend observation window
$\rho < 0.3$	<9%	Standard analysis

8. Validation & Benchmarking

Zetyra calculations are benchmarked against industry-standard software to ensure regulatory-grade accuracy.

Continuous Endpoint Validation

Parameter	Value	Parameter	Value
Alpha ( $\alpha$ )	0.05	Effect ( $\delta$ )	2.0
Power ( $1-\beta$ )	0.80	Std Dev ( $\sigma$ )	25
Correlation ( $\rho$ )	0.70	Allocation	1:1

Unadjusted N

2,453

PASS 2024

1,252

Zetyra