Experimental design

Alex Reinhart – Updated April 1, 2025 notebooks · refsmmat.com

See also Causality, Observational studies.

Many experimental design textbooks, written for practicing scientists, don’t explain why we’re going to all of this trouble to design elaborate treatment allocations. Why, exactly, do I want to use a Latin square over some other allocation of treatments? In most cases, the purpose behind designs is control of estimation variance: by choosing treatment allocation carefully, we can obtain a treatment effect estimate that has the lowest possible variance, given our sample size constraints. This involves clever tricks like making treatment effects orthogonal to other effects by design.

As causal inference

As linear algebra

Most experimental design books focus heavily on sums of squares and tedious algebraic manipulations to show the various properties of estimators, contrasts, and so on. But designs can be represented in matrix form and properties of the estimators derived via linear algebra—though I have never found a textbook that does so. Some papers that give pieces of the results:

Examples

Online experiments

By which I mean “experiments performed on websites”, not “online” as in “online learning”.

Adaptive experiments