Bayesian experimental design

Bayesian experimental design provides a general probability-theoretical framework from which other theories on experimental design can be derived. It is based on Bayesian inference to interpret the observations/data acquired during the experiment. This allows accounting for both any prior knowledge on the parameters to be determined as well as uncertainties in observations.

The theory of Bayesian experimental design is to a certain extent based on the theory for making optimal decisions under uncertainty. The aim when designing an experiment is to maximize the expected utility of the experiment outcome. The utility is most commonly defined in terms of a measure of the accuracy of the information provided by the experiment (e.g. the Shannon information or the negative variance), but may also involve factors such as the financial cost of performing the experiment. What will be the optimal experiment design depends on the particular utility criterion chosen.

Relations to more specialized optimal design theory

Linear theory

If the model is linear, the prior probability density function (PDF) is homogeneous and observational errors are normally distributed, the theory simplifies to the classical optimal experimental design theory.

Approximate normality

In numerous publications on Bayesian experimental design, it is (often implicitly) assumed that all posterior PDFs will be approximately normal. This allows for the expected utility to be calculated using linear theory, averaging over the space of model parameters, an approach reviewed in Chaloner & Verdinelli (1995). Caution must however be taken when applying this method, since approximate normality of all possible posteriors is difficult to verify, even in cases of normal observational errors and uniform prior PDF.

Posterior distribution

Recently, increased computational resources allow inference of the posterior distribution of model parameters, which can directly be used for experiment design. Vanlier et al. (2012) proposed an approach that uses the posterior predictive distribution to assess the effect of new measurements on prediction uncertainty, while Liepe et al. (2013) suggest maximizing the mutual information between parameters, predictions and potential new experiments.

Mathematical formulation

**Notation**
$\theta \,$	parameters to be determined
$y\,$	observation or data
$\xi \,$	design
$p(y\|\theta ,\xi )\,$	PDF for making observation $y$ , given parameter values $\theta$ and design $\xi$
$p(\theta )\,$	prior PDF
$p(y\|\xi )\,$	marginal PDF in observation space
$p(\theta \|y,\xi )\,$	posterior PDF
$U(\xi )\,$	utility of the design $\xi$
$U(y,\xi )\,$	utility of the experiment outcome after observation $y$ with design $\xi$

Given a vector $\theta$ of parameters to determine, a prior PDF $p(\theta )$ over those parameters and a PDF $p(y|\theta ,\xi )$ for making observation $y$ , given parameter values $\theta$ and an experiment design $\xi$ , the posterior PDF can be calculated using Bayes' theorem

p(\theta |y,\xi )={\frac {p(y|\theta ,\xi )p(\theta )}{p(y|\xi )}}\,,

where $p(y|\xi )$ is the marginal probability density in observation space

p(y|\xi )=\int {p(\theta )p(y|\theta ,\xi )d\theta }\,.

The expected utility of an experiment with design $\xi$ can then be defined

U(\xi )=\int {p(y|\xi )U(y,\xi )dy}\,,

where $U(y,\xi )$ is some real-valued functional of the posterior PDF $p(\theta |y,\xi )$ after making observation $y$ using an experiment design $\xi$ .

Gain in Shannon information as utility

Utility may be defined as the prior-posterior gain in Shannon information

U(y,\xi )=\int {\log(p(\theta |y,\xi ))p(\theta |y,\xi )d\theta }-\int {\log(p(\theta ))p(\theta )d\theta }\,.

Note also that

U(y,\xi )=D_{{KL}}(p(\theta |y,\xi )\|p(\theta |\xi ))\,,

the Kullback–Leibler divergence of the prior from the posterior distribution. Lindley (1956) noted that the expected utility will then be coordinate-independent and can be written in two forms

{\begin{alignedat}{2}U(\xi )&=\int {\int {\log(p(\theta |y,\xi ))p(\theta ,y|\xi )d\theta }dy}-\int {\log(p(\theta ))p(\theta )d\theta }\\&=\int {\int {\log(p(y|\theta ,\xi ))p(\theta ,y|\xi )dy}d\theta }-\int {\log(p(y|\xi ))p(y|\xi )dy},\end{alignedat}}\,

of which the latter can be evaluated without the need for evaluating individual posterior PDFs $p(\theta |y,\xi )$ for all possible observations $y$ . Worth noting is that the first term on the second equation line will not depend on the design $\xi$ , as long as the observational uncertainty doesn't. On the other hand, the integral of $p(\theta )\log p(\theta )$ in the first form is constant for all $\xi$ , so if the goal is to choose the design with the highest utility, the term need not be computed at all. Several authors have considered numerical techniques for evaluating and optimizing this criterion, e.g. van den Berg, Curtis & Trampert (2003) and Ryan (2003). Note that

U(\xi )=I(\theta ;y)\,,

the expected information gain being exactly the mutual information between the parameter θ and the observation y. Kelly (1956) also derived just such a utility function for a gambler seeking to profit maximally from side information in a horse race; Kelly's situation is identical to the foregoing, with the side information, or "private wire" taking the place of the experiment.

References

Vanlier; Tiemann; Hilbers; van Riel (2012), "A Bayesian approach to targeted experiment design" (PDF), Bioinformatics, 28 (8): 1136–1142, doi:10.1093/bioinformatics/bts092

Liepe; Filippi; Komorowski; Stumpf (2013), "Maximizing the Information Content of Experiments in Systems Biology", PLOS Computational Biology, 9 (1): e1002888, doi:10.1371/journal.pcbi.1002888

van den Berg; Curtis; Trampert (2003), "Optimal nonlinear Bayesian experimental design: an application to amplitude versus offset experiments" (PDF), Geophysical Journal International, 155 (2): 411–421, doi:10.1046/j.1365-246x.2003.02048.x, archived from the original (PDF) on 2011-07-17

Chaloner, Kathryn; Verdinelli, Isabella (1995), "Bayesian experimental design: a review" (PDF), Statistical Science, 10 (3): 273–304, doi:10.1214/ss/1177009939

DasGupta, A. (1996), "Review of optimal Bayes designs", in Ghosh, S.; Rao, C. R., Design and Analysis of Experiments (PDF), Handbook of Statistics, 13, North-Holland, pp. 1099–1148, ISBN 0-444-82061-2

Lindley, D. V. (1956), "On a measure of information provided by an experiment", Annals of Mathematical Statistics, 27 (4): 986–1005, doi:10.1214/aoms/1177728069

Ryan, K. J. (2003), "Estimating Expected Information Gains for Experimental Designs With Application to the Random Fatigue-Limit Model", Journal of Computational and Graphical Statistics, 12 (3): 585–603, doi:10.1198/1061860032012

Design of experiments

Scientific method	Scientific experiment Statistical design Control Internal and external validity Experimental unit Blinding Optimal design: Bayesian Random assignment Randomization Restricted randomization Replication versus subsampling Sample size

Treatment and blocking	Treatment Effect size Contrast Interaction Confounding Orthogonality Blocking Covariate Nuisance variable

Models and inference	Linear regression Ordinary least squares Bayesian Random effect Mixed model Hierarchical model: Bayesian Analysis of variance (Anova) Cochran's theorem Manova (multivariate) Ancova (covariance) Compare means Multiple comparison

Designs Completely randomized	Factorial Fractional factorial Plackett-Burman Taguchi Response surface methodology Polynomial and rational modeling Box-Behnken Central composite Block Generalized randomized block design (GRBD) Latin square Graeco-Latin square Orthogonal array Latin hypercube Repeated measures design Crossover study Randomized controlled trial Sequential analysis Sequential probability ratio test

Glossary Category Statistics portal Statistical outline Statistical topics

Statistics

Descriptive statistics

Continuous data

Center	Mean arithmetic geometric harmonic Median Mode

Dispersion	Variance Standard deviation Coefficient of variation Percentile Range Interquartile range

Shape	Moments Skewness Kurtosis L-moments

Count data

Index of dispersion

Summary tables

Dependence

Graphics

Data collection

Study design	Population Statistic Effect size Statistical power Sample size determination Missing data

Survey methodology	Sampling Standard error stratified cluster Opinion poll Questionnaire

Controlled experiments	Design control optimal Controlled trial Randomized Random assignment Replication Blocking Interaction Factorial experiment

Uncontrolled studies	Observational study Natural experiment Quasi-experiment

Statistical inference

Statistical theory

Frequentist inference

Point estimation	Estimating equations Maximum likelihood Method of moments M-estimator Minimum distance Unbiased estimators Mean-unbiased minimum-variance Rao–Blackwellization Lehmann–Scheffé theorem Median unbiased Plug-in

Interval estimation	Confidence interval Pivot Likelihood interval Prediction interval Tolerance interval Resampling Bootstrap Jackknife

Testing hypotheses	1- & 2-tails Power Uniformly most powerful test Permutation test Randomization test Multiple comparisons

Parametric tests	Likelihood-ratio Wald Score

Specific tests

Z (normal) Student's t-test F

Goodness of fit	Chi-squared Kolmogorov–Smirnov Anderson–Darling Normality (Shapiro–Wilk) Likelihood-ratio test Model selection Cross validation AIC BIC

Rank statistics	Sign Sample median Signed rank (Wilcoxon) Hodges–Lehmann estimator Rank sum (Mann–Whitney) Nonparametric anova 1-way (Kruskal–Wallis) 2-way (Friedman) Ordered alternative (Jonckheere–Terpstra)

Bayesian inference

Correlation	Pearson product–moment Partial correlation Confounding variable Coefficient of determination

Regression analysis	Errors and residuals Regression model validation Mixed effects models Simultaneous equations models Multivariate adaptive regression splines (MARS)

Linear regression	Simple linear regression Ordinary least squares General linear model Bayesian regression

Non-standard predictors	Nonlinear regression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity

Generalized linear model	Exponential families Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance	Analysis of variance (ANOVA, anova) Analysis of covariance Multivariate ANOVA Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

Categorical

Multivariate

Time-series

General	Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration Structural break Granger causality

Specific tests	Dickey–Fuller Johansen Q-statistic (Ljung–Box) Durbin–Watson Breusch–Godfrey

Time domain	Autocorrelation (ACF) partial (PACF) Cross-correlation (XCF) ARMA model ARIMA model (Box–Jenkins) Autoregressive conditional heteroskedasticity (ARCH) Vector autoregression (VAR)

Frequency domain	Spectral density estimation Fourier analysis Wavelet

Survival

Survival function	Kaplan–Meier estimator (product limit) Proportional hazards models Accelerated failure time (AFT) model First hitting time

Hazard function	Nelson–Aalen estimator

Test	Log-rank test

Applications

Biostatistics	Bioinformatics Clinical trials / studies Epidemiology Medical statistics

Engineering statistics	Chemometrics Methods engineering Probabilistic design Process / quality control Reliability System identification

Social statistics	Actuarial science Census Crime statistics Demography Econometrics National accounts Official statistics Population statistics Psychometrics

Spatial statistics	Cartography Environmental statistics Geographic information system Geostatistics Kriging

Category
Portal
Commons
WikiProject

This article is issued from Wikipedia - version of the 11/26/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.