Exploratory Data Analysis

Exploratory data analysis (EDA) looks at data to see what they seem to say. The distribution of the observed data is examined without imposing an arbitrary probability model on it. We look for trends, such as patterns and linear or nonlinear relationships between variables, and deviations from the trends, such as local anomalies, outliers, or clusters. This facilitates discovering the unexpected as well as confirming suspicions, rather like detective work.

EDA is sometimes viewed as a grab bag of tools, but this is a misconception. It is more accurate to view EDA as a procedure for data analysis. We start from a set of expectations or specific questions arising from the data context and explore the data with these in mind, while remaining open to observing unexpected patterns. The approach involves making many plots and numerical models of the data. Plots allow us to examine the distribution of the data without an imposed probability model; thus, statistical graphics form the backbone of EDA. Plots provide simple, digestible summaries of complex information that enable discovering unexpected structure. With the assistance of techniques such as bootstrapping, permutation, and model selection methods, we can assess whether the observed patterns in the data are more than random noise.

EDA is different from confirmatory statistical analysis. In confirmatory analysis, we start from a hypothesis and work to confirm or reject the hypothesis. EDA is a hypothesis discovery process. EDA provides approximate answers to any question of interest, instead of an exact answer to the wrong question. In the process of exploration, the data may suggest hypotheses, leading to follow-up confirmatory analysis with new data.

Methods in common usage that have arisen from EDA include the boxplot, stem-and-leaf plot, median polish, and projection pursuit.