Nonlinear Processes in Geophysics www.nonlin-processes-geophys.net 1023-5809 1607-7946 16 1 2009 10.5194/npg-16-57-2009 http://www.nonlin-processes-geophys.net/16/57/2009/ http://www.nonlin-processes-geophys.net/16/57/2009/npg-16-57-2009.html http://www.nonlin-processes-geophys.net/16/57/2009/npg-16-57-2009.pdf 57 64 2009-02-06 Information theoretic measures of dependence, compactness, and non-gaussianity for multivariate probability distributions A. H. Monahan monahana@uvic.ca T. DelSole School of Earth and Ocean Sciences, University of Victoria, Victoria, BC, Canada and Canadian Institute for Advanced Research Earth System Evolution Program, Canada Department of Atmospheric, Oceanic, and Earth Sciences, George Mason University, Fairfax, Virginia, and Center for Ocean-Land-Atmosphere Studies, Calverton, Maryland, USA A basic task of exploratory data analysis is the characterisation of "structure" in multivariate datasets. For bivariate Gaussian distributions, natural measures of dependence (the predictive relationship between individual variables) and compactness (the degree of concentration of the probability density function (pdf) around a low-dimensional axis) are respectively provided by ordinary least-squares regression and Principal Component Analysis. This study considers general measures of structure for non-Gaussian distributions and demonstrates that these can be defined in terms of the information theoretic "distance" (as measured by relative entropy) between the given pdf and an appropriate "unstructured" pdf. The measure of dependence, mutual information, is well-known; it is shown that this is not a useful measure of compactness because it is not invariant under an orthogonal rotation of the variables. An appropriate rotationally invariant compactness measure is defined and shown to reduce to the equivalent PCA measure for bivariate Gaussian distributions. This compactness measure is shown to be naturally related to a standard information theoretic measure of non-Gaussianity. Finally, straightforward geometric interpretations of each of these measures in terms of "effective volume" of the pdf are presented. Cover, T M. and Thomas, J A.: Elements of Information Theory, John Wiley & Sons, Inc., New York, 1991. DelSole, T.: Predictability and Information Theory. Part I: Measures of predictability, J. Atmos. Sci., 61, 2425–2440, 2004. DelSole, T. and Tippett, M K.: Predictability: Recent Insights from Information Theory, Rev. Geophys., 45, RG4002, doi:10.1029/2006RG000202, 2007. Haven, K., Majda, A., and Abramov, R.: Quantifying predictability through information theory: Small sample estimation in a non-Gaussian framework, J. Comp. Phys., 206, 334–362, 2005. Kleeman, R. and Majda, A.: Predictability in a model of geophysical turbulence, J. Atmos. Sci., 62, 2864–2879, 2005. Lee, T.-W., Girolami, M., Bell, A., and Sejnowski, T.: A unifying information-theoretic framework for independent component analysis, Comp. Math. App., 39, 1–21, 2000. Majda, A., Kleeman, R., and Cai, D.: A Mathematical Framework for Quantifying Predictability Through Relative Entropy, Meth. Appl. Anal., 9, 425–444, 2002. Majda, A J., Abramov, R V., and Grote, M J.: Information Theory and Stochastics for Multiscale Nonlinear Systems, American Mathematical Society, Providence, RI, USA, 2005. Monahan, A H., Fyfe, J C., and Pandolfo, L.: The vertical structure of wintertime climate regimes of the Northern Hemisphere extratropical atmosphere, J. Climate, 16, 2005–2021, 2003. Pardo, L.: Statistical Inference Based on Divergence Measures, Chapman and Hall, 2006. Peña, D. and van~der Linde, A.: Dimensionless measures of variability and dependence for multivariate continuous distributions, Comm. Stat. Theory Meth., 36, 1845–1854 doi:10.1080/03610920601126449, 2007. Schneidman, E., Still, S., Berry, M J., and Bialek, W.: Network information and connected correlations, Phys. Rev. Lett., 91, 238701-1–238701-4, 2003.