Nonlinear Processes in Geophysics www.nonlin-processes-geophys.net 1023-5809 1607-7946 13 2 2006 10.5194/npg-13-151-2006 http://www.nonlin-processes-geophys.net/13/151/2006/ http://www.nonlin-processes-geophys.net/13/151/2006/npg-13-151-2006.html http://www.nonlin-processes-geophys.net/13/151/2006/npg-13-151-2006.pdf 151 159 2006-05-24 Spatio-temporal filling of missing points in geophysical data sets D. Kondrashov dkondras@atmos.ucla.edu M. Ghil Department of Atmospheric and Oceanic Sciences and Institute of Geophysics and Planetary Physics, University of California, Los Angeles, USA Department of Geosciences and Laboratoire de Météorologie Dynamique (CNRS and IPSL), Ecole Normale Supérieure, Paris, France The majority of data sets in the geosciences are obtained from observations and measurements of natural systems, rather than in the laboratory. These data sets are often full of gaps, due to to the conditions under which the measurements are made. Missing data give rise to various problems, for example in spectral estimation or in specifying boundary conditions for numerical models. Here we use Singular Spectrum Analysis (SSA) to fill the gaps in several types of data sets. For a univariate record, our procedure uses only temporal correlations in the data to fill in the missing points. For a multivariate record, multi-channel SSA (M-SSA) takes advantage of both spatial and temporal correlations. We iteratively produce estimates of missing data points, which are then used to compute a self-consistent lag-covariance matrix; cross-validation allows us to optimize the window width and number of dominant SSA or M-SSA modes to fill the gaps. The optimal parameters of our procedure depend on the distribution in time (and space) of the missing data, as well as on the variance distribution between oscillatory modes and noise. The algorithm is demonstrated on synthetic examples, as well as on data sets from oceanography, hydrology, atmospheric sciences, and space physics: global sea-surface temperature, flood-water records of the Nile River, the Southern Oscillation Index (SOI), and satellite observations of relativistic electrons. Alvera-Azcárate, A., Barth, A., Rixen, M., and Beckers, J M.: Reconstruction of incomplete oceanographic data sets using empirical orthogonal functions: applications to the Adriatic Sea surface temperature, Ocean Modelling, 9, 325-346, 2005. %\bibitem[] Allen, M. R., and L. A. Smith (1996), Monte Carlo SSA: Detecting irregular oscillations in the presence of coloured noise. \textitJ. Clim., \textit9, 3373-3404. Beckers, J. and Rixen, M.: EOF calculations and data filling from incomplete oceanographic data sets, J. Atmos. Ocean. Technol., 20, 1839-1856, 2003. Broomhead, D. S. and King, G P.: Extracting qualitative dynamics from experimental data, Physica D, 20, 217-236, 1986. Colebrook, J M.: Continuous plankton records: zooplankton and environment, North-East Atlantic and North Sea, 1948-1975, Oceanol. Acta, 1, 9-23, 1978. \bibitem [Dettinger (1995)]dett95 Dettinger, M D., Ghil, M., Strong, C M., Weibel, W., and Yiou, P.: Software expedites singular-spectrum analysis of noisy time series, Eos, Trans. American Geophysical Union, v 76(2), p 12, 14, 21, 1995. %\bibitem[] Deser, C., and M. L. Blackmon (1993), Surface climate variations %over the North Atlantic ocean during winter 1900-1989, \textitJ. %Clim., \textit6, 1743-1753. Foster, G.: Wavelets for period analysis of unevenly sampled time series, Astronom. J., 112, 1709-1729, 1996. Fraedrich, K.: Estimating the dimensions of weather and climate attractors, J. Atmos. Sci., 43, 419-432, 1986. %\bibitem [] Fraedrich, K., and Ch. Bantzer (1991), A note on fluctuations of the Nile River flood levels \it Theor. Appl. Climatol., \it 44, 167-171. %\bibitem[] Fraedrich, K., J. Jiang, F. W. Gerstengarbe, and P. C. Werner %(1997), Multiscale detection if abrupt climate changes: Application %to River Nile flood levels, \textitInt. J. Climatol., \textit17, 1301-1315. Ghil, M. and Vautard, R.: Interdecadal oscillations and the warming trend in global temperature time series, Nature, 350, 324-327, 1991. Ghil, M. and Jiang, N.: Recent forecast skill for the El Ni\~no/Southern Oscillation, Geophys. Res. Lett., 25, 171-174, 1998. Ghil, M., Allen, R. M., Dettinger, M. D., Ide, K., Kondrashov, D., et al.: Advanced spectral methods for climatic time series, Rev. Geophys. 40(1), 3.1-3.41, doi:10.1029/2000RG000092, 2002. Johns, C., Nychka, D., Kittel, T., and Daly, C.: Infilling sparse records of spatial fields, J. Amer. Stat. Assoc., 98(464), 796-806, 2003. Kaplan, A., Kushnir, Y., Cane, M., and Blumenthal, M.: Reduced space optimal analysis for historic data sets: 136 years of Atlantic sea-surface temperatures, J. Geophys. Res., 102, 27 835-27 860, 1997. Kondrashov, D., Feliks, Y., and Ghil, M.: Oscillatory modes of extended Nile River records (A.D. 622-1922), Geophys. Res. Lett., 32, L10702, doi:10.1029/2004GL022156, 2005a. Kondrashov, D., Kravtsov, S., and Ghil, M.: A hierarchy of data-based ENSO models. J. Climate, 18, 4425-4444, 2005b. MacDonald, G J.: Spectral analysis of time series generated by nonlinear processes, Rev. Geophys., 27, 449-469, 1989. Mann, M E., Bradley, R S., and Hughes M K.: Global-scale temperature patterns and climate forcing over the past centuries, Nature, 392, 779-787, 1998. Mendelssohn, R., Schwing, F B., and Bograd S J.: Spatial structure of subsurface temperature variability in the California Current, 1950-1993, J. Geophys. Res., 108(C3), 3093, doi:10.1029/2002JC001568, 2003. %\bibitem[] Mann, M., J. Park, and R. S. Bradley (1995), Global interdecadal %and century-scale climate oscillations during the past five centuries, \textitNature, \textit378, %266-270. %\bibitem[] Mann, M., and J. M. Lees (1996), Robust estimation of background %noise and signal detection in climatic time series. \textitClim. %Change, \textit33\textbf, 409-445. %\bibitem[] Mann, M., R. S. Bradley, and M. K. Hughes (1998), Global-scale %temperature patterns and climate forcing over the past six centuries, \textitNature, \textit392, %779-787. %\bibitem[] Moron, V., R. Vautard, and M. Ghil (1998), Trends, interdecadal %and interannual oscillations in global sea-surface temperatures, \textitClim. %Dyn., \textit14, 545-569. %\bibitem[] Nicholson, S. E., and D. Entekhabi (1986), The quasi-periodic behavior of rainfall variability in Africa and its relationship to the Southern Oscillation. \textitJ. Clim. Appl. Meteor., \textit34, 331-348. Popper, W.: The Cairo Nilometer, 269 pp., University of California Press, Berkeley/Los Angeles, 1951. % %Preisendorfer, R W.: Principal Component Analysis in Meteorology and %Oceanography. Elsevier, New York, 425 pp, 1998. Reynolds, R W. and Smith, T M.: Improved global sea-surface temperature analysis using optimum interpolation, J. Climate, 7, 929-948, 1994. %\bibitem[] Ropelewski, C. F., and M. S. Halpert, (1987), Global and regional scale precipitation associated with El Ni\~no Southern Oscillation. \textitMon. Wea. Rev. \textit115, 985-996. Ropelewski, C F. and P. D. Jones: An extension of the Tahiti-Darwin Southern Oscillation Index, Mon. Wea. Rev., 115, 2161-2165, 1987. Schneider, T.: Analysis of incomplete climate data: Estimation of mean values and covariance matrices and imputation of missing values, J. Climate, 14, 853-871, 2001. Schoellhamer, D.: Singular spectrum analysis for time series with missing data, Geophys. Res. Lett., 28(16), 3187-3190, 2001. %\bibitem[] Schwing, F B, and R. Mendelssohn (1997), Increased coastal upwelling in the California Current System, \it J. of Geophys. Res., \it 102, 3,421-3,438. Schulz, M. and Mudelsee, M.: REDFIT: estimating red-noise spectra directly from unevenly spaced paleclimatic time series, Computers and Geosciences, 28, 421-426, 2002. Sherwood, S.: Climatic signals from station arrays with missing data, and an application to winds, J. Geophys. Res., 105, 29 489-29 500, 2000. % %Shprits, Y Y., Thorne, R M., Reeves, G D., and Friedel R.: Radial diffusion modeling with empirical lifetimes: Comparison with CRRES observations, Annales Geophys, 23, 1467-1471, 2005. %\bibitem[]Slepian, S. (1978), Prolate spheroidal wave functions, Fourier analysis and uncertainty-V: The discrete case, \textitBell. Syst. Tech. J., 1371-1430. Smith, T M., Reynolds R W., Livezey R E., and Stokes D C.: Reconstruction of historical sea-surface temperatures using empirical orthogonal functions, J. Climate, 9, 1403-1420, 1996. %\bibitem[] Thomson, J. D. (1982), Spectrum estimation and harmonic analysis. \textitProc. %IEEE, \textit70, 1055-1096. Toussoun, O.: Mémoire sur l'histoire du Nil, Mémoires de l'Institut d'Egypte, 18, pp 366-404, Cairo, 1925. Varadi, F., Pap, J. M., Ulrich, R. K., Bertello, L., and Henney, C. J.: Searching for signal in noise by random-lag singular spectrum analysis, Astrophys. J., 526, 1052-1061, 1999. Vautard, R. and Ghil, M.: Singular spectrum analysis in nonlinear dynamics, with applications to paleoclimatic time series, Physica D, 35, 395-424, 1989. Woodruff, S D., Slutz, R J., Jenne, R L, and Steurer, P M.: A comprehensive ocean-atmosphere data set, Bull. Amer. Meteor. Soc., 68, 1239-1250, 1987. %\bibitem[] Vautard, R., P. Yiou,\nobreakspace and M. Ghil (1992), Singular-spectrum %analysis: A toolkit for short, noisy chaotic signals, \textitPhysica %D, \textit58, 95-126. %\bibitem[] Yiou, P., D. Sornette, and M. Ghil (2000), Data-adaptive wavelets %and multi-scale SSA, \textitPhysica D, \textit142, 254-290. % % % Columbus, C.: How I Discovered America. Hispanic Press, Barcelona, % 1492. % % % Green, R J., Fred, U P., and Norbert, W P.: Things that go bump in the % night, Psych. Today, 46, 345-678, 1900. % % % James, K., Harris, Jr., G., and Wollops, W.: American independence and % magnetism, Revol. Tracts, 32, 34-55, 1776. % % % Phillips, T P.: Possible influence of the magnetosphere on American history, % stylesamewww.sma-london.ac.uk/~phillips/magnet_infl.htm#section42, 1999. % % % Smith, J G. and Weston, H K.: Nothing particular in this year's history, % J. Oddball Res., 2, 14-15, 1954. %