Accounting for observation noise in equation-free forecasting: the hidden-Markov S-map
Published in Methods in Ecology and Evolution on July 1st 2024 Link to Paper
-
Ecosystems contain many species interacting with each other and the environment. Quantitative understanding of these complex systems is limited by which aspects of the ecosystem can be observed. Often relevant potential variables are left out due to them being unknown or difficult to measure. Empirical dynamic modeling and nonlinear forecasting can be used to infer unobserved ecosystem dynamics with minimal assumptions. Enthusiasm is growing for these methods across a wide range of disciplines.
-
However, as traditionally applied in ecology, EDM assumes that the available time series are observed without error. Failing to account for observation noise strongly biases estimates of Lyapunov exponents and reduces forecast accuracy. To address this limitation, we propose incorporating EDM into a hidden Markov framework and using an iterative scheme based on the Expectation Maximization (EM) algorithm to obtain filtered state and parameter estimates.
-
We evaluate the performance of this approach on simulated dynamical systems with a range of additive noise levels, as well as on insect population time series.
-
The results demonstrate improved forecast accuracy robust computation of Lyapunov exponents in the presence of observation noise.
Two examples are provided in this vignette detailing the use of HMS_map on simulated and empirical data.
Note that exact results will not be replicated due to random noise
Example 1 Simulation of Discrete Logistic Map with 10% observation noise Example 1
Example 2 Replication of case study on Global Population Dynamics Database insect Time Series Example 2
The full repository is also made available here.
Github Repository with all code and data used in evaluation of the model
The main file contains the main HMS-map algorithm HMSmap_lags.m as well as it’s dependencies in the lag.m file. This file also contains the False Nearest Neighbors algorithm f_fnn.m Zhou, Y. (2019). used to estimate the Embedding dimension and a script to calculate Lyapunov Exponents from HMS-map regression coefficients lyapunov_QR_lags.m.
The Simulations file contains the code necessary to replicate the results from the paper as well as a simple example Example_Simulation.m. Replication of the results from the paper can be done by running Future_noise_level.m and modifying it to call the simulated system you are interested in (1D,2D,5D). These scripts preform smoothing, leave one out forecasting and future predictions. This file also contains code used for additional tests preformed in the appendix of the paper.
The Empirical data folder contains selected insect data from GPDD (Pendergast et al., 2010) used in (Rogers et al., 2022). The Empirical_Example.m script computes the LE for each of the insect time series and can be compared with the results from (Rogers et al., 2022) in the LE_mean.csv file.
The Figures file contains scripts used to create all figures from simulations in the paper. Figures.m creates LE plots using dropplot.m as well as plots of MSE over various noise levels. MSE_future.m calculates future prediction error from the data generated by the simulation studies. prediction_err.m does the same for multi step leave one out predictions. The data from these two scripts is saved in LOO_3x3.m and Future3x3.m which makes the corresponding figures.
The docs folder contains the code used to launch a Github pages website with instructions for how to apply HMS-map to simulated and empirical data.