AI Research Portal

This paper introduces SiGMoID (Simulation-based Generative Model for Imperfect Data), a framework designed for robust inference of dynamic systems from noisy, sparse, or partially observed data. The core idea is to integrate two deep-learning models: HyperPINN for solving ordinary differential equations (ODEs) corresponding to predetermined parameters, and Wasserstein Generative Adversarial Networks (W-GAN) for adjusting priors and matching solutions to imperfect data. SiGMoID is empirically claimed to quantify data noise, estimate system parameters, and infer unobserved system components with superior accuracy and robustness compared to conventional methods, demonstrating broad applicability across various scientific and engineering domains.

The paper addresses significant challenges in system inference for nonlinear dynamic models represented by ODEs, particularly when data is imperfect. These challenges are categorized into two distinct types:

Noisy and Sparse (NS) Data: Observations of system components are noisy and recorded at sparse time points. While existing methods like penalized likelihood and Gaussian processes (GPs) have been developed for NS data, they often require significant manual tuning or can be computationally expensive.

Improved parameter estimation on NS Data: SiGMoID achieves superior parameter estimation accuracy compared to existing methods on noisy and sparse data.
Enhanced prediction of full dynamics on NSMC Data: SiGMoID accurately infers unobserved components on noisy, sparse, and missing component data, overcoming limitations of current methods that fail to capture these dynamics completely.
Advancing system inference using deep learning-based ODE solvers: The deep learning-based ODE solver (HyperPINN) utilized in SiGMoID exhibits potential scalability beyond NS and NSMC datasets, making it applicable to a wide range of datasets.

The SiGMoID framework is a Simulation-based Generative Model for Imperfect Data that captures system parameters and noise levels. It integrates two deep-learning models: HyperPINN and W-GAN.

HyperPINN Training:

Parameter Sampling: A random parameter value pˢ is sampled from a uniform distribution U[pmin,Pmax] within an empirically determined feasible range.
Solution Computation: A numerical solver computes the solution y(t) = y(t; pˢ) of the ODE system for the sampled parameter pˢ.
Neural Networks: The HyperPINN framework consists of two neural networks: a hypernetwork h and a main network m.
- The hypernetwork h takes the sampled parameter pˢ as input and generates weights and biases θm(pˢ) for the main network m.
- The main network mˢ(t) = m(t; θm(pˢ)) produces the solution y(t) for any arbitrary time t ∈ [0, T].
Joint Training: Networks h and m are trained jointly by minimizing a weighted sum of data loss Ldata and physics loss Lphysics.
- Loss: Ldata = ∑ⱼ=1ᵀᶜᵒˡ ||mˢ(tⱼ) - y(tⱼ; pˢ)||² Lphysics = ∑ⱼ=1ᵀᶜᵒˡ ||d/dt mˢ(tⱼ) - f[mˢ(tⱼ), pˢ]||² L = αLdata + βLphysics
- Here, {tⱼ} are given time collocation points, and α, β > 0 are distinct weights.

W-GAN Training and Inference:

Latent Variable Sampling: Latent variables zᵖ and zᵉ are sampled from a standard normal distribution, N(0, 1).
Generators: These latent variables serve as inputs to two generators, Gᵖ and Gᵉ, to yield a candidate system parameter p̂ and data noise ê for {y₁ⁱ}ᵢ=1,...,dᵧ, respectively.
Solution Generation: Using the trained HyperPINN, the solution ŷᵢ = m̂ᵢ + êᵢ corresponding to y₁ᵢ is obtained.
Sample Generation: This procedure is repeated N times, sampling random parameters and noise values, to obtain N samples {(p̂ₙ, ŷ₁ₙ)}ₙ=1,...,N from the generator.
Discrepancy Measurement: The discrepancy between the observed data {y₁ⁱ}ᵢ=1,...,dᵧ and generated outputs {ŷ₁ₙ}ₙ=1,...,N is measured using the Wasserstein distance: d({Y₁ⁱ}ᵢ=1,...,dᵧ, {Ŷ₁ₙ}ₙ=1,...,N) = sup_||φ||_Lip1 E_y⁰[φ({y₁ⁱ}ᵢ=1,...,dᵧ)] - E_ŷ[φ({ŷ₁ₙ}ₙ=1,...,N)] where {y₁ⁱ}ᵢ=1,...,dᵧ ~ y⁰ and {ŷ₁ₙ}ₙ=1,...,N ~ ŷ denote distributions of observed and generated outputs.
W-GAN Training: The W-GAN is trained by minimizing this objective.
Inference: The parameter values {p̂ₙ}ₙ=1,...,N generated by Gᵖ are expected to converge within a narrow region containing the true system parameter pᵗʳᵘᵉ. Using this parameter distribution, a numerical solver is reapplied to generate missing component (MC) data.

Architecture:
- HyperPINN consists of a hypernetwork h and a main network m.
- W-GAN uses two generators, Gᵖ and Gᵉ.
- Detailed descriptions of HyperPINN and GAN, including stability measures, hyperparameters, and architecture configurations, are presented in Table A1 to Table A2 (not provided in the excerpt).
Key Equations:
- ODE system: dy(t)/dt = f(y(t), p), t ∈ [0,T] (1)
- Observed data model: yº(t) = y(t) + e(t) (2)
- HyperPINN data loss: Ldata = ∑ⱼ=1ᵀᶜᵒˡ ||mˢ(tⱼ) - y(tⱼ; pˢ)||²
- HyperPINN physics loss: Lphysics = ∑ⱼ=1ᵀᶜᵒˡ ||d/dt mˢ(tⱼ) - f[mˢ(tⱼ), pˢ]||²
- Total HyperPINN loss: L = αLdata + βLphysics
- Wasserstein distance for W-GAN: d({Y₁ⁱ}ᵢ=1,...,dᵧ, {Ŷ₁ₙ}ₙ=1,...,N) = sup_||φ||_Lip1 E_y⁰[φ({y₁ⁱ}ᵢ=1,...,dᵧ)] - E_ŷ[φ({ŷ₁ₙ}ₙ=1,...,N)]
Optimization: Not stated in the paper.
Augmentation: Not stated in the paper.
Data Handling:
- Noisy and Sparse (NS) Data: All components are observable but observations are noisy and recorded at sparse time points.
- NS with Missing Components (NSMC) Data: A subset of system components is unobservable in addition to being noisy and sparse.
- For FitzHugh-Nagumo, 100 observed trajectories are generated with a noise level of 0.2 using 41 observation points.
- For Protein transduction, 100 observed trajectories are generated with a noise level of 0.01 using 26 observation time points.
- For Hesl model, simulation noise is modeled as multiplicative using a lognormal distribution with σ = 0.15. A log-transformation is applied to produce Gaussian error distributions.
- For Lorenz system, additive noise with a noise level of 0.1 is added, and 9 observation points are used for X.
Implementation Details: Not stated in the paper.

Datasets:
- FitzHugh-Nagumo (FN) model
- Protein transduction model
- Hesl model
- Lorenz system
Metrics:
- Trajectory RMSE (Root Mean Square Error) RMSE = √(1/N_s ∑ⱼ=1ᴺˢ ||Y_true,i(t_k) - Ŷ_i(t_k)||²)
Baselines:
- MAGI (Manifold-constrained Gaussian process inference)
- FGPGM (Fast GP-based gradient matching)
- AGM (Adaptive gradient matching)
- (RAMSAY ET AL. 2007) (Penalized likelihood approach)
Numeric Results:
- FitzHugh-Nagumo (FN) System - Trajectory RMSEs (Table 1):
  - NS Data:
    - SIGMOID: V = 0.028, R = 0.012
    - MAGI: V = 0.103, R = 0.070
    - FGPGM: V = 0.257, R = 0.094
    - AGM: V = 1.177, R = 0.662
  - NSMC Data:
    - SIGMOID: V = 0.038, R = 0.016
- Protein Transduction System - Trajectory RMSEs (scaled by ×10³) (Table 2):
  - NS Data:
    - SIGMOID: S = 0.3, Sd = 0.8, R = 1.2, SR = 0.4, Rpp = 1.3
    - MAGI: S = 12.2, Sd = 4.3, R = 16.7, SR = 13.5, Rpp = 13.6
    - FGPGM: S = 12.8, Sd = 8.9, R = 21.0, SR = 13.6, Rpp = 30.9
    - AGM: S = 67.1, Sd = 312.5, R = 413.8, SR = 98.0, Rpp = 297.3
  - NSMC Data:
    - SIGMOID: S = 11.2, Sd = 11.2, R = 4.9, SR = 3.8, Rpp = 2.9
- Hesl System - Trajectory RMSEs (Table 3):
  - SIGMOID: P = 0.38, M = 0.12, H = 1.96
  - MAGI: P = 0.97, M = 0.21, H = 2.57
  - (RAMSAY ET AL. 2007): P = 1.30, M = 0.40, H = 59.47
Runtime/Training Cost: The computational costs for all cases are summarized in Table A7 in the Appendix (not provided in the excerpt).

Prior Knowledge Requirement for HyperPINN: Training HyperPINNs requires a priori knowledge of the distribution of parameters within the dataset.
Scalability with Parameter Space: The number of artificial trajectories required for HyperPINN training grows proportionally with the size of the parameter space, leading to increased computational cost and training time.

Method evidence: "In this paper, we propose a Simulation-based Generative Model for Imperfect Data (SiGMOID), that enables precise and robust inference for dynamic systems. The proposed approach integrates two key methods: (1) HyperPINN, and (2) W-GAN." (section: Abstract)
Result evidence: "It is observed to exhibit the lowest root mean square error (RMSE), √(1/N_s ∑ⱼ=1ᴺˢ ||Y_true,i(t_k) - Ŷ_i(t_k)||²), between the true Y_true,i(t) = Y_i(t; p_true) and estimated trajectories Ŷ_i(t) = Y_i(t; p̂) for each component i compared to the other three methods—MAGI (Yang, Wong, and Kou 2021), FGPGM (Wenk et al. 2019), and AGM (Dondelinger et al. 2013) (Table 1, NS)." (section: Results)
Limitation evidence: "Specifically, training HyperPINNs requires a priori knowledge of the distribution of parameters within the dataset (de Avila Belbute-Peres, Chen, and Sha 2021). The number of artificial trajectories required for training grows proportionally with the size of the parameter space, leading to increased computational cost and training time." (section: Discussion)

Learning from Imperfect Data: Robust Inference of Dynamic Systems Using Simulation-Based Generative Model

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures