Discovering Differential Equations with Physics-Informed Neural Networks and Symbolic Regression
Differential Equations serve as a powerful framework to capture and understand the dynamic behaviors of physical systems. By describing how variables change in relation to each other, they provide insights into system dynamics and allow us to make predictions about the system's future behavior.
However, a common challenge we face in many real-world systems is that their governing differential equations are often only partially known, with __ the unknown aspects manifesting in several ways:
- The parameters of the differential equation are unknown. A case in point is wind engineering, where the governing equations of fluid dynamics are well-established, but the coefficients relating to turbulent flow are highly uncertain.
- The functional forms of the differential equations are unknown. For instance, in chemical engineering, the exact functional form of the rate equations may not be fully understood due to the uncertainties in rate-determining steps and reaction pathways.
- Both functional forms and parameters are unknown. A prime example is battery state modeling, where the commonly used equivalent circuit model only partially captures the current-voltage relationship (the functional form of the missing physics is therefore unknown). Moreover, the model itself contains unknown parameters (i.e., resistance and capacitance values).
Such partial knowledge of the governing differential equations hinders our understanding and control of these dynamical systems. Consequently, inferring these unknown components based on observed data becomes a crucial task in dynamical system modeling.
Broadly speaking, this process of using observational data to recover governing equations of dynamical systems falls in the domain of system identification. Once discovered, we can readily use these equations to predict future states of the system, inform control strategies for the systems, or enable theoretical investigations using analytical techniques.
Very recently, Zhang et al.(2023) proposed a promising strategy that leverages physics-informed neural networks (PINN) and symbolic regression to discover unknowns in a system of ordinary differential equations (ODEs). While their focus was on discovering differential equations for Alzheimer's disease modeling, their proposed solution holds promise for general dynamical systems.
In this blog post, we will take a closer look at the concepts put forth by the authors and get hands-on to reproduce one of the case studies investigated in the paper. Toward that end, we will build a PINN from scratch, leverage the PySR library to perform symbolic regression, and discuss the obtained results.
If you are interested in learning best practices in physics-informed neural networks, feel free to check out my blog series here:
Physics-Informed Neural Networks: An Application-Centric Guide
Unraveling the Design Pattern of Physics-Informed Neural Networks.
With that in mind, let's get started!
Table of Content
· 1. Case Study · 2. Why do traditional approaches fall short? · 3. PINN for System Identification (Theory) · 4. PINN for System Identification (Code) ∘ 4.1 Define the Architecture ∘ 4.2 Define ODE loss ∘ 4.3 Define gradient descent step ∘ 4.4 Data preparation ∘ 4.5 PINN Training · 5. Symbolic Regression ∘ 5.1 PySR library ∘ 5.2 Implementation ∘ 5.3 Identification results · 6. Take-away · Reference
1. Case Study
Let's start by introducing the problem we aim to solve. In this blog, we will reproduce the first case study investigated in Zhang et al.'s original paper, i.e., discovering the Kraichnan-Orszag system from data. The system is described by the following ODEs:
with an initial condition of _u_₁(0)=1, _u_₂(0)=0.8, _u_₃(0)=0.5. The Kraichnan-Orszag system is commonly used in turbulence studies and fluid dynamics research, where the goal is to develop theoretical insights into turbulence, its structures, and its dynamics.
To mimic a typical system identification setup, we assume we only know partially about the governing ODEs. Specifically, we assume that we don't know anything about the differential equations for _u_₁ and _u_₂. In addition, we assume we only know that the right-hand side of the differential equation for _u_₃ is a linear transformation of _u_₁ and _u_₂. Then, we can rewrite the ODE system as follows:
where _f_₁ and _f_₂ denote the unknown functions, and a and b are the unknown parameters. Our objective is to calibrate the values of a and b, as well as estimate the analytical functional form of _f_₁ and _f_₂. Essentially, we are dealing with a challenging system identification problem where both unknown parameters and function forms exist.
2. Why do traditional approaches fall short?
In the traditional paradigm of system identification, we typically employ numerical methods (e.g., Euler's method, Runge-Kutta methods, etc.) to simulate and predict system states _u_₁, _u_₂, and _u_₃. However, those methods are fundamentally limited in that they generally require a complete form of governing differential equations, and are incapable of handling scenarios when the differential equations are only partially known.
In cases where the parameters of the equations are unknown, traditional methods often resort to optimization techniques, where an initial guess for the parameters is made, and then refined in an iterative process to minimize the difference between the observed data and the data predicted by the numerical solver. Since each optimization iteration necessitates one run of the numerical solver, this approach, while feasible, can be computationally very expensive.
Note that the above discussion only describes the case of calibrating the unknown parameters. The problem becomes even more complex when we need to estimate unknown functions in differential equations. Theoretically, we can adopt a similar methodology, i.e., making assumptions about the forms of the unknown functions before optimization. However, issues would immediately rise if we go down this path: If we assume an overly simple form, we run into the risk of underfitting, which may lead to substantial prediction errors. On the other hand, if we assume an overly complex form (e.g., with many tunable parameters), we run into the risk of overfitting, which may lead to poor generalization performance.
In summary, the traditional approach faces significant challenges when dealing with partially known differential equations:
1️⃣ Traditional numerical methods rely on having a complete form of governing differential equations to run simulations.
2️⃣ Combining traditional numerical methods with optimization algorithms can address parameter estimation problems, but often at a high computational cost.
3️⃣ For estimating unknown functions embedded in differential equations, traditional approaches may yield results that are highly sensitive to the assumed functional form, which creates risks of underfitting or overfitting.
Given these challenges, traditional approaches often fall short in addressing system identification problems where unknown parameters and functional forms coexist. This naturally leads us to the topic of physics-informed neural networks (PINNs). In the next section, we will see how PINN can effectively address the challenges faced by traditional approaches.
3. PINN for System Identification (Theory)
The physics-informed neural network (or PINN in short) is a powerful concept proposed by Raissi et al. back in 2019. The basic idea of PINN, like other physics-informed Machine Learning techniques, is to create a hybrid model where both the observational data and the known physical knowledge (represented as differential equations) are leveraged in model training. PINN was originally designed as an efficient ODE/PDE solver. However, researchers soon recognized that PINNs have (arguably) even greater potential in tackling inverse, system identification problems.
In the following, we will explain how PINNs can be leveraged to overcome the challenges we discussed in the previous section, one by one.
1️⃣ Traditional numerical methods rely on having a complete form of governing differential equations to run simulations.
