Deep Learning PINN

PINN Project.

Members: Nathan Nguyen, Lukas Jarasunas, Nathan Haronian, Mason Lee

Check-in 1: Title: Hamiltonian Physics Informed Neural Networks Who: mlee168, nharonia, ljarasun, nnguye92 Introduction: What problem are you trying to solve and why? Physics Informed Neural Networks (PINNs) have arisen as a novel method for solving a variety of ODEs, PDEs, and other equations, however there still exist many gaps in understanding especially concerning non-linear dynamics and chaotic systems. Dynamical systems are often modelled through Hamiltonian mechanics, which leads to powerful geometric techinques. Purely data-driven methods fail to capture the advantage that such a prior embedding presents (conservation of energy between momentum and position). We are proposing to recreate a paper (not sure yet), that uses hamiltonian PINNs to model the dynamics of chaotic systems. Related Work: Are you aware of any, or is there any prior work that you drew on to do your project? Please read and briefly summarize (no more than one paragraph) at least one paper/article/blog relevant to your topic beyond the paper you are re-implementing/novel idea you are researching. There is already a relatively large body of literature exploring uses of deep learning to solve high dimensional partial differential equations. For example, this article details multiple methods to address a variety of scenarios; both continuous and discrete time cases are covered, as well as high dimensional linear and semilinear PDEs. In each case, they combine theoretical knowledge of the equations to the development of their neural network architectures as well as the development of appropriate training data. Other related papers include this paper on optimal loss functions for PINNs or this paper on using PINNs to predict order and chaos in chaotic systems. In this section, also include URLs to any public implementations you find of the paper you’re trying to implement. Please keep this as a “living list”--if you stumble across a new implementation later down the line, add it to this list. Data: What data are you using (if any)? If you’re using a standard dataset (e.g. MNIST), you can just mention that briefly. Otherwise, say something more about where your data come from (especially if there’s anything interesting about how you will gather it). How big is it? Will you need to do significant preprocessing? PINNs are not data intensive generally and do not need preprocessing because the data is effectively created during the training process. Methodology: What is the architecture of your model? How are you training the model? Model is trained using the constraints of the PDE, in essence we have as much data as sampling points, as the loss function is determined by the PDE and the output of the model. If you are implementing an existing paper, detail what you think will be the hardest part about implementing the model here. If we do not find the code for the paper, the greatest challenge will be in understanding the custom algorithms they use. Some papers give pseudocode to the reader toward their method, but coding it can be difficult. Often there are certain tricks that are used but not eludicated If you are doing something new, justify your design. Also note some backup ideas you may have to experiment with if you run into issues. If we have the time, after we recreate the results of an experiment we attempt to iterate on the results, adding another layer of complexity → this may be run over longer time periods, different boundary conditions, adding noise to the data? Metrics: What constitutes “success?” What experiments do you plan to run? Prediction of lypanouv spectrum Fraction of chaotic orbits over a transition period For most of our assignments, we have looked at the accuracy of the model. Does the notion of “accuracy” apply for your project, or is some other metric more appropriate? PINNs often use a tri-part MSE: PDE residual + time BC + space BC. Conservation of energy MSE between coordinates of baseline If you are implementing an existing project, detail what the authors of that paper were hoping to find and how they quantified the results of their model. If you are doing something new, explain how you will assess your model’s performance. What are your base, target, and stretch goals? Ethics: Choose 2 of the following bullet points to discuss; not all questions will be relevant to all projects so try to pick questions where there’s interesting engagement with your project. (Remember that there’s not necessarily an ethical/unethical binary; rather, we want to encourage you to think critically about your problem setup.) What broader societal issues are relevant to your chosen problem space? Why is Deep Learning a good approach to this problem? Finding an analytic solution to certain PDEs is incredibly difficult, if not completely impossible. Neural networks provide an alternative approach to analyzing solutions without actually needing to analytically solve the equation. In the same way that it would be almost impossible to create a program that hard codes MNIST identification, it can be impossible to compute solutions to differential equations like the Navier Stokes equations. Using deep learning lets us get what we want from these differential equations without having to do the impossible. What is your dataset? Are there any concerns about how it was collected, or labeled? Is it representative? What kind of underlying historical or societal biases might it contain? Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm? As of now, the scientific community is the main “stakeholder,” as PINNs would allow them to study the behavior of certain dynamical systems that they might not have been able to study before. Mistakes in the algorithm could produce some incorrect results and further studies, which could compound in further research. If PINNs were ever used in predictive models of real world phenomena and decisions were made based on the results, then this could potentially cause real world harm. For instance, if a PINN was used to predict environmental phenomena and incorrect data was produced, anyone making decisions based on these predictions might under prepare or misallocate resources to address the situation. How are you planning to quantify or measure error or success? What implications does your quantification have? Add your own: if there is an issue about your algorithm you would like to discuss or explain further, feel free to do so.

Check-in #2: Introduction: Physics Informed Neural Networks (PINNs) have arisen as a novel method for solving a variety of ODEs, PDEs, and other equations, however there still exist many gaps in understanding especially concerning non-linear dynamics and chaotic systems. Dynamical systems are often modeled through Hamiltonian mechanics, which leads to powerful geometric techniques. Purely data-driven methods fail to capture the advantage that such a prior embedding presents (conservation of energy between momentum and position). We are proposing to recreate a paper (not sure yet), that uses hamiltonian PINNs to model the dynamics of chaotic systems. Challenges: What has been the hardest part of the project you’ve encountered so far? We had to slightly change our final project because tensorflow didn’t allow us to see the gradient of the batch but we needed to see the gradient of the gradient for the loss function (hamiltonian). Moreover, we couldn’t find a notebook/sufficient material on our first idea. Additionally, conceptualizing the material was difficult. We had to find and read some research papers to understand what was going on. This took us a while since the material was very dense and took a while for us to process/understand. Insights: Are there any concrete results you can show at this point? Yes, shown in our notebook. Plan: Are you on track with your project? We’re getting on track. We need to dedicate more time towards working on the code and getting it to work. But so far, we understand what we have to do and we have a good understanding of the concepts. Once we get the code working, we will be in very good shape.

Final Check-in: PINN for the Win: A Deep Learning Approach to Solving Differential Equations Lukas Jarasunas, Nathan Haronian, Nathan Nguyen, Mason Lee

A physics informed neural network (PINN) is a type of neural network that is trained to solve partial differential equations (PDEs) by incorporating prior knowledge of the physical laws governing the system being modeled. This can provide several advantages over traditional numerical schemes for solving PDEs.

First, PINNs can often provide more accurate solutions to PDEs than traditional numerical schemes. This is because traditional numerical schemes discretize the PDE and solve the resulting system of equations using iterative methods, which can introduce errors and lead to approximate solutions. In contrast, PINNs are trained using data and can learn to accurately capture the underlying physics of the system, leading to more accurate solutions.

Second, PINNs can often be faster and more efficient than traditional numerical schemes. This is because traditional numerical schemes require the user to specify a spatial discretization, which can be time-consuming and computationally expensive. In contrast, PINNs can automatically learn a suitable spatial discretization, which can greatly reduce the computational cost and time required to solve the PDE.

Third, PINNs are often more flexible and adaptable than traditional numerical schemes. This is because traditional numerical schemes are typically designed for specific types of PDEs, and cannot easily be applied to other types of equations or systems. In contrast, PINNs can be trained to solve a wide range of PDEs, and can be easily adapted to new problems by providing new training data.

Overall, the use of PINNs can provide significant advantages over traditional numerical schemes for solving PDEs, including improved accuracy, efficiency, and flexibility.

Function of PINNs:

The architecture of a vanilla PINN typically consists of two parts: a feedforward neural network, and a PDE loss function. The feedforward neural network maps input variables (such as space and time) to output variables (such as the solution of the PDE at a given point in space and time). The PDE loss function is used to enforce the constraints imposed by the underlying physics of the system, and ensures that the output of the neural network satisfies the PDE being solved.

The loss function of a physics informed neural network (PINN) is used to enforce the constraints imposed by the underlying physics of the system being modeled. This is typically done by computing the residual of the PDE at a set of points in the domain, and using this residual to compute a loss function. The loss function measures the deviation of the neural network's predictions from the true solutions of the PDE, and is used to fine-tune the weights of the neural network. The PINN loss function can then be defined as the mean squared error between the residual and zero, which enforces the constraint that the residual must be zero at all points in the domain. During training, the PINN is optimized to minimize this loss function, which ensures that the neural network's predictions satisfy the PDE at all points in the domain. This allows the PINN to learn the underlying structure of the PDE and provide an accurate and efficient solution.

To train a vanilla PINN, one first selects a set of training data, which consists of pairs of input and output variables. The input variables are typically chosen to span the domain of the PDE, and the output variables are the corresponding solutions of the PDE at these points. The feedforward neural network is then trained to fit this training data, using a standard optimization algorithm such as stochastic gradient descent.

Once the feedforward neural network is trained, the PDE loss function is used to evaluate the quality of the neural network's predictions. This typically involves computing the residual of the PDE at a set of points in the domain, and using this residual to compute a loss function. The loss function measures the deviation of the neural network's predictions from the true solutions of the PDE, and is used to fine-tune the weights of the neural network.

Finally, the trained PINN is used to make predictions for new input values, providing an approximate solution to the PDE. This solution can be used to simulate the behavior of the system being modeled, or to make predictions about its future behavior.

Solving ODEs with a Vanilla PINN:

A vanilla physics informed neural network (PINN) is typically designed to solve partial differential equations (PDEs), which are equations that involve partial derivatives with respect to multiple variables (such as space and time). In contrast, an ordinary differential equation (ODE) is a type of differential equation that involves only a single variable (such as time) and its derivatives.

To solve an ODE using a PINN, the architecture and loss function of the vanilla PINN must be modified to account for the fact that the equation involves only a single variable. This typically involves using a feedforward neural network with a simpler structure, and defining a loss function that is appropriate for the ODE being solved.

For example, consider a PINN that is being trained to solve the following simple harmonic oscillation ODE:

where y is the dependent variable, t is the independent variable, and k is a constant. In this case, the feedforward neural network could be a simple one-dimensional function, such as:

where W1and W2 are the weights of the neural network. The ODE loss function can then be defined as the mean squared error between the derivative of the neural network's output and the right-hand side of the ODE, which enforces the constraint that the ODE must be satisfied at all points in the domain. This loss function can be written as:

where N is the number of points in the domain, and y, dy/dt, and d2ydt are the predicted values of the dependent variable and its first and second derivatives at these points. During training, the PINN is optimized to minimize this loss function, which ensures that the neural network's predictions satisfy the ODE at all points in the domain. This allows the PINN to learn the underlying structure of the ODE and provide an accurate and efficient solution.

Results:

The code for this project is available in the "working_code" notebook within the following GitHub repository: https://github.com/nharonian/DL_Final_Project. The key findings of this study are depicted in the accompanying graphs.

The results of this study indicate that the use of PINNs in Tensorflow was successful in solving the harmonic differential equation. In contrast, the standard neural network approach did not produce accurate results.

When using the PINN method, the accuracy of the model increased with the number of training steps, suggesting that the model was able to learn and improve over time. This indicates that Pinns are a promising approach for solving differential equations using neural networks.

In contrast, the standard neural network approach was not able to accurately model the harmonic differential equation, as shown in the images. This suggests that the use of PINNs may be a superior method for solving differential equations using neural networks.

Overall, these results suggest that the use of Pinns in Tensorflow is an effective and promising approach for solving differential equations using neural networks. Further research is needed to fully explore the capabilities and limitations of this method.

Limitations and Future Work:

PINN still struggles with complex problems with shock generation. In our case of shock, if collocation points fall within the shock region this continuity will produce large loss, causing steep gradients. The neural network would want to decrease the region of the shock through gradient descent but cannot because shock waves have zero thickness theoretically, this causes the points within a shock region to fall into a paradoxical status. Increasing the gradient increases total loss because the points are no longer governed by the PDE, decreasing the gradient also increases total loss because it conflicts with this physical compression. Higher order finite-volume methods like WENO address this problem by reducing the scheme to a lower order, such a weakening of network expression in strong compression regions is being researched in regard to PINNs. It will be interesting to explore and quantify how much dissipation is being added into the model and where. It seems the PINNs are further becoming integrated into the tricks of numerical methods and time will tell what the effect of PINNs added flexibility will be!

Sources:

https://greydanus.github.io/2019/05/15/hamiltonian-nns/ https://benmoseley.blog/my-research/so-what-is-a-physics-informed-neural-network/ https://github.com/jnafzig/HamiltonianSolver https://sullivantm.com/blog/2021/07/physics_informed_neural_networks/ https://www.sciencedirect.com/science/article/pii/S0021999118307125 https://phys.org/news/2020-06-physics-neural-networks-chaos.html https://github.com/greydanus/hamiltonian-nn https://arxiv.org/abs/2201.05624