Abstract
Surrogate models play a vital role in overcoming the computational challenge in designing and analyzing nonlinear dynamic systems, especially in the presence of uncertainty. This paper presents a comparative study of different surrogate modeling techniques for nonlinear dynamic systems. Four surrogate modeling methods, namely, Gaussian process (GP) regression, a long short-term memory (LSTM) network, a convolutional neural network (CNN) with LSTM (CNN-LSTM), and a CNN with bidirectional LSTM (CNN-BLSTM), are studied and compared. All these model types can predict the future behavior of dynamic systems over long periods based on training data from relatively short periods. The multi-dimensional inputs of surrogate models are organized in a nonlinear autoregressive exogenous model (NARX) scheme to enable recursive prediction over long periods, where current predictions replace inputs from the previous time window. Three numerical examples, including one mathematical example and two nonlinear engineering analysis models, are used to compare the performance of the four surrogate modeling techniques. The results show that the GP-NARX surrogate model tends to have more stable performance than the other three deep learning (DL)-based methods for the three particular examples studied. The tuning effort of GP-NARX is also much lower than its deep learning-based counterparts.
References
1.
Irizarry
, R.
, 2005
, “A Generalized Framework for Solving Dynamic Optimization Problems Using the Artificial Chemical Process Paradigm: Applications to Particulate Processes and Discrete Dynamic Systems
,” Chem. Eng. Sci.
, 60
(21
), pp. 5663
–5681
. 2.
Xu
, B.
, He
, J.
, and Masri
, S. F.
, 2015
, “Data-Based Model-Free Hysteretic Restoring Force and Mass Identification for Dynamic Systems
,” Comput.-Aided Civ. Infrastruct. Eng.
, 30
(1
), pp. 2
–18
. 3.
Pravin
, P.
, Bhartiya
, S.
, and Gudi
, R. D.
, 2019
, “Modeling and Predictive Control of an Integrated Reformer–Membrane–Fuel Cell–Battery Hybrid Dynamic System
,” Ind. Eng. Chem. Res.
, 58
(26
), pp. 11392
–11406
. 4.
Tahmasian
, S.
, 2021
, “Dynamic Analysis and Optimal Control of Drag-Based Vibratory Systems Using Averaging
,” Nonlinear Dyn.
, 104
(3
), pp. 2201
–2217
. 5.
Li
, M.
, and Lai
, A. C. K.
, 2015
, “Review of Analytical Models for Heat Transfer by Vertical Ground Heat Exchangers (GHEs): A Perspective of Time and Space Scales
,” Appl. Energy
, 151
(1
), pp. 178
–191
. 6.
Butt
, W. A.
, Yan
, L.
, and Amezquita S
, K.
, 2013
, “Adaptive Integral Dynamic Surface Control of a Hypersonic Flight Vehicle
,” Int. J. Syst. Sci.
, 46
(10
), pp. 1717
–1728
. 7.
Yoon
, S.
, 2003
, “A Study on Terrain-Surface Modeling and Searching Algorithms for Real-Time Simulation of Off-Road Vehicles
,” Veh. Syst. Dyn.
, 39
(5
), pp. 353
–363
. 8.
Ouyang
, M.
, 2014
, “Review on Modeling and Simulation of Interdependent Critical Infrastructure Systems
,” Reliab. Eng. Syst. Saf.
, 121
(1
), pp. 43
–60
. 9.
Gerdes
, J.
, Bruck
, H. A.
, and Gupta
, S. K.
, 2019
, “A Simulation-Based Approach to Modeling Component Interactions During Design of Flapping Wing Aerial Vehicles
,” Int. J. Micro Air Veh.
, 11
(1
), p. 1756829318822325
. 10.
Tavecchia
, G.
, Miranda
, M. A.
, Borras
, D.
, Bengoa
, M.
, Barcelo
, C.
, Paredes-Esquivel
, C.
, and Schwarz
, C.
, 2017
, “Modelling the Range Expansion of the Tiger Mosquito in a Mediterranean Island Accounting for Imperfect Detection
,” Front. Zool.
, 14
(1
), p. 39
. 11.
Thomas
, N.
, Portyankina
, G.
, Hansen
, C. J.
, and Pommerol
, A.
, 2011
, “HiRISE Observations of Gas Sublimation-Driven Activity in Mars’ Southern Polar Regions: IV. Fluid Dynamics Models of CO2 Jets
,” Icarus
, 212
(1
), pp. 66
–85
. 12.
Rivals
, I.
, and Personnaz
, L.
, 1996
, “Black-Box Modeling With State-Space Neural Networks
,” Neural Adaptive Control Technol.
, 15
(1
), pp. 237
–264
. 13.
Pillonetto
, G.
, and De Nicolao
, G.
, 2010
, “A New Kernel-Based Approach for Linear System Identification
,” Automatica
, 46
(1
), pp. 81
–93
. 14.
Hong
, X.
, Mitchell
, R. J.
, Chen
, S.
, Harris
, C. J.
, Li
, K.
, and Irwin
, G. W.
, 2008
, “Model Selection Approaches for Non-Linear System Identification: A Review
,” Int. J. Syst. Sci.
, 39
(10
), pp. 925
–946
. 15.
Masti
, D.
, and Bemporad
, A.
, 2021
, “Learning Nonlinear State–Space Models Using Autoencoders
,” Automatica
, 129
(1
), p. 109666
. 16.
Deshmukh
, A. P.
, and Allison
, J. T.
, 2017
, “Design of Dynamic Systems Using Surrogate Models of Derivative Functions
,” ASME J. Mech. Des.
, 139
(10
), p. 101402
. 17.
Gedon
, D.
, Wahlström
, N.
, Schön
, T. B.
, and Ljung
, L.
, 2021
, “Deep State Space Models for Nonlinear System Identification
,” IFAC-PapersOnLine
, 54
(7
), pp. 481
–486
. 18.
Schön
, T. B.
, Wills
, A.
, and Ninness
, B.
, 2011
, “System Identification of Nonlinear State-Space Models
,” Automatica
, 47
(1
), pp. 39
–49
. 19.
van Der Voort
, M.
, Dougherty
, M.
, and Watson
, S.
, 1996
, “Combining Kohonen Maps With Arima Time Series Models to Forecast Traffic Flow
,” Transp. Res. C: Emerg. Technol.
, 4
(5
), pp. 307
–318
. 20.
Ing
, C.-K.
, and Wei
, C.-Z.
, 2005
, “Order Selection for Same-Realization Predictions in Autoregressive Processes
,” Ann. Stat.
, 33
(5
), pp. 2423
–2474
. DOI: 10.1214/00905360500000052521.
Li
, D.
, 2012
, “A Note on Moving-Average Models With Feedback
,” J. Time Ser. Anal.
, 33
(6
), pp. 873
–879
. 22.
Chang
, S.
, Wei
, X.
, Su
, F.
, Liu
, C.
, Yi
, G.
, Wang
, J.
, Han
, C.
, and Che
, Y.
, 2020
, “Model Predictive Control for Seizure Suppression Based on Nonlinear Auto-Regressive Moving-Average Volterra Model
,” IEEE Trans. Neural Syst. Rehabil. Eng.
, 28
(10
), pp. 2173
–2183
. 23.
Mustapa
, R. F.
, Dahlan
, N. Y.
, Yassin
, A. I.
, and Nordin
, A. H.
, 2020
, “Quantification of Energy Savings From an Awareness Program Using NARX-ANN in an Educational Building
,” Energy Build.
, 215
(1
), p. 109899
. 24.
Worden
, K.
, Becker
, W. E.
, Rogers
, T. J.
, and Cross
, E. J.
, 2018
, “On the Confidence Bounds of Gaussian Process NARX Models and Their Higher-Order Frequency Response Functions
,” Mech. Syst. Signal Process
, 104
(1
), pp. 188
–223
. 25.
Zhang
, Y.
, Kimberg
, D. Y.
, Coslett
, H. B.
, Schwartz
, M. F.
, and Wang
, Z.
, 2014
, “Multivariate Lesion-Symptom Mapping Using Support Vector Regression
,” Hum. Brain Mapp.
, 35
(12
), pp. 5861
–5876
. 26.
Tao
, J.-W.
, and Wang
, S.-T.
, 2012
, “Kernel Support Vector Machine for Domain Adaptation
,” Zi dong hua xue bao
, 38
(5
), pp. 797
–811
. DOI:10.3724/SP.J.1004.2012.0079727.
Papadopoulos
, S.
, Azar
, E.
, Woon
, W.-L.
, and Kontokosta
, C. E.
, 2017
, “Evaluation of Tree-Based Ensemble Learning Algorithms for Building Energy Performance Estimation
,” J. Build. Perform. Simul.
, 11
(3
), pp. 322
–332
. 28.
Chen
, W.
, Zhang
, S.
, Li
, R.
, and Shahabi
, H.
, 2018
, “Performance Evaluation of the GIS-Based Data Mining Techniques of Best-First Decision Tree, Random Forest, and Naïve Bayes Tree for Landslide Susceptibility Modeling
,” Sci. Total Environ.
, 644
(1
), pp. 1006
–1018
. 29.
Mantas
, C. J.
, Castellano
, J. G.
, Moral-García
, S.
, and Abellán
, J.
, 2019
, “A Comparison of Random Forest Based Algorithms: Random Credal Random Forest Versus Oblique Random Forest
,” Soft Comput.
, 23
(21
), pp. 10739
–10754
. 30.
Loukeris
, N.
, and Eleftheriadis
, I.
, 2015
, “Further Higher Moments in Portfolio Selection and A Priori Detection of Bankruptcy, Under Multi-Layer Perceptron Neural Networks, Hybrid Neuro-Genetic MLPs, and the Voted Perceptron
,” Int. J. Finance Econ.
, 20
(4
), pp. 341
–361
. 31.
Botvinick
, M. M.
, and Plaut
, D. C.
, 2006
, “Short-term Memory for Serial Order: A Recurrent Neural Network Model
,” Psychol. Rev.
, 113
(2
), pp. 201
–233
. 32.
Song
, X.
, Liu
, Y.
, Xue
, L.
, Wang
, J.
, Zhang
, J.
, Wang
, J.
, Jiang
, L.
, and Cheng
, Z.
, 2020
, “Time-Series Well Performance Prediction Based on Long Short-Term Memory (LSTM) Neural Network Model
,” J. Pet. Sci. Eng.
, 186
(1
), p. 106682
. 33.
Chen
, H.
, Zhang
, Y.
, Kalra
, M. K.
, Lin
, F.
, Chen
, Y.
, Liao
, P.
, Zhou
, J.
, and Wang
, G.
, 2017
, “Low-Dose CT With a Residual Encoder-Decoder Convolutional Neural Network
,” IEEE Trans. Med. Imaging
, 36
(12
), pp. 2524
–2535
. 34.
Pak
, U.
, Kim
, C.
, Ryu
, U.
, Sok
, K.
, and Pak
, S.
, 2018
, “A Hybrid Model Based on Convolutional Neural Networks and Long Short-Term Memory for Ozone Concentration Prediction
,” Air Qual. Atmos. Health
, 11
(8
), pp. 883
–895
. 35.
Filiński
, M.
, Wachel
, P.
, and Tiels
, K.
, 2021
, Low-Dimensional Decompositions for Nonlinear Finite Impulse Response Modeling
, Springer International Publishing
, Springer, Cham
, pp. 352
–359
.36.
Shokry
, A.
, Baraldi
, P.
, Zio
, E.
, and Espuña
, A.
, 2020
, “Dynamic Surrogate Modeling for Multistep-Ahead Prediction of Multivariate Nonlinear Chemical Processes
,” Ind. Eng. Chem. Res.
, 59
(35
), pp. 15634
–15655
. 37.
Piga
, D.
, and Tóth
, R.
, 2014
, “A Bias-Corrected Estimator for Nonlinear Systems with Output-Error Type Model Structures
,” Automatica
, 50
(9
), pp. 2373
–2380
. 38.
Rahrooh
, A.
, and Shepard
, S.
, 2009
, “Identification of Nonlinear Systems Using NARMAX Model
,” Nonlinear Anal. Theory Methods Appl.
, 71
(12
), pp. e1198
–e1202
. 39.
ElSaid
, A.
, El Jamiy
, F.
, Higgins
, J.
, Wild
, B.
, and Desell
, T.
, 2018
, “Optimizing Long Short-Term Memory Recurrent Neural Networks Using Ant Colony Optimization to Predict Turbine Engine Vibration
,” Appl. Soft Comput.
, 73
(1
), pp. 969
–991
. 40.
Xue
, P.
, Jiang
, Y.
, Zhou
, Z.
, Chen
, X.
, Fang
, X.
, and Liu
, J.
, 2019
, “Multi-Step Ahead Forecasting of Heat Load in District Heating Systems Using Machine Learning Algorithms
,” Energy
, 188
(1
). DOI: 10.1016/j.energy.2019.11608541.
Zhan
, X.
, Zhang
, S.
, Szeto
, W. Y.
, and Chen
, X.
, 2019
, “Multi-Step-Ahead Traffic Speed Forecasting Using Multi-Output Gradient Boosting Regression Tree
,” Intell. Transp. Syst.
, 24
(2
), pp. 125
–141
. 42.
Wang
, J.
, Jiang
, H.
, Wu
, Y.
, and Dong
, Y.
, 2015
, “Forecasting Solar Radiation Using an Optimized Hybrid Model by Cuckoo Search Algorithm
,” Energy
, 81
(1
), pp. 627
–644
. 43.
Brownlee
, J.
, 2017
, “4 Strategies for Multi-Step Time Series Forecasting
,” https://machinelearningmastery.com/multi-step-time-series-forecasting/, Accessed August 15, 2021.44.
Yu
, H.
, Chung
, C. Y.
, Wong
, K. P.
, Lee
, H. W.
, and Zhang
, J. H.
, 2009
, “Probabilistic Load Flow Evaluation With Hybrid Latin Hypercube Sampling and Cholesky Decomposition
,” IEEE Trans. Power Syst.
, 24
(2
), pp. 661
–667
. 45.
Rasmussen
, C. E.
, and Williams
, C. K.
, 2006
, Gaussian Processes for Machine Learning
, MIT Press
, Cambridge, MA
.46.
Johnson
, M. E.
, Moore
, L. M.
, and Ylvisaker
, D.
, 1990
, “Minimax and Maximin Distance Designs
,” J. Stat. Plan. Inference
, 26
(2
), pp. 131
–148
. 47.
Hu
, Z.
, Hu
, C.
, Mourelatos
, Z. P.
, and Mahadevan
, S.
, 2019
, “Model Discrepancy Quantification in Simulation-Based Design of Dynamical Systems
,” ASME J. Mech. Des.
, 141
(1
), p. 011401
. 48.
Lee
, H. I.
, Choi
, I. Y.
, Moon
, H. S.
, and Kim
, J. K.
, 2020
, “A Multi-Period Product Recommender System in Online Food Market Based on Recurrent Neural Networks
,” Sustainability (Basel, Switzerland)
, 12
(3
), p. 969
. DOI:10.3390/su1203096949.
Zhou
, L.
, 2020
, “Product Advertising Recommendation in e-Commerce Based on Deep Learning and Distributed Expression
,” Electron. Commer. Res.
, 20
(2
), pp. 321
–342
. 50.
Jiang
, X.
, Sun
, J.
, Li
, C.
, and Ding
, H.
, 2018
, “Video Image Defogging Recognition Based on Recurrent Neural Network
,” IEEE Trans. Industr. Inform.
, 14
(7
), pp. 3281
–3288
. 51.
Li
, S.
, Xie
, G.
, Ren
, J.
, Guo
, L.
, Yang
, Y.
, and Xu
, X.
, 2020
, “Urban PM 2.5 Concentration Prediction via Attention-Based CNN–LSTM
,” Appl. Sci.
, 10
(6
), p. 1953
. 52.
Tian
, Y.
, Zhang
, K.
, Li
, J.
, Lin
, X.
, and Yang
, B.
, 2018
, “LSTM-Based Traffic Flow Prediction With Missing Data
,” Neurocomputing
, 318
(1
), pp. 297
–305
. 53.
Li
, Y.
, Su
, H.
, Qi
, C. R.
, Fish
, N.
, Cohen-Or
, D.
, and Guibas
, L. J.
, 2015
, “Joint Embeddings of Shapes and Images via CNN Image Purification
,” ACM Trans. Graph.
, 34
(6
), pp. 1
–12
. DOI:10.1145/2816795.281807154.
Sarin
, H.
, Kokkolaras
, M.
, Hulbert
, G.
, Papalambros
, P.
, Barbat
, S.
, and Yang
, R.-J.
, 2008
, “A Comprehensive Metric for Comparing Time Histories in Validation of Simulation Models With Emphasis on Vehicle Safety Applications
,” Proc. International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Brooklyn, NY
, Aug. 3–6
, pp. 1275
–1286
.55.
Ao
, D.
, Hu
, Z.
, and Mahadevan
, S.
, 2017
, “Dynamics Model Validation Using Time-Domain Metrics
,” J. Verif. Valid. Uncertain. Quantif.
, 2
(1
), p. 011004
. 56.
Sarin
, H.
, Kokkolaras
, M.
, Hulbert
, G.
, Papalambros
, P.
, Barbat
, S.
, and Yang
, R.-J.
, 2010
, “Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics
,” J. Dyn. Syst. Meas. Control
, 132
(6
), p. 061401
. 57.
Wiggins
, S.
, 1987
, “Chaos in the Quasiperiodically Forced Duffing Oscillator
,” Phys. Lett. A
, 124
(3
), pp. 138
–142
. 58.
Pei
, J.-S.
, Gay-Balmaz
, F.
, Luscher
, D. J.
, Beck
, J. L.
, Todd
, M. D.
, Wright
, J. P.
, Qiao
, Y.
, Quadrelli
, M. B.
, Farrar
, C. R.
, and Lieven
, N. A.
, 2021
, “Connecting Mem-Models With Classical Theories
,” Nonlinear Dyn.
, 103
(2
), pp. 1321
–1344
. 59.
Bouc
, R.
, 1967
, “Forced Vibrations of Mechanical Systems With Hysteresis
,” Proceedings of the Fourth Conference on Nonlinear Oscillations
, Prague
, Sept. 5–9
, p. 315
.60.
Bouc
, R.
, 1971
, “A Mathematical Model for Hysteresis
,” Acta Acust. United Acust.
, 24
(1
), pp. 16
–25
.61.
Wen
, Y.-K.
, 1976
, “Method for Random Vibration of Hysteretic Systems
,” J. Eng. Mech.
, 102
(2
), pp. 249
–263
. 62.
Lei
, Y.
, Wu
, Y.
, and Li
, T.
, 2012
, “Identification of Non-Linear Structural Parameters Under Limited Input and Output Measurements
,” Int. J. Non Linear Mech.
, 47
(10
), pp. 1141
–1146
. Copyright © 2022 by ASME
You do not currently have access to this content.
