0
Research Papers: Fuel Combustion

Development of a Stiffness-Based Chemistry Load Balancing Scheme, and Optimization of Input/Output and Communication, to Enable Massively Parallel High-Fidelity Internal Combustion Engine Simulations

[+] Author and Article Information
Janardhan Kodavasal

Argonne National Laboratory,
9700 S. Cass Avenue,
Argonne, IL 60439
e-mail: jkodavasal@anl.gov

Kevin Harms

Argonne Leadership Computing Facility,
9700 S. Cass Avenue,
Argonne, IL 60439
e-mail: harms@alcf.anl.gov

Priyesh Srivastava

Convergent Science, Inc.,
6400 Enterprise Lane,
Madison, WI 53719
e-mail: priyesh.srivastava@convergecfd.com

Sibendu Som

Argonne National Laboratory,
9700 S. Cass Avenue,
Argonne, IL 60439
e-mail: ssom@anl.gov

Shaoping Quan

Convergent Science, Inc.,
6400 Enterprise Lane,
Madison, WI 53719
e-mail: shaoping.quan@convergecfd.com

Keith Richards

Convergent Science, Inc.,
6400 Enterprise Lane,
Madison, WI 53719
e-mail: krichards@convergecfd.com

Marta García

Argonne Leadership Computing Facility,
9700 S. Cass Avenue,
Argonne, IL 60439
e-mail: mgarcia@alcf.anl.gov

1Corresponding author.

Contributed by the Internal Combustion Engine Division of ASME for publication in the JOURNAL OF ENERGY RESOURCES TECHNOLOGY. Manuscript received January 12, 2016; final manuscript received January 12, 2016; published online February 23, 2016. Editor: Hameed Metghalchi.The United States Government retains, and by accepting the article for publication, the publisher acknowledges that the United States Government retains, a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for United States government purposes.

J. Energy Resour. Technol 138(5), 052203 (Feb 23, 2016) (11 pages) Paper No: JERT-16-1022; doi: 10.1115/1.4032623 History: Received January 12, 2016; Revised January 12, 2016

A closed-cycle gasoline compression ignition (GCI) engine simulation near top dead center (TDC) was used to profile the performance of a parallel commercial engine computational fluid dynamics (CFD) code, as it was scaled on up to 4096 cores of an IBM Blue Gene/Q (BG/Q) supercomputer. The test case has 9 × 106 cells near TDC, with a fixed mesh size of 0.15 mm, and was run on configurations ranging from 128 to 4096 cores. Profiling was done for a small duration of 0.11 crank angle degrees near TDC during ignition. Optimization of input/output (I/O) performance resulted in a significant speedup in reading restart files, and in an over 100-times speedup in writing restart files and files for postprocessing. Improvements to communication resulted in a 1400-times speedup in the mesh load balancing operation during initialization, on 4096 cores. An improved, “stiffness-based” algorithm for load balancing chemical kinetics calculations was developed, which results in an over three-times faster runtime near ignition on 4096 cores relative to the original load balancing scheme. With this improvement to load balancing, the code achieves over 78% scaling efficiency on 2048 cores, and over 65% scaling efficiency on 4096 cores, relative to 256 cores.

FIGURES IN THIS ARTICLE
<>
Copyright © 2016 by ASME
Your Session has timed out. Please sign back in to continue.

References

Borgnakke, C. , Arpaci, V. , and Tabaczynski, R. , 1980, “ A Model for the Instantaneous Heat Transfer and Turbulence in a Spark Ignition Engine,” SAE Technical Paper No. 800287.
Hountalas, D. , Kouremenos, D. , Pariotis, E. , Schwarz, V. , and Binder, K. B. , 2002, “ Using a Phenomenological Multi-Zone Model to Investigate the Effect of Injection Rate Shaping on Performance and Pollutants of a DI Heavy Duty Diesel Engine,” SAE Technical Paper No. 2002-01-0074.
Kodavasal, J. , McNenly, M. J. , Babajimopoulos, A. , Aceves, S. M. , Assanis, D. N. , Havstad, M. A. , and Flowers, D. L. , 2013, “ An Accelerated Multi-Zone Model for Engine Cycle Simulation of Homogeneous Charge Compression Ignition Combustion,” Int. J. Engine Res., 14(5), pp. 416–433. [CrossRef]
Som, S. , Longman, D. , Aithal, S. , Bair, R. , Garc′ıa, M. , Quan, S. , Richards, K. J. , Senecal, P. K. , Shethaji, T. , and Weber, M. , 2013, “ A Numerical Investigation on Scalability and Grid Convergence of Internal Combustion Engine Simulations,” SAE Technical Paper No. 2013-01-1095.
Pei, Y. , Kundu, P. , Goldin, G. M. , and Som, S. , 2015, “ Large Eddy Simulation of a Reacting Spray Flame Under Diesel Engine Conditions,” SAE Technical Paper No. 2015-01-1844.
McNenly, M. J. , Whitesides, R. A. , and Flowers, D. L. , 2015, “ Faster Solvers for Large Kinetic Mechanisms Using Adaptive Preconditioners,” Proc. Combust. Inst., 35(1), pp. 581–587. [CrossRef]
Flowers, D. L. , Aceves, S. M. , and Babajimopoulos, A. , 2006, “ Effect of Charge Non-Uniformity on Heat Release and Emissions in PCCI Engine Combustion,” SAE Technical Paper No. 2006-01-1363.
Middleton, R. J. , 2014, “ Simulation of Spark Assisted Compression Ignition Combustion Under EGR Dilute Engine Operating Conditions,” Ph.D. thesis, The University of Michigan, Ann Arbor, MI.
Shi, Y. , Kokjohn, S. L. , Ge, H. , and Reitz, R. D. , 2009, “ Efficient Multidimensional Simulation of HCCI and DI Engine Combustion With Detailed Chemistry,” SAE Technical Paper No. 2009-01-0701.
Amsden, A. A. , 1997, “ KIVA-3V: A Block Structured KIVA Program for Engines with Vertical or Canted Valves,” Los Alamos National Laboratory, Los Alamos, NM, Report No. LA-13313-MS.
Richards, K. J. , Senecal, P. K. , and Pomraning, E. , 2014, “ Converge (v2.2.0),” Theory Manual, Convergent Science, Madison, WI.
Pomraning, E. , and Rutland, C. J. , 2002, “ Dynamic One-Equation Nonviscosity Large-Eddy Simulation Model,” AIAA J., 40(4), pp. 689–701. [CrossRef]
Reitz, R. , and Diwakar, R. , 1987, “ Structure of High Pressure Fuel Sprays,” SAE Technical Paper No. 870598.
Senecal, P. , Richards, K. , Pomraning, E. , Yang, T. , Dai, M. Z. , McDavid, R. M. , Patterson, M. A. , Hou, S. , and Shethaji, T. , 2007, “ A New Parallel Cut-Cell Cartesian CFD Code for Rapid Grid Generation Applied to In-Cylinder Diesel Engine Simulations,” SAE Technical Paper No. 2007-01-0159.
Liu, A. , Mather, D. , and Reitz, R. , 1993, “ Modeling the Effects of Drop Drag and Breakup on Fuel Sprays,” SAE Technical Paper No. 930072.
Amsden, A. A. , O'Rourke, P. J. , and Butler, T. D. , 1989, “ KIVA-II: A Computer Program for Chemically Reactive Flows With Sprays,” Los Alamos National Laboratory, Los Alamos, NM, Laboratory Report No. LA-11560-MS.
Ra, Y. , and Reitz, R. D. , 2009, “ A Vaporization Model for Discrete Multi-Component Fuel Sprays,” Int. J. Multiphase Flow, 35(2), pp. 101–117. [CrossRef]
Senecal, P. K. , Pomraning, E. , Richards, K. J. , Briggs, T. E. , Choi, C. Y. , McDavid, R. M. , and Patterson, M. A. , 2003, “ Multi-Dimensional Modeling of Direct-Injection Diesel Spray Liquid Length and Flame Lift-Off Length Using CFD and Parallel Detailed Chemistry,” SAE Technical Paper No. 2003-01-1043.
Kalghatgi, G. T. , Risberg, P. , and Ångström, H.-E. , 2006, “ Advantages of Fuels With High Resistance to Auto-Ignition in Late-Injection, Low-Temperature, Compression Ignition Combustion,” SAE Technical Paper No. 2006-01-3385.
Manente, V. , Johansson, B. , and Tunestal, P. , 2009, “ Partially Premixed Combustion at High Load Using Gasoline and Ethanol, a Comparison With Diesel,” SAE Technical Paper No. 2009-01-0944.
Sellnau, M. , Sinnamon, J. , Hoyer, K. , and Husted, H. , 2011, “ Gasoline Direct Injection Compression Ignition (GDCI)—Diesel-Like Efficiency With Low CO2 Emissions,” SAE Technical Paper, Paper No. 2011-01-1386.
Adhikary, B. D. , Ra, Y. , Reitz, R. , and Ciatti, S. , 2012, “ Numerical Optimization of a Light-Duty Compression Ignition Engine Fuelled With Low-Octane Gasoline,” SAE Technical Paper No. 2012-01-1336.
Ciatti, S. , Johnson, M. , Adhikary, B. D. , Reitz, R. , and Knock, A. , 2013, “ Efficiency and Emissions Performance of Multizone Stratified Compression Ignition Using Different Octane Fuels,” SAE Technical Paper No. 2013-01-0263.
Kolodziej, C. , Ciatti, S. , Vuilleumier, D. , Adhikary, B. D. , and Reitz, R. D. , 2014, “ Extension of the Lower Load Limit of Gasoline Compression Ignition With 87 AKI Gasoline by Injection Timing and Pressure,” SAE Technical Paper No. 2014-01-1302.
Kolodziej, C. P. , Kodavasal, J. , Ciatti, S. , Som, S. , Shidore, N. , and Delhom, J. , 2015, “ Achieving Stable Engine Operation of Gasoline Compression Ignition Using 87 AKI Gasoline Down to Idle,” SAE Technical Paper No. 2015-01-0832.
Kodavasal, J. , Kolodziej, C. , Ciatti, S. , and Som, S. , 2015, “ Computational Fluid Dynamics Simulation of Gasoline Compression Ignition,” ASME J. Energy Resour. Technol., 137(3), p. 032212. [CrossRef]
Kodavasal, J. , Lavoie, G. A. , Assanis, D. N. , and Martz, J. B. , 2015, “ The Effects of Thermal and Compositional Stratification on the Ignition and Duration of Homogeneous Charge Compression Ignition Combustion,” Combust. Flame, 162(2), pp. 451–461. [CrossRef]
Kodavasal, J. , Lavoie, G. A. , Assanis, D. N. , and Martz, J. B. , 2015, “ The Effect of Diluent Composition on Homogeneous Charge Compression Ignition Auto-Ignition and Combustion Duration,” Proc. Combust. Inst., 35(3), pp. 3019–3026. [CrossRef]
Kodavasal, J. , Lavoie, G. A. , Assanis, D. N. , and Martz, J. B. , 2016, “ Reaction-Space Analysis of Homogeneous Charge Compression Ignition Combustion With Varying Levels of Fuel Stratification Under Positive and Negative Valve Overlap Conditions,” Int. J. Engine Res., (published online).
Adhikary, B. D. , Reitz, R. , and Ciatti, S. , 2013, “ Study of In-Cylinder Combustion and Multi-Cylinder Light Duty Compression Ignition Engine Performance Using Different RON Fuels at Light Load Conditions,” SAE Technical Paper No. 2013-01-0900.
Catania, A. E. , Ferrari, A. , Manno, M. , and Spessa, E. , 2008, “ Experimental Investigation of Dynamics Effects on Multiple-Injection Common Rail System Performance,” ASME J. Eng. Gas Turbines Power, 130(3), p. 032806. [CrossRef]
Liu, Y.-D. , Jia, M. , Xie, M.-Z. , and Pang, B. , 2012, “ Enhancement on a Skeletal Kinetic Model for Primary Reference Fuel Oxidation by Using a Semidecoupling Methodology,” Energy Fuels, 26(12), pp. 7069–7083. [CrossRef]
Carns, P. , Latham, R. , Ross, R. , Iskra, K. , Lang, S. , and Riley, K. , 2009, “ 24/7 Characterization of Petascale I/O Workloads,” IEEE International Conference on Cluster Computing and Workshops (CLUSTER '09), New Orleans, LA, Aug. 31–Sept. 4.
Carns, P. , Harms, K. , Allcock, W. , Bacon, C. , and Lang, S. , 2011, “ Understanding and Improving Computational Science Storage Access Through Continuous Characterization,” ACM Trans. Storage, 7(3), pp. 8:1–8:26. [CrossRef]
Karypis, G. , 2011, “ metis—A Software Package for Partitioning Unstructured Graphs, Partitioning Meshes, and Computing Fill-Reducing Orderings of Sparse Matrices Version 5.0. Software Package,” University of Minnesota, Minneapolis, MN.
Babajimopoulos, A. , Assanis, D. N. , Flowers, D. L. , Aceves, S. M. , and Hessel, R. P. , 2005, “ A Fully Coupled Computational Fluid Dynamics and Multi-Zone Model With Detailed Chemical Kinetics for the Simulation of Premixed Charge Compression Ignition Engines,” Int. J. Engine Res., 6(5), pp. 497–512. [CrossRef]
Kodavasal, J. , Keum, S. , and Babajimopoulos, A. , 2011, “ An Extended Multi-Zone Combustion Model for PCI Simulation,” Combust. Theory Modell., 15(6), pp. 893–910. [CrossRef]
Cohen, S. D. , and Hindmarsh, A. C. , 1995, “ CVODE, A Stiff/Nonstiff ODE Solver in C,” Stanford University, Lawrence Livermore National Laboratory, Livermore, CA, Report No. UCRL-JC-121014, Rev. 1.

Figures

Grahic Jump Location
Fig. 1

CFD domain of 360 deg shown during injection. Note that the profiling and optimization of the code has been done near TDC, during ignition, and the spray is shown for context, since the simulation itself was run from IVC through injection.

Grahic Jump Location
Fig. 2

Darshan output for I/O sizes with the original code showing on the order of half a billion file write calls in the 4–12 bytes range

Grahic Jump Location
Fig. 3

Darshan output for I/O sizes with improvement to file write subroutines—fewer file write calls on the order of a thousand, with bigger chunks of data on the order of MB written in each file write

Grahic Jump Location
Fig. 4

Scaling performance of binary file write operation

Grahic Jump Location
Fig. 5

Illustration of data transferred between ranks. Here, rank 0 sends to rank 1 C neighbor cells each having K properties (pressure, temperature, etc.), represented by P.

Grahic Jump Location
Fig. 6

Original communication scheme

Grahic Jump Location
Fig. 7

Improved collective communication scheme

Grahic Jump Location
Fig. 8

Scaling performance of the communication operation during mesh load balancing. Results shown for the first load balancing cycle during initialization.

Grahic Jump Location
Fig. 9

Illustration of chemistry load imbalance during combustion. R1 denotes a region where chemical kinetics is solved by an arbitrary rank 1 and R2 denotes a region where chemical kinetics is solved by another arbitrary rank 2.

Grahic Jump Location
Fig. 10

Improvement in chemistry load balance with stiffness-based load balancing schemes. Imbalance shown in terms of the ratio of chemistry time spent in every time-step by the slowest and fastest ranks.

Grahic Jump Location
Fig. 11

Computation time for simulation with the original code, and the improved code with stiffness-based chemistry load balancing

Grahic Jump Location
Fig. 12

Actual compared to ideal speedup in computation time for the original code, and improved code with stiffness-based chemistry load balancing

Grahic Jump Location
Fig. 13

Computational expense in terms of CPU-hours for the original code, and the improved code with stiffness-based chemistry load balancing, over the range of configurations studied

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In