SciMLBenchmarks.jl: Benchmarks for Scientific Machine Learning (SciML) and Differential Equation Solver Software
SciMLBenchmarks.jl holds webpages, pdfs, and notebooks showing the benchmarks for the SciML Scientific Machine Learning Software ecosystem, including crosslanguage benchmarks of differential equation solvers and methods for parameter estimation, training universal differential equations (and subsets like neural ODEs), and more.
Interactive Notebooks
To run the tutorials interactively via Jupyter notebooks and benchmark on your own machine, install the package and open the tutorials like:
]add "https://github.com/SciML/SciMLBenchmarks.jl"
using SciMLBenchmarks
SciMLBenchmarks.open_notebooks()
Table of Contents
 MultiLanguage Wrapper Benchmarks
 ODE Solver MultiLanguage Wrapper Package WorkPrecision Benchmarks (MATLAB, SciPy, Julia, deSolve (R))
 Torchdiffeq vs DifferentialEquations.jl (/ DiffEqFlux.jl) Benchmarks
 torchdiffeq vs Julia DiffEqFlux Neural ODE Training Benchmark
 torchsde vs DifferentialEquations.jl / DiffEqFlux.jl
 JITCODE vs SciPy vs DifferentialEquations.jl on large network dynamics
 DifferentialEquations.jl vs Mujuco and DiffTaichi
 Nonstiff ODEs
 Stiff ODEs
 Method of Lines PDEs
 Filament PDE Discretization WorkPrecision Diagrams
 AllenCahn Finite Difference WorkPrecision Diagrams
 AllenCahn Pseudospectral WorkPrecision Diagrams
 Burger's Finite Difference WorkPrecision Diagrams
 Burger's Pseudospectral WorkPrecision Diagrams
 KdV Finite Difference WorkPrecision Diagrams
 KdV Pseudospectral WorkPrecision Diagrams
 Kuramoto–Sivashinsky Finite Difference WorkPrecision Diagrams
 Kuramoto–Sivashinsky Pseudospectral WorkPrecision Diagrams
 Dynamical ODEs
 Nbody problems
 Nonstiff SDEs
 Stiff SDEs
 Nonstiff DDEs
 Stiff DDEs
 Jump Equations
 Parameter Estimation
 Lorenz Equation Parameter Estimation by Optimization Methods
 Bayesian LotkaVolterra Parameter Estimation
 Bayesian Lorenz Equation Estimation
 Bayesian FitzHughNagumo Equation Estimation
 Lotka Volterra Equation Parameter Estimation by Optimization Methods
 FitzHughNagumo Equation Parameter Estimation by Optimization Methods
The following tests were developed for the paper Adaptive Methods for Stochastic Differential Equations via Natural Embeddings and Rejection Sampling with Memory. These notebooks track their latest developments.

SDE Adaptivity
Current Summary
The following is a quick summary of the benchmarks. These paint broad strokes over the set of tested equations and some specific examples may differ.
NonStiff ODEs
 OrdinaryDiffEq.jl's methods are the most efficient by a good amount
 The
Vern
methods tend to do the best in every benchmark of this category  At lower tolerances,
Tsit5
does well consistently.  ARKODE and Hairer's
dopri5
/dop853
perform very similarly, but are both far less efficient than theVern
methods.  The multistep methods,
CVODE_Adams
andlsoda
, tend to not do very well.  The ODEInterface multistep method
ddeabm
does not do as well as the other multistep methods.  ODE.jl's methods are not able to consistently solve the problems.
 Fixed time step methods are less efficient than the adaptive methods.
Stiff ODEs
 In this category, the best methods are much more problem dependent.
 For smaller problems:
Rosenbrock23
,lsoda
, andTRBDF2
tend to be the most efficient at high
tolerances.Rodas4
andRodas5
tend to be the most efficient at low tolerances.
 For larger problems (Filament PDE):
CVODE_BDF
does the best at all tolerances. The ESDIRK methods like
TRBDF2
andKenCarp4
can come close.
radau
is always the most efficient when tolerances go to the low extreme (1e13
) Fixed time step methods tend to diverge on every tested problem because the high stiffness results in divergence of the Newton solvers.
 ARKODE is very inconsistent and requires a lot of tweaking in order to not
diverge on many of the tested problems. When it doesn't diverge, the similar
algorithms in OrdinaryDiffEq.jl (
KenCarp4
) are much more efficient in most cases.  ODE.jl and GeometricIntegrators.jl fail to converge on any of the tested problems.
Dynamical ODEs
 Higher order (generally order >=6) symplectic integrators are much more efficient than the lower order counterparts.
 For high accuracy, using a symplectic integrator is not preferred. Their extra cost is not necessary since the other integrators are able to not drift simply due to having low enough error.
 In this class, the
DPRKN
methods are by far the most efficient. TheVern
methods do well for not being specific to the domain.
NonStiff SDEs
 For simple 1dimensional SDEs at low accuracy, the
EM
andRKMil
methods can do well. Beyond that, they are simply outclassed.  The
SRA
andSRI
methods both are very similar withinclass on the simple SDEs. SRA3
is the most efficient when applicable and the tolerances are low. Generally, only low accuracy is necessary to get to sampling error of the mean.
 The adaptive method is very conservative with error estimates.
Stiff SDEs
 The high order adaptive methods (
SRIW1
) generally do well on stiff problems.  The "standard" loworder implicit methods,
ImplicitEM
andImplicitRK
, do not do well on all stiff problems. Some exceptions apply to wellbehaved problems like the Stochastic Heat Equation.
NonStiff DDEs
 The efficiency ranking tends to match the ODE Tests, but the cutoff from low to high tolerance is lower.
Tsit5
does well in a large class of problems here. The
Vern
methods do well in low tolerance cases.
Stiff DDEs
 The Rosenbrock methods, specifically
Rodas5
, perform well.
Parameter Estimation
 Broadly two different approaches have been used, Bayesian Inference and Optimisation algorithms.
 In general it seems that the optimisation algorithms perform more accurately but that can be attributed to the larger number of data points being used in the optimisation cases, Bayesian approach tends to be slower of the two and hence lesser data points are used, accuracy can increase if proper data is used.
 Within the different available optimisation algorithms, BBO from the BlackBoxOptim package and GN_CRS2_LM for the global case while LD_SLSQP,LN_BOBYQA and LN_NELDERMEAD for the local case from the NLopt package perform the best.
 Another algorithm being used is the QuadDIRECT algorithm, it gives very good results in the shorter problem case but doesn't do very well in the case of the longer problems.
 The choice of global versus local optimization make a huge difference in the timings. BBO tends to find the correct solution for a global optimization setup. For local optimization, most methods in NLopt, like :LN_BOBYQA, solve the problem very fast but require a good initial condition.
 The different backends options available for Bayesian method offer some tradeoffs beteween time, accuracy and control. It is observed that sufficiently high accuracy can be observed with any of the backends with the fine tuning of stepsize, constraints on the parameters, tightness of the priors and number of iterations being passed.
Contributing
All of the files are generated from the Weave.jl files in the benchmarks
folder. To run the generation process, do for example:
]activate SciMLBenchmarks # Get all of the packages
using SciMLBenchmarks
SciMLBenchmarks.weave_file("NonStiffODE","linear_wpd.jmd")
To generate all of the files in a folder, for example, run:
SciMLBenchmarks.weave_folder("NonStiffODE")
To generate all of the notebooks, do:
SciMLBenchmarks.weave_all()
Each of the benchmarks displays the computer characteristics at the bottom of the benchmark. Since performancenecessary computations are normally performed on compute clusters, the official benchmarks use a workstation with an Intel Xeon CPU E52680 v4 @ 2.40GHz to match the performance characteristics of a standard node in a high performance computing (HPC) cluster or cloud computing setup.
Inspecting Benchmark Results
To see benchmark results before merging, click into the BuildKite, click onto Artifacts, and then investigate the trained results.