BenchmarkCI.jl
BenchmarkCI.jl provides an easy way to run benchmark suite via GitHub Actions. It only needs a minimal setup if there is a benchmark suite declared by BenchmarkTools.jl / PkgBenchmark.jl API.
Warning This package is still experimental. Make sure to fix the version number in your CI setup.
Setup
BenchmarkCI.jl requires PkgBenchmark.jl to work. See
Defining a benchmark suite · PkgBenchmark.jl
for more information. BenchmarkCI.jl also requires a Julia project
benchmark/Project.toml that is used for running the benchmark.
Create a workflow file (required)
Create (say) .github/workflows/benchmark.yml with the following
configuration:
name: Run benchmarks
on:
pull_request:
jobs:
Benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@latest
with:
version: 1
- uses: julia-actions/julia-buildpkg@latest
- name: Install dependencies
run: julia -e 'using Pkg; pkg"add PkgBenchmark BenchmarkCI@0.1"'
- name: Run benchmarks
run: julia -e 'using BenchmarkCI; BenchmarkCI.judge()'
- name: Post results
run: julia -e 'using BenchmarkCI; BenchmarkCI.postjudge()'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}If you don't want to benchmark your code for every push of every PR, then you can conditionally trigger the jobs on a label:
name: Run benchmarks
on:
pull_request:
types: [labeled, opened, synchronize, reopened]
# Only trigger the benchmark job when you add `run benchmark` label to the PR
jobs:
Benchmark:
runs-on: ubuntu-latest
if: contains(github.event.pull_request.labels.*.name, 'run benchmark')
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@latest
with:
version: 1
- uses: julia-actions/julia-buildpkg@latest
- name: Install dependencies
run: julia -e 'using Pkg; pkg"add PkgBenchmark BenchmarkCI@0.1"'
- name: Run benchmarks
run: julia -e 'using BenchmarkCI; BenchmarkCI.judge()'
- name: Post results
run: julia -e 'using BenchmarkCI; BenchmarkCI.postjudge()'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}Setup with benchmark/Manifest.toml
If benchmark/Manifest.toml is checked into the repository,
benchmark/Project.toml must include parent project as well. Run
dev .. in benchmark/ directory to add it:
shell> cd ~/.julia/dev/MyProject/
shell> cd benchmark/
(@v1.x) pkg> activate .
Activating environment at `~/.julia/dev/MyProject/benchmark/Project.toml`
(benchmark) pkg> dev ..
Additional setup (recommended)
It is recommended to add following two lines in .gitignore:
/.benchmarkci
/benchmark/*.json
This is useful for running BenchmarkCI locally (see below).
Printing benchmark result (optional)
Posting the benchmark result as a comment for every push for each PR
may be too noisy. In such case, using
BenchmarkCI.displayjudgement() instead of BenchmarkCI.postjudge()
may be useful.
- name: Print judgement
run: julia -e 'using BenchmarkCI; BenchmarkCI.displayjudgement()'Store benchmark result in a Git branch (optional; very experimental)
Alternatively, the benchmark result and report markdown can be pushed
to a git branch benchmark-results
(example).
- name: Push results
run: julia -e "using BenchmarkCI; BenchmarkCI.pushresult()"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SSH_KEY: ${{ secrets.DOCUMENTER_KEY }}This method can also be used in Travis CI. See
this example .travis.yml.
WARNING: Storage format may be changed across releases.
Running BenchmarkCI interactively
shell> cd ~/.julia/dev/MyProject/
julia> using BenchmarkCI
julia> BenchmarkCI.judge()
...
julia> BenchmarkCI.displayjudgement()
...