ParameterSchedulers

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Momentum, Exp(start = 1e-2, decay = 0.8))

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. The higher order schedules should make it so that you will rarely need to write a schedule from scratch.

You can read this paper for more information on the schedules below.

Schedule	Description	Type	Example
`Step(; start, decay, step_sizes)`	Exponential decay by `decay` every step in `step_sizes`	Decay	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = Step(start = 1.0, decay = 0.8, step_sizes = [2, 3, 2]) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`Exp(start, decay)`	Exponential decay by `decay` every iteration	Decay	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = Exp(start = 1.0, decay = 0.5) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`CosAnneal(;l0, l1, period)`	Cosine annealing	Cyclic	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = CosAnneal(l0 = 0.0, l1 = 1.0, period = 4) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`OneCycle(nsteps, maxval)`	One cycle cosine	Complex	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = OneCycle(10, 1.0) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`Triangle(l0, l1, period)`	Triangle wave function	Cyclic	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = Triangle(l0 = 0.0, l1 = 1.0, period = 2) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`TriangleDecay2(l0, l1, period)`	Triangle wave function with half the amplitude every `period`	Cyclic	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = TriangleDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`TriangleExp(l0, l1, period, decay)`	Triangle wave function with exponential amplitude decay at rate `decay`	Cyclic	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = TriangleExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`Poly(start, degree, max_iter)`	Polynomial decay at degree `degree`.	Decay	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = Poly(start = 1.0, degree = 2, max_iter = t[end]) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`Inv(start, decay, degree)`	Inverse decay at rate `(1 + t * decay)^degree`	Decay	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = Inv(start = 1.0, degree = 2, decay = 0.8) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`Sin(;l0, l1, period)`	Sine function	Cyclic	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = Sin(l0 = 0.0, l1 = 1.0, period = 2) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`SinDecay2(l0, l1, period)`	Sine function with half the amplitude every `period`	Cyclic	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = SinDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`
`SinExp(l0, l1, period)`	Sine function with exponential amplitude decay at rate `decay`	Cyclic	`using UnicodePlots, ParameterSchedulers # hide t = 1:10 \|> collect # hide s = SinExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide`