# DynamicProgramming.jl

A Julia package for discrete stochastic dynamic programming.

Build Status |
Coverage |
---|---|

## Installation

This package is not yet registered. **Until it is, things may change. It is perfectly
usable but should not be considered stable**.

You can install it by typing

`julia> Pkg.add(url="https://github.com/odow/DynamicProgramming.jl")`

## Initialise Model

A SDP model is stored in the `SDPModel`

type. A `SDPModel`

can be intialised
using the following `SDPModel() do ... end`

block structure:

```
m = SDPModel(
stages = 1. # Int
sense = :Min # Symbol (:Min or :Max)
) do sp, t
... problem definition ...
end
```

`sp`

can be given any name but refers to the *stage problem*. `t`

can also be
given any name, but refers to an index that runs from `1`

to `T`

where `T`

is
the number of stages.

Inside the `SDPModel`

definition, we define our subproblems. We first need to
add some state variables, some control (or action) variables, and some noise (or
stochastic) variables.

## Initialise Variables

States can be added with the following macro:

```
@states(sp, begin
x in linspace(0, 1, 10)
end)
```

This creates a state variable `x`

that is discretised into the set
`linspace(0, 1, 10)`

. Note that currently, all state dimensions get converted
into `Float64`

representations. The discretisation should be any type that can
be converted to a `Vector{Float64}`

type.

Controls can be added with the `@controls`

macro that has similar syntax.
However there is less restriction on the type. The discretisation should just be
an iterable subtype of `AbstractVector`

.

Noise (or stochastic variables) can be added with the `@noises`

macro:

```
@noises(sp, begin
u in DiscreteDistribution([1,2,3], [0.5, 0.25, 0.25])
v in 1:10
end)
```

In contrast to the other two macros, there is a slight subtlety. The
discretisations can either be subtypes of `AbstractVector`

(in which case their
realisations are assumed to be uniformly sampled), or a `DiscreteDistribution`

.

The `DiscreteDistribution`

constructor is
`DiscreteDistribution(observations::AbstractVector, probability::AbstractVector)`

.
This realisations `observations`

are sampled with probability `probability`

.

If more than one noise is defined, then the multiple noises are assumed to be independent.

## Dynamics

You must provide a function that takes four inputs.

```
function foo(states_out, states, controls, noises)
end
```

However, we prefer the anonymous function syntax:

```
dynamics!(sp) do y, x, u, w
... definitions ...
end
```

This anonymous function must take the current state `x`

, a control `u`

and a
noise `w`

and update the new state `y`

.

You can refer to model variables using the `[]`

indexing operator. For example,
if we defined a state variable `quantity`

, we could refer it as `x[quantity]`

.

By thinking about variable scopes it is possible to encapsulate all the necessary data into this syntax.

**Important:** the function should return a single `Float64`

value corresponding
to the cost (or profit) accrued in the current stage.

## Terminal Objective

The terminal objective function takes as input a vector of the final state at
the end of the finite time horizon. It returns a single `Float64`

value
corresponding to the cost (or profit) of ending in that state.

```
terminalobjective!(sp) do x
... definitions ...
end
```

## Constraints

The constraints function takes as input vectors for the initial state, control
and noise. It should return a single `Bool`

value indicating if the state,
control, noise combination is feasible. Typically this can be implemented by a
chained series of boolean comparisons.

```
constraints!(sp) do x, u, w
... definitions ...
end
```

## Solve

The `solve`

function takes as input the initalised `SDPModel`

object, as well as
two keyword arguments.

The `realisation`

must be one of `WaitAndSee`

, `HereAndNow`

or `ExpectedValue`

.

The `WaitAndSee`

model observes the noise before choosing the optimal control.
The `HereAndNow`

model chooses the best control before observing the noise.
The `ExpectedValue`

model substitutes the noise for the expected value of each
independent noise.

The `riskmeasure`

is a nested `λE[x] + (1-λ)CVaRᵦ[x]`

.

```
solve(m::SDPModel,
realisation=WaitAndSee,
riskmeasure=NestedCVaR(beta=0.5, lambda=0.5)
)
```

## Simulate

Once a `SDPModel`

has been solved, it is possible to simulate the performance of
the policy using the function `simulate(m::SDPModel, N::Int; kwargs...)`

.
`m`

is the solved `SDPModel`

to be simulated. `N`

is the number of realisations
to perform. Initial values for the state variables are given via the keyword
arguments.

For example:

```
results = simulate(m,
500,
contracts = 0,
price = 4.5,
production = 0.
)
```

## Visualise

It is possible to create an interactive visualisation of the simulated policy
with the `@visualise`

macro. The following keywords should be wrapped with
parentheses.

`"cumulative" = false`

Plot the cumulation of the variable over stages`"title" = ""`

Plot title`"xlabel" = "Stages"`

Label for x axis`"ylabel" = ""`

Label for y axis`"interpolate" = "linear"`

D3.js interpolation method to use. See the D3 wiki for more.

The following example gives an example of possible syntax:

```
@visualise(results, stage, replication, begin
results[:Current][stage][replication], (title="Accumulated Profit", ylabel="Accumulated Profit (\$)", cumulative=true)
results[:x][stage][replication], (title="Value of a State", ylabel="Level")
results[:u][stage][replication], (title="Value of a Control")
results[:w][stage][replication], (title="Value of a Noise", interpolate="step")
end)
```