# Divergences.jl

`Divergences.jl`

is a Julia package that makes it easy to evaluate divergence measures between two vectors. The package allows calculating the *gradient* and the diagonal of the *Hessian* of several divergences. These divergences are used to good effect by the MomentBasedEstimators package.

## Supported divergences

The package defines an abstract `Divergence`

type with the following suptypes:

- Kullback-Leibler divergence
`KullbackLeibler`

- Chi-square distance
`ChiSquared`

- Reverse Kullback-Leibler divergence
`ReverseKullbackLeibler`

- Cressie-Read divergences
`CressieRead`

These divergences differ from the equivalent ones defined in the `Distances`

package because they are normalized. Also, the package provides methods for calculating their gradient and the (diagonal elements of the) Hessian matrix.

The constructors for the types above are straightforward

```
KullbackLeibler()
ChiSqaured()
ReverseKullbackLeibler()
```

The `CressieRead`

type define a family of divergences indexed by a parameter `alpha`

. The constructor for `CressieRead`

is

`CR(::Real)`

The Hellinger divergence is obtained by `CR(-1/2)`

. For a certain value of `alpha`

, `CressieRead`

correspond to a divergence that has a specific type defined. For instance `CR(1)`

is equivalent to `ChiSquared`

although the underlying code for evaluation and calculation of the gradient and Hessian are different.

Three versions of each divergence in the above list are implemented currently. A *vanilla* version, a modified version, and a fully modified version. These modifications extend the domain of the divergence.

The **modified** version takes an additional argument that specifies the point at which the divergence is modified by a convex extension.

```
ModifiedKullbackLeibler(theta::Real)
ModifiedReverseKullbackLeibler(theta::Real)
ModifiedCressieRead(alpha::Real, theta::Real)
```

Similarly, the **fully modified** version takes **two** additional arguments that specify the points at which the divergence is modified by a convex extensions.

```
FullyModifiedKullbackLeibler(phi::Real, theta::Real)
FullyModifiedReverseKullbackLeibler(phi::Real, theta::Real)
FullyModifiedCressieRead(alpha::Real, phi::Real, theta::Real)
```

## Basic usage

### Divergence between two vectors

Each divergence corresponds to a *divergence type*. You can always compute a certain divergence between two vectors using the following syntax

`d = evaluate(div, x, y)`

Here, `div`

is an instance of a divergence type. For example, the type for Kullback Leibler divergence is `KullbackLeibler`

(more divergence types are described in some details in what follows), then the Kullback Leibler divergence between `x`

and `y`

can be computed

`d = evaluate(KullbackLeibler(), x, y)`

We can also calculate the diverge between the vector `x`

and the unit vector

`r = evaluate(KullbackLeibler(), x)`

The `Divergence`

type is a subtype of `PreMetric`

defined in the `Distances`

package. As such, the divergences can be evaluated row-wise and column-wise for `X::Matrix`

and `Y::Matrix`

.

`rowise(div, X, Y)`

`colwise(div, X, Y)`

### Gradient of the divergence

To calculate the gradient of `div::Divergence`

with respect to `x::AbstractArray{Float64, 1}`

the
`gradient`

method can be used

`g = gradient(div, x, y)`

or through its in-place version

`gradient!(Array(Float64, size(x)), div, x, y)`

### Hessian of the divergence

The `hessian`

method calculate the Hessian of the divergence with respect to `x`

`h = hessian(div, x, y)`

Its in-place variant is also defined

`hessian!(Array(Float64, size(x)), div, x, y)`

Notice that the Hessian of a divergence is sparse, where the diagonal entries are the only ones different from zero. For this reason, `hessian(div, x, y)`

return an `Array{Float64,1}`

with the diagonal entries of the hessian.

## List of divergences

[To be added]