Continuous Bernoulli distribution

Not to be confused with Bernoulli distribution.

Continuous Bernoulli distribution
Probability density function
Parameters	$\lambda =1/(1+e^{-\theta })\in (0,1)$	$\theta \in ℝ$, natural parameter
Support	$x\in [0,1]$	$x\in [0,1]$
PDF	$C(\lambda ){\lambda }^x(1-\lambda {)}^{1-x}$ where $C(\lambda )=\{\begin{array}{cc}2& \text{if }\lambda =\frac{1}{2}\\ \frac{2{\tanh }^{-1}(1-2\lambda )}{1-2\lambda }& \text{ otherwise}\end{array}$	$f(x\mid \theta )=\{\begin{array}{cc}1& \theta =0\\ \exp (x\theta -\log \{(e^{\theta }-1)/\theta \})& \theta \ne 0\end{array}$
CDF	$F(x\mid \lambda )=\{\begin{array}{cc}x,& \lambda =\frac{1}{2}\\ \frac{{\lambda }^x(1-\lambda {)}^{1-x}+\lambda -1}{2\lambda -1},& \text{otherwise}\end{array}$	$F(x\mid \theta )=\{\begin{array}{cc}x& \theta =0\\ (e^{\theta x}-1)/(e^{\theta }-1)& \theta \ne 0\end{array}$
Mean	$\mathrm{E}[X]=\{\begin{array}{cc}\frac{1}{2}& \lambda =\frac{1}{2}\\ \frac{\lambda }{2\lambda -1}+\frac{1}{2{\tanh }^{-1}(1-2\lambda )},& \text{otherwise}\end{array}$	$\mathrm{E}[X]=\{\begin{array}{cc}1/2& \theta =0\\ e^{\theta }/(e^{\theta }-1)-{\theta }^{-1}& \theta \ne 0\end{array}$
Variance	$\mathrm{Var}(X)=\{\begin{array}{cc}\frac{1}{12},& \lambda =\frac{1}{2}\\ -\frac{\lambda (1-\lambda )}{(1-2\lambda {)}^2}+\frac{1}{(2{\tanh }^{-1}(1-2\lambda ){)}^2},& \text{otherwise}\end{array}$	$\mathrm{Var}(X)=\{\begin{array}{cc}1/12& \theta =0\\ (2-e^{\theta }-e^{-\theta }{)}^{-1}+{\theta }^2& \theta \ne 0\end{array}$

In probability theory, statistics, and machine learning, the continuous Bernoulli distribution is a family of continuous probability distributions parameterized by a single shape parameter $\lambda \in (0,1)$, defined on the unit interval $x\in [0,1]$, by:

$p(x|\lambda )\propto {\lambda }^x(1-\lambda {)}^{1-x}.$

The continuous Bernoulli distribution arises in deep learning and computer vision, specifically in the context of variational autoencoders, for modeling the pixel intensities of natural images. As such, it defines a proper probabilistic counterpart for the commonly used binary cross entropy loss, which is often applied to continuous, $[0,1]$-valued data. This practice amounts to ignoring the normalizing constant of the continuous Bernoulli distribution, since the binary cross entropy loss only defines a true log-likelihood for discrete, $\{0,1\}$-valued data.

The continuous Bernoulli also defines an exponential family of distributions. Writing $\theta =\log (\lambda /(1-\lambda ))$ for the natural parameter, the density can be rewritten in canonical form: $p(x|\theta )\propto \exp (\theta x)$.

Statistical inference

Given an independent sample of $n$ points $x_1,\dots ,x_n$ with $x_i\in [0,1]\mathrm{\forall }i$ from continuous Bernoulli, the log-likelihood of the natural parameter $\theta$ is

$ℒ(\theta )=\theta \sum _{i=1}^n{x}_i-n\log \{(e^{\theta }-1)/\theta \}$

and the maximum likelihood estimator of the natural parameter $\theta$ is the solution of ${ℒ}^{\prime }(\theta )=0$, that is, $\hat{\theta }$ satisfies

$\frac{e^{\hat{\theta }}}{e^{\hat{\theta }}-1}-\frac{1}{\hat{\theta }}=\frac{1}{n}\sum _{i=1}^n{x}_i$

where the left hand side $e^{\hat{\theta }}/(e^{\hat{\theta }}-1)-{\hat{\theta }}^{-1}$ is the expected value of continuous Bernoulli with parameter $\hat{\theta }$. Although $\hat{\theta }$ does not admit a closed-form expression, it can be easily calculated with numerical inversion.

Further properties

The entropy of a continuous Bernoulli distribution is

$\mathrm{H}[X]=\{\begin{array}{cc}0& \text{ if }\lambda =\frac{1}{2}\\ \frac{\lambda \log \left(\lambda \right)-(1-\lambda )\log (1-\lambda )}{1-2\lambda }-\log \left(\frac{2{\tanh }^{-1}(1-2\lambda )}{e(1-2\lambda )}\right)& \text{ otherwise}\end{array}$

Related distributions

Bernoulli distribution

The continuous Bernoulli can be thought of as a continuous relaxation of the Bernoulli distribution, which is defined on the discrete set $\{0,1\}$ by the probability mass function:

$p(x)=p^x(1-p{)}^{1-x},$

where $p$ is a scalar parameter between 0 and 1. Applying this same functional form on the continuous interval $[0,1]$ results in the continuous Bernoulli probability density function, up to a normalizing constant.

Uniform distribution

The Uniform distribution between the unit interval [0,1] is a special case of continuous Bernoulli when $\lambda =1/2$ or $\theta =0$.

Exponential distribution

An exponential distribution with rate $\Lambda$ restricted to the unit interval [0,1] corresponds to a continuous Bernoulli distribution with natural parameter $\theta =-\Lambda <0$.

Continuous categorical distribution

The multivariate generalization of the continuous Bernoulli is called the continuous-categorical.