Log-space variance

harmonic computes estimators of the reciprocal evidence and its variance, denoted by \(\hat{\rho}\) and \(\hat{\sigma}^2\) respectively.

We compute the natural logarithm of these quantities for numerical stability (using a number of numerical tricks, such as the LogSumExp trick). We therefore directly compute \(\log ( \hat{\rho} )\) and \(\log ( \hat{\sigma} )\).

While quantities are computed in log space for numerical stability, the variance that is computed relates to the underlying reciprocal evidence (not the log reciprocal evidence), i.e. the terms computed can be considered as the estimate and error \(\hat{\rho} \pm \hat{\sigma}\). This is of course not equivalent to a variance estimate of the log space estimate, which can be seen straightforwardly since \(\log ( \text{var}(x) ) \not = \text{var}(\log (x ))\) as summation and the natural logarithm do not commute.

In some settings one may be interested in an error estimate defined in log space, i.e. the log-space error \(\hat{\zeta}_\pm\) defined by

\[\log ( \hat{\rho} \pm \hat{\sigma} ) = \log (\hat{\rho}) + \hat{\zeta}_\pm .\]

The log-space error estimate can be computed by

\[\hat{\zeta}_\pm = \log (\hat{\rho} \pm \hat{\sigma} ) - \log (\hat{\rho}) .\]

This may also be expressed as

\[\hat{\zeta}_\pm = \log(1 \pm \hat{\sigma} / \hat{\rho}) ,\]

where

\[\hat{\sigma} / \hat{\rho} = \exp \bigl( \log(\hat{\sigma}) - \log(\hat{\rho}) \bigr) .\]

The advantage of this second approach is that it avoids explicitly computing \(\exp(\log(\hat{\rho})) \pm \exp(\log(\hat{\sigma}))\), which has the potental to over- or under-flow. The second, more stable, approach is implemented in Evidence.compute_ln_inv_evidence_errors.