Re-targeted harmonic mean estimator

The original harmonic mean estimator was revised by Gelfand (1994) by introducing an arbitrary density \(\varphi(\theta)\) to relate the reciprocal of the marginal likelihood to the likelihood through the following expectation:

\[\rho = \mathbb{E}_{\text{P}(\theta | y)} \biggl[\frac{\varphi(\theta)}{\mathcal{L}(\theta) \pi(\theta)} \biggr] = \int \,\text{d} \theta\frac{\varphi(\theta)}{\mathcal{L}(\theta) \pi(\theta)} \text{P}(\theta | y) = \int \,\text{d} \theta \frac{\varphi(\theta)}{\mathcal{L}(\theta) \pi(\theta)} \frac{\mathcal{L}(\theta) \pi(\theta)}{z} = \frac{1}{z},\]

where the final line follows since the density \(\varphi(\theta)\) must be normalised. The above expression motivates the estimator:

\[\label{eqn:harmonic_mean_retargeted} \hat{\rho} = \frac{1}{N} \sum_{i=1}^N \frac{\varphi(\theta_i)}{\mathcal{L}(\theta_i) \pi(\theta_i)}, \quad \theta_i \sim \text{P}(\theta | y).\]

The normalised density \(\varphi(\theta)\) can be interpreted as an alternative importance sampling target distribution, as we will see, hence we refer to this approach as the re-targeted harmonic mean estimator. Note that the original harmonic mean estimator is recovered for the target distribution \(\varphi(\theta) = \pi(\theta)\).

Importance sampling interpretation

With the introduction of the distribution \(\varphi(\theta)\), the importance sampling interpretation of the harmonic mean estimator reads

\[\rho = \int \,\text{d} \theta \: \frac{\varphi(\theta)}{\mathcal{L}(\theta) \pi(\theta)} \: \text{P}(\theta | y) = \int \,\text{d} \theta \: \frac{1}{z} \: \frac{\varphi(\theta)}{\text{P}(\theta | y)} \: \text{P}(\theta | y).\]

It is clear that the distribution \(\varphi(\theta)\) now plays the role of the importance sampling target distribution. One is free to choose \(\varphi(\theta)\), with the only constraint being that it is a normalised distribution. It is therefore possible to select the target distribution \(\varphi(\theta)\) such that it has narrower tails than the posterior, which we recall plays the role of the importance sampling density, thereby avoiding the problematic configuration of the original harmonic mean estimator. The selection of appropriate target densities \(\varphi(\theta)\) for general problems remains and open question that is known to be difficult, particularly difficult in high dimensions (Chib 1995).