Confidence intervals & Credible intervals

https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval

user28: 

My understanding is as follows:

Background

Suppose that you have some data x$x$ and you are trying to estimate θ$\theta$. You have a data generating process that describes how x$x$ is generated conditional on θ$\theta$. In other words you know the distribution of x$x$ (say, f(x|θ)$f(x|\theta )$.

Inference Problem

Your inference problem is: What values of θ$\theta$ are reasonable given the observed data x$x$ ?

Confidence Intervals

Confidence intervals are a classical answer to the above problem. In this approach, you assume that there is true, fixed value of θ$\theta$. Given this assumption, you use the data x$x$ to get to an estimate of θ$\theta$(say, θ^$\hat{\theta }$). Once you have your estimate you want to assess where the true value is in relation to your estimate.

Notice that under this approach the true value is not a random variable. It is a fixed but unknown quantity. In contrast, your estimate is a random variable as it depends on your data x$x$ which was generated from your data generating process. Thus, you realize that you get different estimates each time you repeat your study.

The above understanding leads to the following methodology to assess where the true parameter is in relation to your estimate. Define an interval, I≡[lb(x),ub(x)]$I\equiv [lb(x),ub(x)]$ with the following property:

P(θ∈I)=0.95$P(\theta \in I)=0.95$

An interval constructed like the above is what is called a confidence interval. Since, the true value is unknown but fixed, the true value is either in the interval or outside the interval. The confidence interval then is a statement about the likelihood that the interval we obtain actually has the true parameter value. Thus, the probability statement is about the interval (i.e., the chances that interval which has the true value or not) rather than about the location of the true parameter value.

In this paradigm, it is meaningless to speak about the probability that a true value is less than or greater than some value as the true value is not a random variable.

Credible Intervals

In contrast to the classical approach, in the bayesian approach we assume that the true value is a random variable. Thus, we capture the our uncertainty about the true parameter value by a imposing a prior distribution on the true parameter vector (say f(θ)$f(\theta )$).

Using bayes theorem, we construct the posterior distribution for the parameter vector by blending the prior and the data we have (briefly the posterior is f(θ|−)∝f(θ)f(x|θ)$f(\theta |-)\propto f(\theta )f(x|\theta )$).

We then arrive at a point estimate using the posterior distribution (e.g., use the mean of the posterior distribution). However, since under this paradigm, the true parameter vector is a random variable, we also want to know the extent of uncertainty we have in our point estimate. Thus, we construct an interval such that the following holds:

P(l(θ)≤θ≤ub(θ))=0.95$P(l(\theta )\le \theta \le ub(\theta ))=0.95$

The above is a credible interval.

Summary

Credible intervals capture our current uncertainty in the location of the parameter values and thus can be interpreted as probabilistic statement about the parameter.

In contrast, confidence intervals capture the uncertainty about the interval we have obtained (i.e., whether it contains the true value or not). Thus, they cannot be interpreted as a probabilistic statement about the true parameter values.