In this lecture we will build on the understanding of the sampling distribution of the mean and the standard error developed in the previous lecture and talk more about the uncertainty that is inherent in estimation. We will learn how interval estimates can be used to quantify this uncertainty around point estimates. In particular, we will focus on the confidence interval and talk about how it is constructed using the t distribution and how to interpret it.

Before we dive in, let’s just reiterate what the point of doing statistics is. The reason why we are performing statistical tests and build statistical models of the world is to help us analyse and interpret the results of quantitative research. This research aims to describe and explain aspects of the world around us and make predictions about it. In statistical terms, the specific part of the world that a research study is focusing on can be viewed as a population and the particular aspect of interest can be seen as some parameter of this population. It can be anything really, the typical value in the population, the variability in the population, or the relationship between two or multiple variables in the population.

However, accessing and measuring the entire population and, by extension, observing these parameters is not possible for reasons we talked about on multiple occasions. The best we can then do is sample observations from the population of interest and estimate the values of these parameters based on some statistics calculated on the sample.

In other words, statistics is all about making inferences about populations based on samples. If we could measure the entire population, we wouldn’t really need statistics!

Point estimates

You’ve already heard of things like the sample mean, median, or mode. These measures of central tendency tell us about the “most typical” value in the distribution. Just like with everything, we don’t know the value of these measures in the population and when we calculate their values based on a sample for the purpose of doing statistics, we do so in order to get an estimate of what the value of the population parameter is most likely to be.

Because each one of these estimates is a single value representing some point along the range of the values of a variable, we call these estimates point estimates. Essentially, all estimates that are expressed as a single number are point estimates that represent our best guesses about the corresponding population parameters.

It’s not just the measures of location, however: Measures of spread (SD, \(\sigma^2\), etc.) are also point estimates, as they give us a single number that represents our estimate of the variability in the population.

There exist point estimates even for things like relationships between variables. The correlation coefficient r takes values from −1 to 1, where 0 is no relationship, 1 is a perfect positive relationship and −1 is a perfect negative relationship (see Figure 1).

df <- tibble(x = rnorm(100), y1 = rnorm(100), y2 = x + rnorm(100))
# values of r nicely formatted
r1 <- signs::signs(cor(df$x, df$y1), trim_leading_zeros = T, accuracy=.001)
r2 <- signs::signs(cor(df$x, df$y2), trim_leading_zeros = T, accuracy=.001)
# plots
p1 <- df %>% ggplot(aes(x, y1)) + geom_point(color = theme_col)
p2 <- df %>% ggplot(aes(x, y2)) + geom_point(color = theme_col)
# arrange plots and add nice titles
cowplot::plot_grid(p1 + ggtitle(paste0("*r* = ", r1)),
                   p2 + ggtitle(paste0("*r* = ", r2)))

Figure 1: Relationships between variables as single numbers: Left-hand-side plot shows no relationship (r almost 0) and right-hand-side plot shows a strong positive relationship between variables plotted on the two axes.

Accuracy and uncertainty

So to reiterate, a point estimate, such as the sample mean (\(\bar{x}\)) is the best estimate (\(\hat{\mu}\)) of population parameter, in this case \(\mu\). However, as we saw in lecture 1, the mean of almost any sample will differ from the mean of the population that the sample is drawn from. The same is true for any point estimate!

This is where the standard error (SE) enters the game. The SE of a parameter expresses the uncertainty about the estimate of that population parameter. Although we’ve only been talking about the standard error of the mean (SEM), it can be calculated for all point estimates, not just the mean.

Because SE gives us a kind of margin of error, we can use it to quantify uncertainty around point estimates and calculate so-called interval estimates.

Interval estimates

In addition to estimating a single value (i.e., a point estimate), we can use SE to estimate an interval around it. For instance, we can estimate \(\hat{\mu}\) based on our sample as having the value of 4.13 with an interval from, say, −0.2 to 8.46. Thus, by putting a range on the value of a point estimate, we communicate the uncertainty around it. The wider the interval, the more uncertainty we acknowledge with respect to our estimate.

There are different kinds of interval estimates appropriate for different scenarios, situations, and statistical methods. One of them, the confidence interval, is going to be of special interest to us.

Confidence interval

We can use SE – the standard deviation of the sampling distribution – to calculate a confidence interval (CI) with a certain coverage, e.g., 90%, 95%, 99%, around our point estimate. Let’s unpack what coverage is using an example.

Imagine we bought a bag of Minstrels with 200 chocolates inside. We can use the mean weight of a single Minstrel in sample of N = 200 to estimate \(\mu\), the population mean weight of a Minstrel.

First of all, we weight up the bag and record the weight in grammes. Then we divide the number by 200 to get the sample mean weight, let’s say \(\bar{x}\) = 4.11g. This is our best guess about the value of the population mean, our point estimate \(\hat{\mu}\)

Let’s now, for the sake of the example, assume that we know the parameters of the population of Minstrel weight: \(\mu\) = 4, \(\sigma\) = 0.56.

We can apply the ±1.96 × SE range around our \(\hat{\mu}\) to get a confidence interval with a 95% coverage:

\[\begin{aligned}95\%\text{CI} &= \bar{x}\pm1.96\times SE\\&= \bar{x}\pm1.96\times 0.04\\&=4.11 \pm 0.0784\\&=[4.0316, 4.1884]\end{aligned}\] OK so we have our confidence interval with a 95% coverage (95% CI) but what does a 95% coverage actually mean?

This means that if we got a hundred bags of Minstrels (with 200 pieces in each bag), calculated the mean weight of each one and constructed a 95% CI around each sample mean, we would expect 95 of these 100 confidence intervals to include the true value of the population mean, \(\mu\) = 4.

Now, as it happens, in our particular example, the interval does not include 4. It’s one of the 5% that “got it wrong”. Aren’t we lucky!

Let’s illustrate coverage using another example.

This time, we’re sampling bags of oranges, with 30 oranges in each bag. In the picture below, the cluster of circles represents the population of oranges. As you can see, they have different sizes. Again, we’re assuming that we know the true mean size of an orange, \(\mu\) = 10.7, as shown by the vertical red line in the right-hand-side panel.

Let’s sample our first bag of 30 oranges. Click/tap the picture once.

OK, the highlighted circles are our sampled oranges. Their sizes are plotted on a histogram below. Click/tap again…

Now, the mean size in the sample – our \(\hat{\mu}\) – along with a 95% CI is shown on the right-hand-side panel. You can see that the interval intersects the vertical red line. That means that this particular 95% CI around our estimate does indeed include the value of the population parameter.

Click again to start a resampling animation that will go on until we have 100 estimates with 100 CIs.

Right, so as you can see 5/100 CIs did not include \(\mu\) (did not intersect the vertical red line). So, in this case, our simulation is perfectly in line with the definition of coverage: 95% of the confidence intervals include the population mean, while the remaining 5% do not!

The animation below illustrates how a 95% confidence interval is constructed. If you were following the explanation above, this should make sense.

Putting the CI around our sample estimate is actually very easy if we know the exact specification of the sampling distribution. Click on the picture to draw the sampling distribution.

The green area represents the inner 95% of the sampling distribution. Because this distribution is normal (as per CLT), the borders of the green area lie at ±1.96 SE from the centre of the distribution. Click on the picture again to visualise this relationship.

We can use exactly this distance on each side of any given sample mean to construct the 95% CI. Just a few more clicks see this.

So there you have it, easy if we know the exact specification of the sampling distribution. But you can probably tell by the repeated emphasis that there’s a catch…

In this case, the catch is that the sampling distribution is never known. We will never know its mean (true population mean), nor will we ever know its SD (SE), which we need to calculate the range of the interval. The reason why we don’t know SE is that it is calculated using population standard deviation \(\sigma\), which, like any population parameter, we cannot measure.

And because of this, we have to approximate the sampling distribution using the t distribution. Before we can move on to discussing how this approximation is done, we need to talk about the t distribution a little.

OK, now that we know some basics about the t distribution, let’s go back to CIs.

We said that 95% CI around estimated population mean is mean ±1.96 SE. This is absolutely true, but the problem is that we don’t know the value of SE because it’s calculated from population standard deviation \(\sigma\), which is also unknown. What we do instead is calculate the estimate of SE, \(\widehat{SE}\), from the sample standard deviation s.

Because of this one detail, we cannot use the normal distribution and its ±1.96 critical value. Rather, we approximate the shape of the sampling distribution using the t distribution. And because the shape of the t distribution changes as a function of df (and therefore sample size), the critical values for cutting off the outer 5% of the area change too. As a result, we need to replace the 1.96 with the appropriate critical value for the t distribution with a given number of dfs.

In our example with oranges, for our bag of N = 30, the critical value is t_crit(df=29) = ±2.045.

qt(p = 0.975, df = 29)

[1] 2.04523

So, a 95% CI around the mean for a sample of 30 is \(\bar{x}\) ±2.05 × \(\widehat{SE}\).

If you would like to know the full explanation for why we need to approxiamte the sampling distribution via the t distribution, check out the “Extra” box below.

To reiterate, here’s the full calculation for our bag of 200 minstrels with mean = 4.11 and s = 0.54:

First, the estimate of SE is

\[\begin{aligned}\widehat{SE}&=\frac{s}{\sqrt{N}}\\&=\frac{0.54}{\sqrt{200}}\\&\approx0.038\end{aligned}\] Now, what we have our estimate of SE (ass opposed to the true SE of 0.04), let’s plug it in along with our sample mean and the correct critical value of t to get the 95% CI:

\[\begin{aligned}95\%\ CI &= \bar{x}\pm t(N-1)_{crit}\times \widehat{SE}\\&=4.11\pm1.972\times0.038\\&=4.11\pm0.075\\&=[4.035, 4.185]\end{aligned}\]

Interpreting confidence intervals

Ok so now that you know how CIs are made, let’s go over what they actually are once again. They are tricky beast them!

It’s important to remember that the width of the interval tells us about how much we can expect the mean of a different sample of the same size to vary from the one we got. Just compare the two Minstrel examples and you’ll see that each of the means lies within the confidence interval of the other.

Moreover, there’s a x % chance that any given x % CI contains the true population mean. In our first example, we got a little unlucky as our CI, despite having a 95% coverage, missed the population mean by a fraction. However, the other one did contain the value of \(\mu\).

Frustratingly, we can never know whether our particular CI does or doesn’t contain the value of \(\mu\). That’s just one of the uncertainties inherent in statistics. Don’t worry, you’ll get used to the discomfort it causes…

Finally, remember that CIs can be calculated for any point estimate, not just the mean! Remember our Figure 1 with r coefficients representing the relationships between two variables? Well, we can put CIs around them too:

# Getting the plot titles right is rather involved
# and probably not worth the effort
r1 <- cor.test(df$x, df$y1)
r1_coef <- signs::signs(r1$estimate, trim_leading_zeros = T, accuracy=.001)
r1 <- paste0(
  r1_coef, "; 95% CI [",
  paste0(
    signs::signs(r1$conf.int, trim_leading_zeros = T, accuracy=.001),
    collapse = ", "),
  "]")
r2 <- cor.test(df$x, df$y2)
r2_coef <- signs::signs(r2$estimate, trim_leading_zeros = T, accuracy=.001)
r2 <- paste0(
  r2_coef, "; 95% CI [",
  paste0(
    signs::signs(r2$conf.int, trim_leading_zeros = T, accuracy=.001),
    collapse = ", "),
  "]")

# p1 and p2 are the same as before
cowplot::plot_grid(p1 + ggtitle(paste0("*r* = ", r1)),
                   p2 + ggtitle(paste0("*r* = ", r2)))

With this level of understanding of samples, populations, sampling distributions, standard error, and confidence intervals, you are now more than ready to learn about statistical testing.

But that’s a story for another day…