# Lecture 19 – Bootstrapping, Percentiles, and Confidence Intervals¶

## DSC 10, Spring 2023¶

### Announcements¶

• Lab 5 is due on Saturday 5/20 at 11:59PM.
• Homework 5 is due on Tuesday 5/23 at 11:59PM.
• The Final Project will be released later this week. It's on Meteorite landings ☄️.
• You can work with a different partner if you'd like.
• It won't be due until Tuesday 6/6 – you'll have quite a bit of time to work on it, but it is longer than the Midterm Project.
• The Diagrams page on the course website has been updated with all of the most recent interactive diagrams from lecture.

### Agenda¶

• Bootstrapping.
• Percentiles.
• Confidence intervals.

### Resources¶

• You may have noticed that we've quickly moved into much more theoretical material.

• Remember to read the textbook for more context and examples.

• Additionally, this site contains a helpful visual explanation of permutation testing, and the Diagrams page contains all of the interactive diagrams we've seen so far in lecture.

## Bootstrapping 🥾¶

### City of San Diego employee salary data¶

All City of San Diego employee salary data is public. We are using the latest available data.

We only need the 'TotalWages' column, so let's get just that column.

### The median salary¶

• We can use .median() to find the median salary of all city employees.
• This is not a random quantity.

### Let's be realistic...¶

• In practice, it is costly and time-consuming to survey all 12,000+ employees.
• More generally, we can't expect to survey all members of the population we care about.
• Instead, we gather salaries for a random sample of, say, 500 people.
• Hopefully, the median of the sample is close to the median of the population.

### The sample median¶

Let's survey 500 employees at random. To do so, we can use the .sample method.

We won't reassign my_sample at any point in this notebook, so it will always refer to this particular sample.

### How confident are we that this is a good estimate?¶

• Our estimate depended on a random sample.
• If our sample was different, our estimate may have been different, too.
• How different could the sample median have been – that is, what is the distribution of the sample median?
• "Narrow" distribution $\Rightarrow$ not too different.
• "Wide" distribution $\Rightarrow$ quite different.
• Our confidence in the estimate depends on the answer to this question.

### An impractical approach¶

• One idea: repeatedly collect random samples of 500 from the population and compute their medians.
• This is what we did in Lecture 13 to compute an empirical distribution of the sample mean of flight delays.
• This shows an empirical distribution of the sample median. It is an approximation of the true probability distribution of the sample median, based on 1000 samples.

### The problem¶

• Drawing new samples like this is impractical – we usually can't just ask for new samples from the population.
• If we were able to do this, why not just collect more data in the first place?
• Key insight: our original sample, my_sample, looks a lot like the population.
• Their distributions are similar.

Note that unlike the previous histogram we saw, this is depicting the distribution of the population and of one particular sample (my_sample), not the distribution of sample medians for 1000 samples.

### Bootstrapping¶

• Shortcut: Use the sample in lieu of the population.
• The sample itself looks like the population.
• So, resampling from the sample is kind of like sampling from the population.
• The act of resampling from a sample is called bootstrapping.
• In our case specifically:
• We have a sample of 500 salaries.
• We want another sample of 500 salaries, but we can't draw from the population.
• However, the original sample looks like the population.
• So, let's just resample from the sample!

### To replace or not replace?¶

• Our goal when bootstrapping is to create a sample of the same size as our original sample.
• Let's repeatedly resample 3 numbers without replacement from an original sample of [1, 2, 3].
• Let's repeatedly resample 3 numbers with replacement from an original sample of [1, 2, 3].
• When we resample without replacement, resamples look just like the original samples.

• When we resample with replacement, resamples can have a different mean, median, max, and min than the original sample.

• So, we need to sample with replacement to ensure that our resamples can be different from the original sample.

### Bootstrapping the sample of salaries¶

We can simulate the act of collecting new samples by sampling with replacement from our original sample, my_sample.

### Bootstrap distribution of the sample median¶

The population median (blue dot) is near the middle.

In reality, we'd never get to see this!

### What's the point of bootstrapping?¶

We have a sample median wage:

• As such, we think the population median is close to \$72,016. However, we're not quite sure how close. • How do we capture our uncertainty about this guess? • 💡 Idea: Find a range that captures most (e.g. 95%) of the bootstrapped distribution of sample medians. ### Confidence intervals¶ Let's be a bit more precise. • Goal: Estimate an unknown population parameter. • We have been saying: We think the population parameter is close to our sample statistic,$x$. • We want to say: We think the population parameter is between$a$and$b$. • To do this, we'll use the bootstrapped distribution of a sample statistic to compute an interval that contains "the bulk" of the sample statistics. Such an interval is called a confidence interval. ### Finding endpoints¶ • We want to find two points,$x$and$y$, such that: • The area to the left of$x$in the bootstrapped distribution is about 2.5%. • The area to the right of$y$in the bootstrapped distribution is about 2.5%. • The interval$[x,y]$will contain about 95% of the total area, i.e. 95% of the total values. As such, we will call$[x, y]$a 95% confidence interval. •$x$and$y$are the 2.5th percentile and 97.5th percentile, respectively. ### Computing a confidence interval¶ You will use the code above very frequently moving forward! ### Visualizing our 95% confidence interval¶ • Let's draw the interval we just computed on the histogram. • 95% of the bootstrap medians fell into this interval. • In this case, our 95% confidence interval (gold line) contains the true population parameter (blue dot). • It won't always, because you might have a bad original sample! • In reality, you won't know where the population parameter is, and so you won't know if your confidence interval contains it. • Note that the histogram is not centered around the population median (\$74,441), but it is centered around the sample median (\\$72,016). ### Concept Check ✅ – Answer at cc.dsc10.com¶ We computed the following 95% confidence interval: If we instead computed an 80% confidence interval, would it be wider or narrower? A. Wider B. Narrower C. Impossible to tell </center ### Reflection¶ Now, instead of saying We think the population median is close to our sample median, \$72,016.

We can say:

A 95% confidence interval for the population median is \$66,996 to \\$76,527.

Some lingering questions: What does 95% confidence mean? What are we confident about? Is this technique always "good"?

## Summary, next time¶

### Summary¶

• By bootstrapping a single sample, we can generate an empirical distribution of a sample statistic. This distribution gives us a sense of how different the sample statistic could have been if we had collected a different original sample.
• The $p$th percentile of a collection is the smallest value in the collection that is at least as large as $p$% of all the values.
• After using the bootstrap to generate the empirical distribution of a sample statistic, we can create a $c$% confidence interval by taking the middle $c$% of values of the bootstrapped distribution.
• Such an interval allows us to quantify the uncertainty in our estimate of a population parameter.
• Instead of providing just a single estimate of a population parameter, e.g. \$72,016, we can provide a range of estimates, e.g. \\$66,996 to \\$76,527..
• Confidence intervals are used in a variety of fields to capture uncertainty. For instance, political researchers create confidence intervals for the proportion of votes their favorite candidate will receive, given a poll of voters.

### Next time¶

We will:

• Give more context to what the confidence level of a confidence interval means.
• Look at statistics for which the bootstrap doesn't work well.
• Use confidence intervals for hypothesis testing.
• Start looking at measures of central tendency (mean, median, standard deviation).