# Lecture 20 – Confidence Intervals, Center and Spread¶

## DSC 10, Fall 2022¶

### Announcements¶

• Friday is a holiday 🎖️, so there is no lecture. Check the calendar for the latest office hours schedule.
• Lab 6 is due Saturday 11/12 at 11:59pm.
• Homework 5 is due Tuesday 11/15 at 11:59pm.
• We'll be releasing the Final Project later today.
• Get a partner lined up! You don't have to work with your partner from the Midterm Project.
• The rules for working with a partner have changed.
• Come hang out with your instructors:
• Tonight from 5-6pm in CSE 4140, come watch Suraj participate in "Professor Talks: Spicy Challenge 🌶" with faculty from CSE. See more details here.
• On Tuesday 11/15 from 10-11am in the SDSC Auditorium, come talk to Janine, Suraj, and other HDSI faculty at the HDSI faculty/student mixer!

### Agenda¶

• Interpreting confidence intervals.
• Confidence intervals for hypothesis testing.
• Mean and median.
• Standard deviation.

## Interpreting confidence intervals¶

### Recap: City of San Diego employee salaries¶

Let's rerun our code from last time to compute a 95% confidence interval for the median salary of all San Diego city employees, based on a sample of 500 people.

Step 1: Collect a single sample of size 500 from the population.

Step 2: Bootstrap! That is, resample from the sample a large number of times, and each time, compute the median of the resample. This will generate an empirical distribution of the sample median.

Step 3: Take the middle 95% of the empirical distribution of sample medians (i.e. boot_medians). This creates our 95% confidence interval.

We think the population median is close to our sample median, \$72,016. We can say: A 95% confidence interval for the population median is \$66,987 to \\$76,527. Today, we'll address: What does 95% confidence mean? What are we confident about? Is this technique always "good"? ### Interpreting confidence intervals¶ • We create a confidence interval such that 95% of our bootstrap medians fell within this interval. • We're pretty confident that the true population median does, too. • How confident should we be about this? What does a 95% confidence interval mean? ### Capturing the true value¶ • Consider the process of: 1. Collecting a new original sample from the population, 2. Bootstrap resampling from it many times, computing the statistic (e.g. median) in each resample, and 3. Constructing a new 95% confidence interval. • A 95% confidence level means that approximately 95% of the time, the intervals you create through this process will contain the true population parameter. • The confidence is in the process that generates the interval. ### Many confidence intervals¶ • We repeated the process outlined on the previous slide 200 times, to come up with 200 confidence intervals. • We did this in advance (it took a really long time) and saved the results to a file. • The resulting CIs are stored in the array many_cis below. In the visualization below, • The blue line represents the population parameter. This is not random. • Each gold line represents a separate confidence interval, created using the specified process. • Most of these confidence intervals contain the true parameter – but not all! ### Which confidence intervals don't contain the true parameter?¶ • 11 of our 200 confidence intervals didn't contain the true parameter. • That means 189/200, or 94.5% of them, did. • This is pretty close to 95%! • In reality, you will have only one confidence interval, and no way of knowing if it contains the true parameter, since you have no way of knowing if your one original sample is good. ### Confidence tradeoffs¶ • When we use an "unlucky" sample to make our CI, the CI won't contain the population parameter. • If we choose a 99% confidence level, • only about 1% of our samples will be "unlucky", but • our intervals will be very wide and thus not that useful. • If we choose an 80% confidence level, • more of our samples will be "unlucky", but • our intervals will be narrower and thus more precise. • At a fixed confidence level, how do we decrease the width of a confidence interval? • By taking a bigger sample! • We'll soon see how sample sizes, confidence levels, and CI widths are related to one another. ### Misinterpreting confidence intervals¶ Confidence intervals can be hard to interpret. Does this interval contain 95% of all salaries? No! However, this interval does contain 95% of all bootstrapped median salaries. Is there is a 95% chance that this interval contains the population parameter? No! Why not? • The population parameter is fixed. In reality, we might not know it, but it is not random. • The interval above is not random, either (but the process that creates it is). • For a given interval, the population parameter is in the interval, or it is not. There is no randomness involved. • Remember that the 95% confidence is in the process that created an interval. • This process (sampling, then bootstrapping, then creating an interval) creates a good interval roughly 95% of the time. ### Bootstrap rules of thumb¶ • The bootstrap is an awesome tool! We only had to collect a single sample from the population to get the (approximate) distribution of the sample median. • But it has limitations: • It is not good for sensitive statistics, like the max or min. • It only gives useful results if the sample is a good approximation of population. • If our original sample was not representative of the population, the resulting bootstrapped samples will also not be representative of the population. ### Example: Estimating the max of a population¶ • Suppose we want to estimate the maximum salary of all San Diego city employees, given just a single sample, my_sample. • Our estimate will be the max in the sample. This is a statistic. • To get the empirical distribution of this statistic, we bootstrap: ### Visualize¶ Since we have access to the population, we can find the population maximum directly, without bootstrapping. Does the population maximum lie within the bulk of the bootstrapped distribution? No, the bootstrapped distribution doesn't capture the population maximum (blue dot) of \$359,138. Why not? 🤔

• The largest value in our original sample was \$329,949. • Therefore, the largest value in any bootstrapped sample is at most \$329,949.
• Generally, the bootstrap works better for measures of central tendency or variation (means, medians, variances) than it does for extreme values (maxes and mins).

## Confidence intervals for hypothesis testing¶

### Using a confidence interval for hypothesis testing¶

It turns out that we can use bootstrapped confidence intervals for hypothesis testing!

• Null Hypothesis: The population parameter is equal to some value, $x$.
• Alternative Hypothesis: The population parameter is not equal to $x$.
• Cutoff for p-value: p%.
• Strategy:
• Construct a (100-p)% confidence interval for the population parameter.
• If $x$ is not in the interval, we reject the null hypothesis.
• If $x$ is in the interval, we fail to reject the null hypothesis (since our results are consistent with the null).

### Setting up a hypothesis test¶

• Suppose we only have access to a sample of 300 Fire-Rescue Department workers.
• We want to understand the median salary of all Fire-Rescue Department workers.