 |
 |
 |
# Run this cell to set up packages for lecture.
from lec01_imports import *
| |
|
|
In addition, we have many other course staff members who are here to support you in discussion, office hours, and online.
Learn more about them at dsc10.com/staff, and come say hi in office hours!
Everyone seems to have their own definition of data science.Data science is about drawing useful conclusions from data using computation. Throughout the quarter, we'll touch on several aspects of data science:
The site The Pudding is home to several interactive data-rich articles.
(source)Always refer to the course website for the current schedule, but here is a general idea:
| Sunday | Monday | Tuesday | Wednesday | Thursday | Friday | Saturday |
|---|---|---|---|---|---|---|
| Lecture | Lecture* | Lecture | ||||
| Discussion | Quiz | |||||
| Homework due | Lab due |
*No live lecture at 1PM on Wednesdays. Come to the 9AM or 10AM lectures, or watch the podcast recording.
We will have four quizzes and two exams this quarter. All will be conducted in person and on paper.
*We will have a 1PM Wednesday class for the midterm only.
Let us know of any conflicts and select a time slot for quizzes on the Welcome Survey.
Your first task is to complete the following by Sunday, September 29th at 11:59PM.
After that, start working on Lab 0, which is due Thursday, October 3rd at 11:59PM.
This is a tough, fast-paced course, but we're here to help you – here's how:
At the end of each quarter, we ask DSC 10 students to give advice to future students in the course. Here are some responses from last quarter's students:
Start the assignments (especially the midterm/final projects) early! It became so manageable with more time to split up sections and think things through without a crazy overbearing time pressure.
Be prepared to spend a lot of time in this class, regardless of whether you have any prior knowledge in programming or statistics. Everything is doable, but you will need to put in a significant amount of effort to succeed and sometimes you'll have to think outside of the box to come up with solutions.
Go to office hours!! It is the best resource available. The tutors are more than willing to help you out. The tutors made my time at DSC 10 not only manageable but also enjoyable. Also, prepare for the quizzes at least one day in advance so that you can retain the material better.
Practice is the most important thing you can do to succeed in this course. Also, grab a friend - two (or more) heads are better than one! And don't be afraid to ask for help when needed.
Regardless of your background, you can succeed in this course with lots of hard work. No prior programming or statistics experience will be assumed! We'll start at the beginning, but we will move fast!
Inspirational TED talk: 🎥 We’re All Data Scientists by Rebecca Nugent.
# Read in 'lw.txt' to a variable called little_women_text.
little_women_text = open('data/lw.txt').read()
# See the first three thousand characters.
little_women_text[:3000]
'The Project Gutenberg EBook of Little Women, by Louisa May Alcott\n\nThis eBook is for the use of anyone anywhere at no cost and with\nalmost no restrictions whatsoever. You may copy it, give it away or\nre-use it under the terms of the Project Gutenberg License included\nwith this eBook or online at www.gutenberg.net\n\n\nTitle: Little Women\n\nAuthor: Louisa May Alcott\n\nPosting Date: September 13, 2008 [EBook #514]\nRelease Date: May, 1996\n[This file last updated on August 19, 2010]\n\nLanguage: English\n\n\n*** START OF THIS PROJECT GUTENBERG EBOOK LITTLE WOMEN ***\n\n\n\n\nLITTLE WOMEN\n\n\nby\n\nLouisa May Alcott\n\n\n\n\nCONTENTS\n\n\nPART 1\n\n ONE PLAYING PILGRIMS\n TWO A MERRY CHRISTMAS\n THREE THE LAURENCE BOY\n FOUR BURDENS\n FIVE BEING NEIGHBORLY\n SIX BETH FINDS THE PALACE BEAUTIFUL\n SEVEN AMY\'S VALLEY OF HUMILIATION\n EIGHT JO MEETS APOLLYON\n NINE MEG GOES TO VANITY FAIR\n TEN THE P.C. AND P.O.\n ELEVEN EXPERIMENTS\n TWELVE CAMP LAURENCE\n THIRTEEN CASTLES IN THE AIR\n FOURTEEN SECRETS\n FIFTEEN A TELEGRAM\n SIXTEEN LETTERS\n SEVENTEEN LITTLE FAITHFUL\n EIGHTEEN DARK DAYS\n NINETEEN AMY\'S WILL\n TWENTY CONFIDENTIAL\n TWENTY-ONE LAURIE MAKES MISCHIEF, AND JO MAKES PEACE\n TWENTY-TWO PLEASANT MEADOWS\n TWENTY-THREE AUNT MARCH SETTLES THE QUESTION\n\n\nPART 2\n\n TWENTY-FOUR GOSSIP\n TWENTY-FIVE THE FIRST WEDDING\n TWENTY-SIX ARTISTIC ATTEMPTS\n TWENTY-SEVEN LITERARY LESSONS\n TWENTY-EIGHT DOMESTIC EXPERIENCES\n TWENTY-NINE CALLS\n THIRTY CONSEQUENCES\n THIRTY-ONE OUR FOREIGN CORRESPONDENT\n THIRTY-TWO TENDER TROUBLES\n THIRTY-THREE JO\'S JOURNAL\n THIRTY-FOUR FRIEND\n THIRTY-FIVE HEARTACHE\n THIRTY-SIX BETH\'S SECRET\n THIRTY-SEVEN NEW IMPRESSIONS\n THIRTY-EIGHT ON THE SHELF\n THIRTY-NINE LAZY LAURENCE\n FORTY THE VALLEY OF THE SHADOW\n FORTY-ONE LEARNING TO FORGET\n FORTY-TWO ALL ALONE\n FORTY-THREE SURPRISES\n FORTY-FOUR MY LORD AND LADY\n FORTY-FIVE DAISY AND DEMI\n FORTY-SIX UNDER THE UMBRELLA\n FORTY-SEVEN HARVEST TIME\n\n\n\nCHAPTER ONE\n\nPLAYING PILGRIMS\n\n"Christmas won\'t be Christmas without any presents," grumbled Jo, lying\non the rug.\n\n"It\'s so dreadful to be poor!" sighed Meg, looking down at her old\ndress.\n\n"I don\'t think it\'s fair for some girls to have plenty of pretty\nthings, and other girls nothing at all," added little Amy, with an\ninjured sniff.\n\n"We\'ve got Father and Mother, and each other," said Beth contentedly\nfrom her corner.\n\nThe four young faces on which the firelight shone brightened at the\ncheerful words, but darkened again as Jo said sadly, "We haven\'t got\nFather, and shall not have him for a long time." She didn\'t say\n"perhaps never," but each silently added it, thinking of Father far\naway, where the fighting was.\n\nNobody spoke for a minute; then Meg said in an altered tone, "You know\nthe reason Mother proposed not having any presents this Christmas was\nbecause it is going to b'
# Print the first three thousand characters.
print(little_women_text[:3000])
The Project Gutenberg EBook of Little Women, by Louisa May Alcott
This eBook is for the use of anyone anywhere at no cost and with
almost no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included
with this eBook or online at www.gutenberg.net
Title: Little Women
Author: Louisa May Alcott
Posting Date: September 13, 2008 [EBook #514]
Release Date: May, 1996
[This file last updated on August 19, 2010]
Language: English
*** START OF THIS PROJECT GUTENBERG EBOOK LITTLE WOMEN ***
LITTLE WOMEN
by
Louisa May Alcott
CONTENTS
PART 1
ONE PLAYING PILGRIMS
TWO A MERRY CHRISTMAS
THREE THE LAURENCE BOY
FOUR BURDENS
FIVE BEING NEIGHBORLY
SIX BETH FINDS THE PALACE BEAUTIFUL
SEVEN AMY'S VALLEY OF HUMILIATION
EIGHT JO MEETS APOLLYON
NINE MEG GOES TO VANITY FAIR
TEN THE P.C. AND P.O.
ELEVEN EXPERIMENTS
TWELVE CAMP LAURENCE
THIRTEEN CASTLES IN THE AIR
FOURTEEN SECRETS
FIFTEEN A TELEGRAM
SIXTEEN LETTERS
SEVENTEEN LITTLE FAITHFUL
EIGHTEEN DARK DAYS
NINETEEN AMY'S WILL
TWENTY CONFIDENTIAL
TWENTY-ONE LAURIE MAKES MISCHIEF, AND JO MAKES PEACE
TWENTY-TWO PLEASANT MEADOWS
TWENTY-THREE AUNT MARCH SETTLES THE QUESTION
PART 2
TWENTY-FOUR GOSSIP
TWENTY-FIVE THE FIRST WEDDING
TWENTY-SIX ARTISTIC ATTEMPTS
TWENTY-SEVEN LITERARY LESSONS
TWENTY-EIGHT DOMESTIC EXPERIENCES
TWENTY-NINE CALLS
THIRTY CONSEQUENCES
THIRTY-ONE OUR FOREIGN CORRESPONDENT
THIRTY-TWO TENDER TROUBLES
THIRTY-THREE JO'S JOURNAL
THIRTY-FOUR FRIEND
THIRTY-FIVE HEARTACHE
THIRTY-SIX BETH'S SECRET
THIRTY-SEVEN NEW IMPRESSIONS
THIRTY-EIGHT ON THE SHELF
THIRTY-NINE LAZY LAURENCE
FORTY THE VALLEY OF THE SHADOW
FORTY-ONE LEARNING TO FORGET
FORTY-TWO ALL ALONE
FORTY-THREE SURPRISES
FORTY-FOUR MY LORD AND LADY
FORTY-FIVE DAISY AND DEMI
FORTY-SIX UNDER THE UMBRELLA
FORTY-SEVEN HARVEST TIME
CHAPTER ONE
PLAYING PILGRIMS
"Christmas won't be Christmas without any presents," grumbled Jo, lying
on the rug.
"It's so dreadful to be poor!" sighed Meg, looking down at her old
dress.
"I don't think it's fair for some girls to have plenty of pretty
things, and other girls nothing at all," added little Amy, with an
injured sniff.
"We've got Father and Mother, and each other," said Beth contentedly
from her corner.
The four young faces on which the firelight shone brightened at the
cheerful words, but darkened again as Jo said sadly, "We haven't got
Father, and shall not have him for a long time." She didn't say
"perhaps never," but each silently added it, thinking of Father far
away, where the fighting was.
Nobody spoke for a minute; then Meg said in an altered tone, "You know
the reason Mother proposed not having any presents this Christmas was
because it is going to b
# Create a variable "chapters" by splitting the text on 'CHAPTER '.
chapters = little_women_text.split('CHAPTER ')
# Create a DataFrame with one column - the text of each chapters.
bpd.DataFrame().assign(chapters=chapters)
| chapters | |
|---|---|
| 0 | The Project Gutenberg EBook of Little Women, b... |
| 1 | ONE\n\nPLAYING PILGRIMS\n\n"Christmas won't be... |
| 2 | TWO\n\nA MERRY CHRISTMAS\n\nJo was the first t... |
| 3 | THREE\n\nTHE LAURENCE BOY\n\n"Jo! Jo! Where ... |
| 4 | FOUR\n\nBURDENS\n\n"Oh, dear, how hard it does... |
| ... | ... |
| 43 | FORTY-THREE\n\nSURPRISES\n\nJo was alone in th... |
| 44 | FORTY-FOUR\n\nMY LORD AND LADY\n\n"Please, Mad... |
| 45 | FORTY-FIVE\n\nDAISY AND DEMI\n\nI cannot feel ... |
| 46 | FORTY-SIX\n\nUNDER THE UMBRELLA\n\nWhile Lauri... |
| 47 | FORTY-SEVEN\n\nHARVEST TIME\n\nFor a year Jo a... |
48 rows × 1 columns
# Number of occurrences of each name in each chapter.
counts = bpd.DataFrame().assign(
Amy=np.char.count(chapters, 'Amy'),
Beth=np.char.count(chapters, 'Beth'),
Jo=np.char.count(chapters, 'Jo'),
Meg=np.char.count(chapters, 'Meg'),
Laurie=np.char.count(chapters, 'Laurie'),
)
counts
| Amy | Beth | Jo | Meg | Laurie | |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 23 | 26 | 44 | 26 | 0 |
| 2 | 13 | 12 | 21 | 20 | 0 |
| 3 | 2 | 2 | 62 | 36 | 16 |
| 4 | 14 | 18 | 34 | 17 | 0 |
| ... | ... | ... | ... | ... | ... |
| 43 | 31 | 8 | 61 | 3 | 29 |
| 44 | 13 | 0 | 9 | 0 | 10 |
| 45 | 1 | 2 | 6 | 2 | 0 |
| 46 | 2 | 1 | 56 | 4 | 2 |
| 47 | 10 | 3 | 37 | 6 | 13 |
48 rows × 5 columns
# Cumulative number of times each name appears.
cumulative_counts = bpd.DataFrame().assign(
Amy=np.cumsum(counts.get('Amy')),
Beth=np.cumsum(counts.get('Beth')),
Jo=np.cumsum(counts.get('Jo')),
Meg=np.cumsum(counts.get('Meg')),
Laurie=np.cumsum(counts.get('Laurie')),
Chapter=np.arange(1, 49, 1)
)
cumulative_counts
| Amy | Beth | Jo | Meg | Laurie | Chapter | |
|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| 1 | 23 | 26 | 44 | 26 | 0 | 2 |
| 2 | 36 | 38 | 65 | 46 | 0 | 3 |
| 3 | 38 | 40 | 127 | 82 | 16 | 4 |
| 4 | 52 | 58 | 161 | 99 | 16 | 5 |
| ... | ... | ... | ... | ... | ... | ... |
| 43 | 619 | 459 | 1435 | 673 | 571 | 44 |
| 44 | 632 | 459 | 1444 | 673 | 581 | 45 |
| 45 | 633 | 461 | 1450 | 675 | 581 | 46 |
| 46 | 635 | 462 | 1506 | 679 | 583 | 47 |
| 47 | 645 | 465 | 1543 | 685 | 596 | 48 |
48 rows × 6 columns
# Putting it all together, we get a helpful visualization.
cumulative_counts_df = cumulative_counts.drop(columns=['Chapter']).to_df().melt().rename(columns={'variable': 'name', 'value': 'Count'})
cumulative_counts_df = cumulative_counts_df.assign(Chapter=list(range(1, 49)) * 5)
px.line(cumulative_counts_df, x='Chapter', y='Count', color='name', width=900, height=600, title='Cumulative Number of Times Each Name Appears', template='ggplot2')
This button!shift + enter (or shift + return) on your keyboard (strongly preferred), or23
23
-15 + 2.718
-12.282
4 ** 3
64
(2 + 3 + 4) / 3
3.0
# Only one value is displayed. Why?
9 + 10
13 / 4
21
21
| Operation | Operator | Example | Value |
|---|---|---|---|
| Addition | + |
2 + 3 |
5 |
| Subtraction | - |
2 - 3 |
-1 |
| Multiplication | * |
2 * 3 |
6 |
| Division | / |
7 / 3 |
2.66667 |
| Remainder | % |
7 % 3 |
1 |
| Exponentiation | ** |
2 ** 0.5 |
1.41421 |
5 * 2 ** 3
40
(5 * 2) ** 3
1000
In the cell below, write an expression that's equivalent to
$$(19 + 6 \cdot 3) - 15 \cdot \left(\sqrt{100} \cdot \frac{1}{30}\right) \cdot \frac{3}{5} + \frac{4^2}{2^3} + \left( 6 - \frac{2}{3} \right) \cdot 12 $$Try to use parentheses only when necessary.
max, min, and round.