Free Data Science learning guide

Data Science with Python: A Beginner's Guide

Name: Data Science with Python: A Beginner's Guide
Author: Tanmay Kalbande

Data Science with Python: A Beginner's Guide — a free beginner-level guide covering data science with python for beginners. Learn with clear...

79 min read9 chaptersbeginner

What you will learn

1. Introduction to Data Science and Python
2. Python Programming Fundamentals
3. Working with Data: NumPy and Pandas Basics
4. Data Visualization with Matplotlib & Seaborn
5. Data Cleaning and Preparation
6. Exploratory Data Analysis (EDA)
7. Introduction to Statistical Concepts
8. Machine Learning Foundations with Scikit‑Learn
9. End‑to‑End Mini Project

1. 1. Introduction to Data Science and Python

What Is Data Science? Imagine you receive a massive spreadsheet from a retail chain that contains every transaction made in the last year—millions of rows, dozens of columns, and countless hidden patterns. Data science is the discipline that turns that raw, chaotic data into clear, actionable insights. It blends three core ingredients: | Ingredient | What It Means | |------------|---------------| | Domain Knowledge | Understanding the business, health, environment, etc., that the data comes from. | | Statistical Thinking | Using math to describe uncertainty, detect trends, and test hypotheses. | | Computational Tools | Writing code to clean, explore, model, and visualize data at scale. | When these ingredients are combined, data scientists can answer questions such as: Why did sales of winter coats spike in October? Which patients are at highest risk of readmission after surgery? What traffic patterns cause the longest commute times in a city? The output can be a simple report, an interactive dashboard, or a predictive model that runs automatically every day. Real‑World Applications | Field | Typical Problem | Data‑Science Solution | |-------|----------------|-----------------------| | Retail | Forecasting demand for seasonal items | Time‑series forecasting models that suggest inventory levels | | Healthcare | Identifying early signs of disease from electronic records | Classification algorithms that flag high‑risk patients | | Finance | Detecting fraudulent credit‑card transactions | Anomaly‑detection pipelines that raise alerts in real time | | Sports | Optimizing player line‑ups based on performance metrics | Clustering analysis that groups similar playing styles | | Public Policy | Measuring the impact of a new traffic law | Causal inference studies that compare before/after outcomes | These examples illustrate that data science is not a single task but a workflow: acquire data → clean it → explore it → model it → communicate findings. --- Why Python Is the Language of Choice Python is a high‑level, interpreted programming language. For a beginner, its most compelling qualities are: Readability – The syntax resembles plain English (if age 18:), reducing the learning curve. Rich Ecosystem – Libraries such as NumPy, pandas, Matplotlib, and scikit‑learn provide ready‑made tools for every stage of the data‑science pipeline. Community Support – Millions of developers contribute tutorials, Stack Overflow answers, and open‑source packages. Cross‑Platform – Runs on Windows, macOS, and Linux without modification. Integration – Works smoothly with databases, web services, and cloud platforms. Quick Comparison with Other Languages | Language | Typical Use | Strengths | Weaknesses for Beginners | |----------|-------------|----------|--------------------------| | R | Statistics & graphics | Built‑in statistical functions | Steeper syntax, less general‑purpose | | SQL | Database querying | Declarative, great for data extraction | Limited to data manipulation, not full analysis | | Java …

2. 2. Python Programming Fundamentals

A Real‑World Problem Starts with a Simple Script Imagine you have just received a spreadsheet from the marketing team that lists the daily number of units sold for a new product. The manager asks, “Did we ever sell fewer than 10 units in a day? And if so, on which dates?” You could scroll through the file manually, but a tiny Python script can answer the question in seconds, and the same skills will let you automate far more complex analyses later. This chapter equips you with the core Python syntax you need to write, run, and understand such scripts. --- 1. Writing and Running Python Statements 1.1 What Is a Statement? A statement is a line of code that tells Python to do something. The simplest statements are expressions—pieces of code that produce a value, such as 2 + 3. When you type an expression at the Python prompt, the interpreter evaluates it and prints the result. In a script (a file with a .py extension), you typically write many statements one after another. The interpreter executes them in order, from the top of the file to the bottom. 1.2 Running Code in Different Environments - Jupyter Notebook – Already introduced in Chapter 1, notebooks let you mix explanatory text, code cells, and output. They are ideal for exploratory work and quick testing. - Command‑Line / Terminal – You can run a script with python myscript.py. This is useful for larger programs or when you want to schedule tasks. - Integrated Development Environment (IDE) – Tools like VS Code or PyCharm provide features such as auto‑completion and debugging. Whichever environment you choose, the fundamental syntax remains the same. 1.3 Comments: Communicating With Future You Comments are pieces of text that Python ignores when executing a program. They are crucial for readability—especially when you return to code weeks later. - Single‑line comment starts with : - Multi‑line comment (often called a docstring when placed at the start of a module, class, or function) uses triple quotes: Good commenting practice: describe why something is done, not just what the code does. --- 2. Variables, Data Types, and Type Conversion 2.1 Variables: Naming Boxes for Data A variable is a name that points to a value stored in memory. Python creates the variable the first time you assign a value to it—no explicit declaration required. Naming rules (keep them in mind as you write code): - Must start with a letter (a–z, A–Z) or underscore (). - Can contain letters, numbers, and underscores. - Case‑sensitive (sales and Sales are different variables). 2.2 Core Data Types | Type | Description | Example | |------|-------------|---------| | int | Whole numbers (no decimal point) | …

3. 3. Working with Data: NumPy and Pandas Basics

A Real‑World Spark: Turning a CSV of Store Sales into Actionable Insights Imagine you just received a CSV file containing last month’s sales for a chain of 150 retail stores. The file lists store ID, date, product category, units sold, and revenue. Your manager asks: “Which stores are under‑performing, and by how much? Can we spot any seasonal patterns?” Answering these questions quickly requires two things: 1. A fast, memory‑efficient container for numeric calculations – this is where NumPy shines. 2. A flexible table‑like structure for mixed‑type data – this is the realm of pandas. In the sections that follow we’ll build the tools you need to: Create, index, and slice NumPy ndarrays. Perform element‑wise arithmetic and compute basic statistics with NumPy. Load a CSV file into a pandas DataFrame. Select, filter, and sort data using pandas methods. By the end of the chapter you’ll be able to turn a raw CSV into a clean, searchable data set ready for deeper analysis. --- Introducing NumPy Arrays What Is an ndarray? The core object in NumPy is the n‑dimensional array, abbreviated ndarray. Think of it as a grid of numbers that lives in a single block of memory, which makes operations on large collections of data extremely fast. Unlike Python’s built‑in list, an ndarray: Holds elements of the same data type (e.g., all float64). Provides vectorized operations – a single statement can act on every element simultaneously. Offers advanced indexing (slicing, boolean masks) that would require loops in vanilla Python. Creating ndarrays The most common way to create an array is with the function np.array(): Other handy constructors include: | Function | Description | |----------|-------------| | np.zeros(shape) | Array of all zeros | | np.ones(shape) | Array of all ones | | np.arange(start, stop, step) | Sequence of evenly spaced values | | np.random.rand(shape) | Random numbers from a uniform distribution | Example – creating a 2‑dimensional array of daily sales for a week: Indexing and Slicing Because an ndarray is a regular grid, you can retrieve individual elements or sub‑sections using square brackets. Single element: array[row, col] (zero‑based indexing). Row slice: array[start:stop] – returns rows start through stop‑1. Column slice: array[:, start:stop]. Example – extracting the sales for the third day: You can also use negative indices to count from the end: Element‑wise Arithmetic NumPy’s power lies in its ability to apply arithmetic operators to whole arrays without explicit loops. Common element‑wise operations: | Operator | Meaning | |----------|---------| | + / - | Add / subtract a scalar or another array of the same shape | | / / | Multiply / divide element‑wise | | | Exponentiation | | np.sqrt(array) | Square‑root of each element | Example – …

4. 4. Data Visualization with Matplotlib & Seaborn

A Real‑World Prompt: Turning Store Data into a Story Imagine you are the analyst for a small retail chain. Each month you receive a CSV file that contains: | month | category | sales | price | quantity | |------|----------|-------|-------|----------| | Jan | Shoes | 12 400| 45.99 | 270 | | Jan | Apparel | 9 800| 32.50 | 300 | | … | … | … | … | … | Your manager asks three questions: 1. How have total sales changed over the year? – a line plot will reveal trends. 2. Which product categories contributed the most each month? – a bar plot makes the comparison obvious. 3. Are there outliers in transaction amounts that we should investigate? – a box‑ or violin‑plot will show the distribution. Answering these questions requires visualizing data. This chapter walks you through creating the necessary plots with Matplotlib (the core Python plotting library) and Seaborn (a high‑level wrapper that makes statistical graphics easy). By the end, you will be able to generate, customize, and export professional‑looking figures. --- 1. Getting Started with Matplotlib 1.1 Importing the Library Matplotlib’s pyplot module mimics MATLAB’s plotting commands: each function creates or modifies a figure (the whole canvas) or an axes (the region that holds the data). 1.2 The Figure‑Axes Model - Figure – the outer container; a single figure can hold many axes. - Axes – the actual plot area (including the x‑ and y‑axis, tick marks, and data). When you call a plotting function without explicitly creating a figure, Matplotlib automatically makes a current figure and a current axes behind the scenes. 1.3 Inline Display in Jupyter Because the earlier chapters introduced Jupyter Notebook, you can see plots directly in a notebook cell: If you prefer an interactive window (useful for exploring large datasets), replace the line with %matplotlib notebook or %matplotlib widget. --- 2. Core Plot Types with Matplotlib Below we use a small synthetic dataset that mirrors the retail example. The data is loaded with pandas, which you already saw in Chapter 3. 2.1 Line Plot – Visualizing Trends A line plot connects data points with straight segments, making it ideal for showing change over a continuous variable (time, temperature, etc.). Key terms introduced - marker – shape that marks each data point ('o' = circle). - linewidth – thickness of the plotted line. - grid – optional background lines that help read values. 2.2 Bar Plot – Comparing Categories A bar plot displays categorical data as rectangular bars whose lengths correspond to a value. It is perfect for side‑by‑side comparisons. Why use kind='bar'? Pandas’ Series.plot method forwards arguments to Matplotlib, letting you stay within the familiar pandas workflow while …

5. 5. Data Cleaning and Preparation

A Real‑World Wake‑Up Call Imagine you’ve just received a CSV file containing the last six months of online‑retail sales. The file landed in your Jupyter Notebook with a promising headline: “$2 M in revenue – ready for analysis!” Yet, as soon as you load it with pandas, the first few rows look like this: | OrderID | CustomerID | OrderDate | ProductID | Quantity | UnitPrice | Country | |--------|------------|------------|-----------|----------|-----------|---------| | 1001 | 502 | 2023‑01‑15 | 2001 | 2 | 19.99 | USA | | 1002 | | 2023‑01‑16 | 2003 | 1 | | Canada | | 1003 | 504 | 2023‑01‑16 | 2001 | 2 | 19.99 | USA | | 1003 | 504 | 2023‑01‑16 | 2001 | 2 | 19.99 | USA | | 1004 | 505 | 2023‑02‑02 | 2005 | 5 | 9.99 | Mexico | - Missing values (CustomerID, UnitPrice) - Duplicate rows (OrderID 1003 appears twice) - Potential outlier (Quantity = 5 might be unusually high for this product) If you ignore these imperfections, any downstream insight—average basket size, revenue forecasts, or churn predictions—will be skewed. This chapter equips you with the first‑line tools to detect, diagnose, and repair such data problems using Python’s pandas and NumPy libraries. --- 1. Spotting Imperfections 1.1. Loading the Data The familiar head() call (introduced in Chapter 3) gives a quick visual cue. For larger datasets, df.sample(5) is equally handy. 1.2. Missing Values - Missing value – a placeholder (often NaN) indicating that no data was recorded. - NaN stands for “Not a Number” and is a special floating‑point value that propagates through most arithmetic operations, preventing silent errors. The output tells you how many missing entries each column holds. 1.3. Duplicate Rows - Duplicate – two or more rows that are identical across all columns (or a subset you care about). duplicated() returns a Boolean Series: True for rows that have appeared before. 1.4. Outliers - Outlier – a data point that lies far away from the majority of observations. - Outliers can arise from data‑entry errors, measurement glitches, or genuine rare events. A quick visual cue uses a box plot (from Matplotlib/Seaborn, covered in Chapter 4): Points beyond the whiskers (typically 1.5 × IQR) are flagged as outliers. --- 2. Dealing with Missing Data 2.1. When to Drop vs. When to Impute | Situation | Recommended Action | |-----------|---------------------| | < 5 % of rows missing a non‑critical column | Drop rows (df.dropna()) | | 5 % missing in a key column (e.g., UnitPrice) | Impute (fill) missing values | | Missingness is systematic (e.g., all rows from a specific region) | Investigate why before deciding | Tip: Always back‑up the original DataFrame …

6. 6. Exploratory Data Analysis (EDA)

A Real‑World Spark: Why “Looking First” Saves Hours Imagine you have just received a CSV file containing 2 million rows of retail transaction data. The marketing team is eager to launch a promotion, but before you feed the file into any model, you need to answer questions like: Which products sell the most? Are there seasonal spikes? Do sales differ between online and in‑store channels? Jumping straight to modeling would be like trying to solve a puzzle without first laying out the pieces. Exploratory Data Analysis (EDA) is the systematic “look‑first” step that uncovers patterns, spots problems, and guides the rest of your workflow. In this chapter you’ll learn how to: 1. Summarize data with descriptive statistics. 2. Visualize distributions and correlations. 3. Use groupby operations to compare subsets. 4. Capture the story of your exploration in a clean, reproducible notebook. All of the tools you need—pandas, NumPy, Matplotlib, and Seaborn—were introduced in earlier chapters, so we’ll focus on how to apply them for discovery. --- 1. Summarizing Data with Descriptive Statistics Descriptive statistics turn raw rows into concise numbers that tell you what the data looks like “on average”. The most common measures are: | Statistic | What it tells you | Typical pandas function | |-----------|-------------------|--------------------------| | Count | Number of non‑missing observations | df.count() | | Mean | Arithmetic average | df.mean() | | Median | Middle value (robust to outliers) | df.median() | | Std. deviation | How spread out the values are | df.std() | | Min / Max | Range of the variable | df.min(), df.max() | | Quantiles | Percentiles (e.g., 25th, 75th) | df.quantile([0.25, 0.75]) | 1.1 Quick Summary with describe() Pandas’ DataFrame.describe() bundles most of the above into a single table: The output includes count, mean, std, min, max, and the 25 %, 50 % (median), and 75 % quartiles for every numeric column. For beginners, this is often the first checkpoint after loading data. 1.2 Spotting Skewness and Outliers A mean much larger than the median usually signals a right‑skewed distribution (a long tail to high values). For example, in a Healthcare dataset of patient charges: If the mean is \$12,000 while the median is \$5,000, you know a few very expensive cases are pulling the average up. Recognizing skewness early helps you decide whether to transform the data (e.g., log‑transform) before modeling. Tip: Use df['column'].skew() to get the numerical skewness value; a positive number means right‑skewed, negative means left‑skewed. --- 2. Visualizing Distributions and Correlations Numbers are powerful, but visuals make patterns pop out instantly. This section builds on the Matplotlib/Seaborn basics you practiced earlier. 2.1 Histograms and Kernel Density Estimates (KDE) A histogram shows how many observations fall …

7. 7. Introduction to Statistical Concepts

Why Do Retail Stores Keep Track of “Average” Sales? Imagine a small boutique that sells handmade scarves. The owner, Maya, checks the cash register every night and writes down the total revenue for that day. After a month, she has 30 numbers ranging from \$250 to \$1,200. Maya wants to know whether the recent promotion really boosted sales or if the variation she sees is just random noise. The answer lies in basic statistical concepts—mean, variance, hypothesis testing, confidence intervals, and the tools that let Python do the heavy lifting. This chapter equips you with the language and the code to answer questions like Maya’s, using the same libraries (NumPy, pandas, SciPy) you met in earlier chapters. --- 1. Probability – The Language of Uncertainty 1.1 What Is Probability? Probability quantifies how likely an event is to occur, expressed as a number between 0 (impossible) and 1 (certain). - Experiment – any process that yields an outcome (e.g., flipping a coin). - Sample space – the set of all possible outcomes (heads or tails). - Event – a subset of outcomes (e.g., “getting heads”). The probability of an event $A$ is \[ P(A)=\frac{\text{Number of favorable outcomes}}{\text{Total number of outcomes}} \] When the outcomes are equally likely, this simple ratio works. In data science we often deal with empirical probabilities, estimated from observed data: 1.2 From Probability to Probability Distributions A probability distribution describes how probabilities are spread over all possible values of a random variable. Two common types you’ll encounter early: | Distribution | Typical Use | Shape | |--------------|-------------|-------| | Uniform | Random numbers from a fixed range | Flat | | Normal (Gaussian) | Heights, measurement errors, many aggregated phenomena | Bell‑shaped | Python’s numpy.random can generate samples to illustrate these ideas: --- 2. Descriptive Statistics – Summarizing Data Descriptive statistics turn a cloud of numbers into a concise story. The most common measures are mean, median, variance, and standard deviation. 2.1 Mean (Arithmetic Average) The mean $\bar{x}$ is the sum of all observations divided by their count: \[ \bar{x}= \frac{1}{n}\sum{i=1}^{n} xi \] In Python: 2.2 Median (Middle Value) The median is the value that separates the higher half from the lower half. It is robust to outliers—unlike the mean, a single extreme value won’t skew it. 2.3 Variance and Standard Deviation - Variance measures the average squared deviation from the mean: \[ \text{Var}(X)=\frac{1}{n-1}\sum{i=1}^{n}(xi-\bar{x})^{2} \] - Standard deviation (σ) is the square root of variance, bringing the unit back to the original scale. Why “$n-1$’’ Instead of “$n$’’? Using $n-1$ (the Bessel correction) yields an unbiased estimator of the population variance when the data are a sample. This nuance is covered in many introductory statistics textbooks; for now, …

8. 8. Machine Learning Foundations with Scikit‑Learn

Why Machine Learning Matters – A Quick Story Imagine you are a data analyst at a small e‑commerce firm. Your manager asks, “Can we predict next month’s revenue for each product so we can order the right amount of stock?” A few weeks earlier you cleaned the sales logs, explored the data with Matplotlib and Seaborn, and noticed that price, discount, and previous month’s sales are strongly linked to the current month’s revenue. Now you need a method that can learn those relationships from the historical data and generalize to unseen months. This is exactly what machine learning does, and scikit‑learn gives you a clean, Pythonic way to turn that intuition into a working model. --- The Scikit‑Learn Workflow Scikit‑learn (often imported as sklearn) follows a simple, repeatable pattern that mirrors the data‑science workflow you have already practiced: 1. Load & prepare the data (you already did cleaning and EDA). 2. Split the data into a training set (to teach the model) and a test set (to evaluate it). 3. Select an algorithm (e.g., linear regression for a continuous target, logistic regression for a categorical target). 4. Fit the model on the training data. 5. Predict on new data (the test set). 6. Evaluate the predictions with appropriate metrics. Each step is a single line of code in scikit‑learn, which makes the whole process transparent and reproducible. --- 1. Splitting Data – Training vs. Test Why split at all? When a model sees the same rows it was trained on, it can memorize them and appear perfect, yet fail miserably on new data. By holding out a portion of the data we can estimate how the model will behave in production. Key points - testsize can be a fraction (e.g., 0.2) or an absolute number of rows. - randomstate fixes the random split so you get the same results each run—useful for learning and debugging. - The split is stratified for classification problems (see later) to preserve class proportions; for regression the default random split is fine. --- 2. Linear Regression – Learning a Straight Line Linear regression assumes the target variable $y$ can be expressed as a weighted sum of the features plus an intercept: \[ \hat{y}= \beta0 + \beta1 x1 + \beta2 x2 + \dots + \betap xp \] where $\beta$s are the coefficients the algorithm learns. Training the Model Interpreting Coefficients - Positive coefficient → increase in the feature tends to increase revenue. - Negative coefficient → increase in the feature tends to decrease revenue. - The intercept is the predicted revenue when every feature is zero (often not meaningful on its own, but required for the equation). Tip: If a coefficient is close to zero, that …

9. 9. End‑to‑End Mini Project

A Real‑World Question to Kick‑Start Your Project Imagine you are hired by a small city government that wants to reduce traffic congestion on a popular commuter corridor. They have collected daily bike‑sharing usage data for the past three years and ask: “Can we predict how many bikes will be rented tomorrow, given the weather forecast and calendar information?” This single, concrete question will guide every decision you make in the mini‑project that follows. By the end, you will have an end‑to‑end data‑science workflow that mirrors what professionals deliver to stakeholders: a clean dataset, insightful visualizations, a tested predictive model, and a concise report that tells a story. Why start with a question? A clear problem statement frames the data you need, the analyses you will perform, and the metrics you will use to judge success. It also keeps the project focused and prevents “analysis paralysis.” --- 1. Selecting a Public Dataset & Defining the Problem 1.1. Where to Find Open Data Begin by browsing reputable portals that host ready‑to‑download CSV, JSON, or Excel files: Kaggle Datasets – searchable by domain, size, and popularity. UCI Machine Learning Repository – classic benchmark datasets. data.gov (U.S. government) – transportation, health, finance, and more. European Data Portal – multilingual datasets across EU countries. For our bike‑sharing scenario, the Capital Bikeshare system provides a public CSV file named 2023-capital-bikeshare-trip-data.csv. It contains fields such as starttime, endtime, startstation, durationsec, membercasual, and weather‑linked columns (tempf, precipinches). 1.2. Crafting a Precise Problem Statement A good problem statement answers what, why, and how: What – Predict the number of bike rentals for the next day. Why – Enable the city to allocate additional bikes to high‑demand stations, reducing shortages. How – Build a regression model that takes weather forecasts and calendar features (weekday, holiday) as inputs. Write the statement in one sentence and keep it visible (e.g., as a comment at the top of your notebook). This habit mirrors professional data‑science “project charters.” --- 2. Setting Up a Reproducible Workspace 2.1. Directory Layout A tidy folder structure makes collaboration and future revisions painless: 2.2. Managing Dependencies Create an environment file (or requirements.txt) that lists the exact versions of NumPy, pandas, Matplotlib, Seaborn, and scikit‑learn you used. This ensures that anyone who clones the repository can reproduce the analysis with a single conda env create -f environment.yml command. Tip: Throughout the book you have learned how to import these libraries; now you will see how they fit into a reproducible pipeline. --- 3. Data Acquisition & Initial Inspection 3.1. Loading the Data Recall from Chapter 3 how pandas makes loading CSVs a one‑liner. 3.2. Quick sanity check Key things to verify: Rows vs. columns – Is the shape reasonable (e.g., …