Pustakam Library

Free Data Science learning guide

Data Science with Python for Beginners: Step-by-Step Guide

Data Science with Python for Beginners: Step-by-Step Guide — a free beginner-level guide covering data science with python for beginners. Learn with...

84 min read9 chaptersbeginner

What you will learn

  1. 1. Introduction to Data Science and Python
  2. 2. Setting Up the Python Environment
  3. 3. Python Fundamentals for Data Science
  4. 4. Working with Data: NumPy and Pandas
  5. 5. Data Visualization with Matplotlib & Seaborn
  6. 6. Exploratory Data Analysis (EDA)
  7. 7. Introduction to Statistics for Data Science
  8. 8. Machine Learning Basics with Scikit-learn
  9. 9. End-to-End Project: From Raw Data to Insights

1. 1. Introduction to Data Science and Python

What Is Data Science? Imagine you receive a spreadsheet that contains every transaction made at a small online shop over the past year. The file has thousands of rows, each row showing a product ID, the price paid, the date of purchase, and the customer’s location. A business owner asks, “Can we tell which products are most profitable, which customers are most likely to buy again, and whether we should run a promotion next month?” Answering those questions is precisely what data science does: it turns raw data into actionable insights. At its core, data science is an interdisciplinary field that combines statistics, computer science, and domain expertise (knowledge about the specific industry or problem) to extract knowledge and inform decisions. Core Components of Data Science | Component | What It Means | Typical Tools / Techniques | |-----------|---------------|----------------------------| | Domain Knowledge | Understanding the problem context (e.g., retail, healthcare, finance). | Interviews, industry reports, subject‑matter experts. | | Mathematics & Statistics | The mathematical foundation for summarizing data and making predictions. | Probability theory, hypothesis testing, regression. | | Programming | Writing code to manipulate data, build models, and automate tasks. | Python, R, SQL, Bash. | | Data Engineering | Collecting, storing, and preparing data for analysis. | Databases, ETL pipelines, cloud storage. | | Machine Learning | Algorithms that learn patterns from data without explicit programming. | Decision trees, clustering, neural networks. | | Communication & Visualization | Translating technical results into clear stories for non‑technical audiences. | Charts, dashboards, presentations. | These pieces work together like a puzzle: without the domain context, a model may be mathematically correct but irrelevant; without programming, the model cannot be applied at scale; without communication, the insight never reaches the decision‑makers. --- The Data Science Workflow Although projects differ, most data‑science efforts follow a repeatable workflow that can be visualized as a loop rather than a straight line. The five canonical stages are: 1. Collect – Gather raw data from sources such as databases, APIs, sensors, or files. 2. Clean – Fix errors, handle missing values, and transform data into a usable format. 3. Explore – Perform exploratory data analysis (EDA) to discover patterns, spot anomalies, and form hypotheses. 4. Model – Apply statistical or machine‑learning techniques to answer the original question. 5. Communicate – Present findings through visualizations, reports, or interactive dashboards. Below is a more detailed look at each stage, with beginner‑friendly explanations of the terminology you’ll encounter. 1. Collect Data rarely arrives in a perfectly tidy package. Sources can include: - CSV or Excel files downloaded from a web portal. - Relational databases (e.g., MySQL, PostgreSQL) accessed via SQL queries. - Web APIs that deliver JSON or …

2. 2. Setting Up the Python Environment

A Real‑World Prompt: “I Got a CSV, What Next?” Imagine you have just received a CSV file containing the daily sales of a small online shop. Your manager asks you to quickly spot any abnormal spikes and report the findings by the end of the day. You know the data needs to be examined, visualized, and perhaps modeled, but you haven’t yet set up a Python workspace on your laptop. This chapter walks you through exactly that: from installing Python to firing up a Jupyter notebook where you can load the CSV, plot the sales trend, and start asking questions. By the end, you’ll have a clean, reproducible environment ready for any data‑science task. --- 1. Installing Python – Two Paths to the Same Destination 1.1 Why the installer matters Python is the programming language that powers most data‑science work. A distribution bundles the interpreter with additional tools (package managers, scientific libraries, IDEs). The two most common routes for beginners are Anaconda and the official Python installer. | Feature | Anaconda | Official Installer | |---------|----------|---------------------| | Packages included out‑of‑the‑box | numpy, pandas, matplotlib, scikit‑learn, Jupyter, … | only the interpreter (you add packages later) | | Package manager | conda (handles binaries, works on Windows, macOS, Linux) | pip (Python’s default, works everywhere) | | Disk space | ~3 GB (larger) | ~100 MB (smaller) | | Learning curve | Slightly higher (new commands) | Simpler if you already know pip | | Ideal for | Beginners who want a “one‑click” start | Users who prefer a lightweight install or already have Python elsewhere | Both ways give you a working interpreter; pick the one that fits your comfort level and system constraints. The instructions below cover Windows, macOS, and Linux. 1.2 Installing Anaconda 1. Download the installer from the Anaconda Distribution page. Choose the Python 3.11 version (the latest stable release) and the installer for your OS. 2. Run the installer Windows – double‑click the .exe file, click Next through the wizard, and accept the default installation location. macOS – open the .pkg file and follow the prompts. Linux – open a terminal, navigate to the download folder, and execute: Accept the license, confirm the install path (default is fine), and answer yes to “initialize Anaconda3 by running conda init?”. 3. Close and reopen any command‑line window (Command Prompt, PowerShell, Terminal) to let the installer modify your PATH. 1.3 Installing the Official Python Release 1. Download the installer from the Python.org downloads page. Choose the latest Python 3.11 stable release for your OS. 2. Run the installer Windows – check Add Python 3.11 to PATH before clicking Install Now. macOS – open the .pkg file; the installer adds …

3. 3. Python Fundamentals for Data Science

A Real‑World Prompt: Cleaning a Mini Sales Log Imagine you have received a tiny CSV file from a small retailer that looks like this: | date | product | unitssold | |------------|-----------|------------| | 2023‑01‑01 | “Widget” | 5 | | 2023‑01‑02 | “Gadget” | 3 | | 2023‑01‑03 | “Widget” | 7 | | 2023‑01‑04 | “Doohickey”| 2 | Your first task is to load the data, summarize it (e.g., total units sold per product), and filter out any product that sold fewer than 5 units in total. You could hand‑write a spreadsheet, but a few lines of Python will do the job faster, repeatably, and—most importantly for a data scientist—scale to thousands of rows. The building blocks you need to write that script are exactly what this chapter covers: variables, basic data types, core data structures, control flow, and functions. Let’s assemble them piece by piece. --- Variables and Basic Data Types Naming and Assigning A variable is simply a name that points to a value stored in memory. In Python you create a variable by writing the name, an equals sign, and the value: Naming rules (always good to keep in mind): Start with a letter (a–z, A–Z) or an underscore . Subsequent characters may be letters, digits, or underscores. Avoid Python’s reserved words (if, for, def, …). Following snakecase (all lower‑case with underscores) is the conventional style for variable names in data‑science scripts. Primitive Data Types | Type | Description | Example | |------|-------------|---------| | int | Whole numbers, no decimal point | 42 | | float | Real numbers with a fractional part | 3.1415 | | str | Text surrounded by quotes | "sales" | | bool | Truth value, either True or False | False | Python automatically infers the type from the literal you write. You can check a variable’s type with type(): Basic Operators | Category | Operators | Example | |----------|-----------|---------| | Arithmetic | + - / // % | 5 + 2 = 7, 5 / 2 = 2.5 | | Comparison | == != < = <= | unitssold 5 | | Logical | and or not | isactive and unitssold 0 | | Assignment | = (simple) += -= = /= | total += unitssold | Quick example – average units sold Output: --- Core Data Structures Data rarely lives in a single scalar variable. Python offers four built‑in containers that are indispensable for data‑science workflows. Lists – Ordered, Mutable Collections A list holds an ordered sequence of items and can be changed (items added, removed, or reordered). Common list operations: len(dailysales) – number of elements. dailysales.append(4) – add a new value at the end. dailysales[0] – first …

4. 4. Working with Data: NumPy and Pandas

Why Arrays and DataFrames Matter Imagine you have just received a zip file containing 30 GB of raw sensor readings from a fleet of delivery trucks. Each truck logs temperature, speed, GPS coordinates, and fuel level every second. Opening the file in a spreadsheet would freeze your computer; looping through the rows with plain Python would take hours. The secret to handling data at this scale lies in two powerful libraries introduced earlier in the workflow: NumPy and Pandas. NumPy gives you fast, memory‑efficient multidimensional arrays and the ability to apply operations to whole blocks of data at once (called vectorized operations). Pandas builds on NumPy to provide a tabular data structure—DataFrames—that feels like an Excel sheet but can be queried, cleaned, and aggregated with just a few lines of code. In this chapter you will learn how to: Create and manipulate NumPy arrays. Perform vectorized arithmetic and understand broadcasting. Load, inspect, and clean data using Pandas DataFrames. Filter, sort, group, and aggregate datasets to extract insights. All of this can be done in a matter of minutes, turning a massive, unwieldy file into a tidy dataset ready for visualization and modeling later in the book. --- Getting Started with NumPy Installing and Importing If you followed the environment‑setup instructions in Chapter 2, NumPy is already installed. In any notebook or script, import it with the conventional alias: Creating Arrays NumPy arrays are created from Python lists, tuples, or other arrays. The key is that all elements share the same data type (dtype), which enables the library’s speed gains. Inspecting Array Attributes Every NumPy array carries metadata that tells you its shape, dimensionality, and data type. Understanding these attributes is essential when you later need to reshape or broadcast arrays. Indexing, Slicing, and Fancy Indexing NumPy follows Python’s zero‑based indexing, but it also supports powerful slicing syntax. Vectorized Operations The term vectorized means applying an operation to an entire array without writing an explicit Python loop. NumPy automatically loops in compiled C code, which is orders of magnitude faster. Because the operation is applied element‑wise, the two arrays must be compatible in shape—a concept clarified by broadcasting. Broadcasting Explained Broadcasting allows NumPy to perform arithmetic on arrays of different shapes by automatically “stretching” the smaller array along missing dimensions. The rules are: 1. If the arrays differ in dimensionality, prepend 1‑s to the shape of the smaller array until they match. 2. Arrays are compatible when, for each dimension, the sizes are equal or one of them is 1. 3. The resulting array has the maximum size along each dimension. Here, b is virtually repeated across rows, while A is repeated across columns, yielding a 3×3 result without extra memory …

5. 5. Data Visualization with Matplotlib & Seaborn

Why Visualizing Data Matters Imagine you have just finished cleaning a massive CSV file containing monthly sales figures for 30 stores across three years. The numbers sit neatly in a Pandas DataFrame, but the story they tell is still hidden. A quick line chart that overlays each store’s performance instantly reveals: Seasonal spikes in November–December One under‑performing store that never recovers after a mid‑year dip A gradual upward trend that suggests a successful marketing campaign Without a visual, you would have to scan thousands of rows manually, a task that can take minutes—or hours—while a well‑crafted plot delivers the same insight at a glance. This chapter shows you how to turn raw numbers into clear, compelling pictures using Matplotlib (the workhorse of Python plotting) and Seaborn (its statistical‑visualization companion). --- Getting Started with Matplotlib Installing and Importing If you followed Setting Up the Python Environment you already have matplotlib installed, but the usual import pattern is worth repeating: Tip: In a script (outside Jupyter) replace the magic line with plt.show() at the end of your plotting code. The Anatomy of a Matplotlib Figure A Matplotlib Figure is the canvas; inside it lives one or more Axes (the actual plotting area). Think of a Figure as a sheet of paper and each Axes as a separate chart on that sheet. figsize – width and height in inches (helps control resolution later). ax – the object you will call methods like ax.plot() or ax.settitle() on. Your First Plot: A Simple Line Chart The code above: 1. Loads a CSV (recall the data‑wrangling steps from Chapter 4). 2. Sets the date column as the index so Matplotlib can interpret it as a time axis. 3. Draws a line (ax.plot) and adds a title and axis labels. --- Plotting the Basics 1. Line Charts Line charts are ideal for trends over time or any ordered sequence. Use label= inside plot and call ax.legend() to generate a legend automatically. 2. Bar Charts Bar charts excel at categorical comparisons (e.g., sales by region). ax.bar takes the category names as the x‑coordinates and the heights as the values. 3. Scatter Plots Scatter plots reveal relationships between two numeric variables. alpha controls point transparency—useful when points overlap. --- Customizing Your Plots Titles, Axis Labels, and Ticks tickparams lets you fine‑tune the size, direction, and color of tick marks. Legends Legends map colors or markers to data series. title adds a heading inside the legend box. frameon=False removes the surrounding box for a cleaner look. Colors, Styles, and Themes Matplotlib ships with several styles (e.g., ggplot, seaborn-darkgrid). Activate one with: You can also set a color palette manually: Subplots and Multiple Axes Often you need more than one chart …

6. 6. Exploratory Data Analysis (EDA)

A Real‑World Question: Why Did Sales Spike Last December? Imagine you have just received a CSV file containing a year‑long sales log from a small online boutique. The file includes columns such as orderid, orderdate, productcategory, unitssold, unitprice, and customerregion. Your manager asks: “What drove the huge sales spike in December? Is it a repeatable pattern or a one‑off anomaly?” Before building any predictive model, you need to explore the data—summarize it, spot inconsistencies, and uncover hidden relationships. This is exactly what Exploratory Data Analysis (EDA) is for. In this chapter you will learn how to: Compute core descriptive statistics (mean, median, standard deviation, percentiles) with Pandas and NumPy. Detect, diagnose, and handle missing values and outliers. Use correlation matrices and pivot tables to investigate relationships between variables. Communicate your discoveries in a concise, reproducible EDA report. All the tools you need—Python, Jupyter notebooks, Pandas, NumPy, Matplotlib, and Seaborn—were introduced in earlier chapters. Here we will combine them into a systematic workflow that you can apply to any dataset. --- 1. The EDA Workflow in Practice A typical EDA process follows a repeatable loop: 1. Load & glimpse the data. 2. Summarize each variable numerically and visually. 3. Inspect data quality: missing values, duplicate rows, inconsistent formats. 4. Detect outliers and decide how to treat them. 5. Explore relationships using correlation, cross‑tabulation, and pivot tables. 6. Document findings in a short report. The following sections walk through each step using the boutique sales data (sales.csv). Feel free to replace the file with any other dataset you own; the code adapts automatically. --- 2. Loading and Getting a First Look parsedates tells Pandas to interpret the orderdate column as actual dates, which enables time‑based operations later. 2.1 Quick Structural Overview The output tells you: Number of rows (observations) and columns (features). Data types (int64, float64, object, datetime64[ns]). Presence of missing values (shown as non‑null counts). If you see many object columns that should be numeric (e.g., unitprice stored as text), you’ll need to convert them—a task we’ll cover in the data‑quality section. 2.2 Peek at the First Few Records A preview helps you verify that columns were parsed correctly and that the data looks sensible (e.g., dates are in the right order). --- 3. Descriptive Statistics: Summarizing What the Data Tells Us Descriptive statistics provide a numerical snapshot of each variable’s central tendency, spread, and shape. Pandas offers built‑in methods that wrap NumPy’s powerful functions. 3.1 Central Tendency – Mean and Median Mean (average) is sensitive to extreme values. Median (the 50th percentile) is robust to outliers. describe() returns a table with count, mean, std, min, 25th percentile, median, 75th percentile, and max for each numeric column. For beginners, think of …

7. 7. Introduction to Statistics for Data Science

Why Statistics Is the Backbone of Data‑Driven Decisions Imagine you have just scraped 10 000 product reviews from an e‑commerce site (see Chapter 2 for how to pull data from a web API). The raw text is now in a Pandas DataFrame, and you have computed the average rating: 4.2 ★. That number looks impressive, but is it reliable? - If the next 1 000 reviews averaged 3.8 ★, would your business strategy change? - How likely is it that the true satisfaction level of all customers is actually lower than 4 ★? Answering these questions requires probability and statistical inference—the tools that let us move from “what we observed” to “what we can conclude about the whole population.” This chapter equips you with those tools, using Python’s scientific stack to turn vague intuitions into quantifiable statements. --- Probability Foundations Sample Space, Events, and Outcomes - Sample space (Ω) – the set of all possible outcomes of a random experiment. - Event – any subset of Ω. - Outcome – a single element of Ω. Example: Tossing a fair coin twice gives Ω = {HH, HT, TH, TT}. The event “at least one heads” = {HH, HT, TH}. Probability Axioms 1. Non‑negativity – P(A) ≥ 0 for any event A. 2. Normalization – P(Ω) = 1. 3. Additivity – If A and B are disjoint, P(A ∪ B) = P(A) + P(B). From these axioms we derive useful rules such as the complement rule: \[ P(A^{c}) = 1 - P(A) \] Conditional Probability & Independence - Conditional probability measures the chance of A given B: \[ P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \quad P(B) 0 \] - Independence means the occurrence of B does not affect A: \[ P(A \mid B) = P(A) \quad \Longleftrightarrow \quad P(A \cap B) = P(A)P(B) \] These concepts appear constantly in data science—for example, when evaluating the probability of a customer churning given that they have not made a purchase in the last month. Using Python to Compute Simple Probabilities The output (~0.75) matches the theoretical value \(1 - (0.5)^2 = 0.75\). --- Common Probability Distributions Statistical models assume that data are generated from an underlying probability distribution. Below are the most frequently encountered families in introductory data science. 1. Discrete Distributions | Distribution | Typical Use | Key Parameters | |--------------|------------|----------------| | Bernoulli | Single binary outcome (e.g., click / no‑click) | p = success probability | | Binomial | Number of successes in n independent trials | n, p | | Poisson | Count of rare events in a fixed interval (e.g., arrivals per hour) | λ = average rate | Python illustration – Poisson arrivals 2. Continuous Distributions | Distribution …

8. 8. Machine Learning Basics with Scikit-learn

What is Machine Learning? Imagine you have a spreadsheet of housing prices and you want to predict the price of a new house based on its size, number of bedrooms, and age. Or picture a bank that receives dozens of loan applications each day and needs a quick way to flag high‑risk applicants. Both tasks can be tackled automatically with machine learning (ML) – algorithms that learn patterns from data and then make predictions on new, unseen cases. ML comes in two broad families: | Family | How it works | Typical goal | |--------|--------------|--------------| | Supervised learning | The algorithm is given input features and the target variable (the answer you want to predict). It learns a mapping from inputs to the target. | Predict a numeric value (regression) or a class label (classification). | | Unsupervised learning | Only the input features are supplied; there is no target. The algorithm looks for hidden structure – clusters, outliers, or lower‑dimensional representations. | Group similar records, detect anomalies, or compress data. | In this chapter we focus on supervised learning, because it directly answers “what will happen next?” – a question that most beginner data‑science projects start with. --- Getting Started with Scikit‑learn Scikit‑learn (often imported as sklearn) is the de‑facto Python library for building and evaluating ML models. It sits on top of the scientific stack you already explored in Chapter 2 (NumPy, Pandas) and follows the same fit‑predict workflow you used for simple data manipulations. Step 1 – Load the data If you have a CSV file (as introduced in Chapter 4), use pd.readcsv. Step 2 – Split into training and test sets The training set teaches the model; the test set evaluates how well it generalizes to new data. Tip: The randomstate argument makes the split reproducible, which is handy when you share notebooks with classmates. From here onward, every example follows the same three‑step pattern: 1. Instantiate the model (e.g., LinearRegression()). 2. Fit it on the training data (model.fit(Xtrain, ytrain)). 3. Predict on new data (model.predict(Xtest)). --- 1️⃣ Linear Regression – Predicting Continuous Values 1.1 Why Linear Regression? Linear regression fits a straight line (or hyperplane in higher dimensions) that best explains the relationship between the input features and a continuous target variable. It’s the simplest regression technique and a great entry point for understanding model coefficients. 1.2 Hands‑on Example: House‑Price Prediction 1.3 Interpreting Coefficients Scikit‑learn stores the learned parameters in linreg.coef (one coefficient per feature) and linreg.intercept (the constant term). Suppose the output is: Size coefficient (150.0) – Holding other factors constant, each additional square foot adds roughly $150 to the predicted price. Age coefficient (‑20.5) – Each extra year of house age decreases the …

9. 9. End-to-End Project: From Raw Data to Insights

9.1 Project Overview – From Raw Data to a Deployable Model Imagine you have just received a CSV file containing the daily bike‑sharing activity of a major city. The file holds thousands of rows of information: dates, temperature, humidity, wind speed, whether a holiday occurred, and the number of rides recorded that day. Your mission is to turn this raw dump into a set of actionable insights—and into a model that can predict future demand so the city can allocate bikes more efficiently. This chapter walks you through every step of that journey, using a single, publicly available dataset as the backbone for three mini‑case studies: | Scenario | Goal | Why it matters | |----------|------|----------------| | Bike‑Sharing Demand (primary walkthrough) | Predict daily ride count | Helps city planners balance supply and demand | | Titanic Survival | Predict whether a passenger survived | Classic binary classification problem for quick experimentation | | California Housing Prices | Predict house price from census data | Demonstrates regression on a larger, more feature‑rich dataset | By the end of the chapter you will be able to collect, clean, explore, model, communicate, and deploy a data‑science solution—all within a single Jupyter notebook and a lightweight deployment script. --- 9.2 Collecting the Data 9.2.1 Choosing a Public Source Public repositories such as Kaggle, UCI Machine Learning Repository, and data.gov host thousands of ready‑to‑download datasets. For our primary example we’ll use the Bike Sharing Dataset from the UCI repository: The file we need is day.csv, a comma‑separated values (CSV) file that can be downloaded with a single line of code (recall the requests library introduced in Setting Up the Python Environment). 9.2.2 Downloading with Python Unzip and locate day.csv (the same logic works for any zip‑archived dataset). For the Titanic and Housing examples, simply replace the URL with the appropriate download link and adjust the filename accordingly. --- 9.3 Loading Data into Python 9.3.1 Reading a CSV with Pandas Having installed the pandas library in Chapter 4, loading a CSV is a one‑liner: The first few rows reveal the column names and data types. Notice that columns like dteday (the date) are read as object strings. We will convert them to proper datetime objects in the cleaning step. 9.3.2 Quick sanity check If the dataset were stored in an Excel workbook (.xlsx) or a relational database, you would use pd.readexcel or pd.readsql, respectively—both covered in Chapter 4. --- 9.4 Cleaning & Preparing the Data Cleaning is where Domain Knowledge meets Programming. The goal is to produce a tidy dataframe that can be fed directly into a model. 9.4.1 Handling Dates 9.4.2 Missing Values The Bike Sharing dataset is complete, but many real‑world files contain …

Continue learning