Pustakam Library

Free Data Science learning guide

Data Science with Python for Beginners: A Step-by-Step Guide

Data Science with Python for Beginners: A Step-by-Step Guide — a free beginner-level guide covering data science with python for beginners. Learn with...

60 min read10 chaptersbeginner

What you will learn

  1. Python Fundamentals for Data Science
  2. Numerical Computing with NumPy
  3. Data Manipulation with Pandas
  4. Exploratory Data Analysis (EDA)
  5. Data Visualization
  6. Introduction to Linear Algebra and Statistics
  7. Supervised Learning: Regression
  8. Supervised Learning: Classification
  9. Unsupervised Learning: Clustering
  10. Model Optimization and Validation

1. Python Fundamentals for Data Science

Your First Step into the Data Scientist's Toolkit Imagine you have a massive spreadsheet containing a million rows of customer purchase history. You need to find every customer who spent more than \$500 in January, calculate their average spend, and group them by city. Doing this manually in a spreadsheet would be a nightmare of clicking, dragging, and accidental deletions. This is where Python comes in. Python is not just a programming language; it is a tool that allows you to write a set of instructions (a script) that can process that million-row dataset in seconds. For data scientists, Python is the industry standard because it reads almost like English, has a massive community of contributors, and possesses specialized libraries that handle complex mathematics and data manipulation effortlessly. Before we can build predictive models or visualize trends, we must master the "grammar" of the language. This is the foundation upon which every data analysis project is built. --- Setting Up Your Environment To write and run Python code, you need two things: the Python interpreter (the engine that reads your code and executes it) and an Integrated Development Environment (IDE) (the software where you actually type your code). Installing Python The most straightforward way to get started is by installing Anaconda. Anaconda is a distribution of Python specifically designed for data science. It bundles Python with the most important libraries and the Jupyter environment, saving you from installing dozens of separate packages. 1. Download the installer from the Anaconda website. 2. Follow the installation prompts (the default settings are usually sufficient). 3. Once installed, open the "Anaconda Navigator" application. The Jupyter Notebook While professional software engineers often use complex editors, data scientists prefer Jupyter Notebooks. A Jupyter Notebook is an interactive web-based environment. Unlike a traditional script that runs from top to bottom, a notebook is divided into cells. You can write a few lines of code in one cell, run it, see the result immediately, and then write more code in the next cell. This "trial-and-error" workflow is essential for data science, where you often need to inspect your data at every step. To launch your first notebook: 1. Open Anaconda Navigator. 2. Click "Launch" under the Jupyter Notebook icon. 3. In the browser window that opens, navigate to a folder and select New $\rightarrow$ Python 3. --- The Building Blocks: Basic Data Types In Python, every piece of information is an object, and every object has a type. The type tells Python how to treat the data—for example, you can multiply two numbers, but you cannot multiply two sentences. Integers and Floats Numbers in Python primarily fall into two categories: Integers (int): Whole numbers without a decimal point. …

2. Numerical Computing with NumPy

Why Python Lists Aren't Enough for Data Science Imagine you are building a weather monitoring system. You have a sensor recording the temperature every minute for an entire year. That is 525,600 data points. To analyze this data, you need to convert every single reading from Fahrenheit to Celsius. If you used a standard Python list, you would have to write a for loop to visit every single item, perform the math, and store the result in a new list. For a few hundred items, this is fine. For millions of items—which is common in data science—this approach is painfully slow. This is where NumPy (short for Numerical Python) comes in. NumPy provides a specialized object called the ndarray (N-dimensional array) that allows you to perform the same operation on millions of data points simultaneously. This shift from looping through items one-by-one to operating on the whole collection at once is called vectorization. Getting Started with NumPy To use NumPy, you must first import the library. By convention, the data science community aliases numpy as np to keep the code concise. Creating Your First Arrays An array is a grid of values, all of the same type. While a Python list can hold a mix of integers, strings, and booleans, a NumPy array requires homogeneity (every element must be the same data type). This restriction is exactly what makes NumPy so fast; the computer knows exactly how much memory to allocate for each element. From Lists to Arrays The simplest way to create an array is to pass a Python list into the np.array() function. Built-in Array Generators Often, you need to create arrays with placeholder values or specific sequences. NumPy provides optimized functions for this: np.zeros(shape): Creates an array filled with 0s. np.ones(shape): Creates an array filled with 1s. np.arange(start, stop, step): Similar to Python's range(), but returns an array. np.linspace(start, stop, num): Creates a specified number of evenly spaced values between two numbers (extremely useful for plotting graphs). Array Dimensions and Shapes To manipulate data effectively, you must understand the "geometry" of your array. Dimensions (Axes) In NumPy, dimensions are called axes. 1D Array: A single line of elements (an axis 0). 2D Array: A table with rows (axis 0) and columns (axis 1). 3D Array: A "cube" of data (e.g., a color image, which has height, width, and three color channels: Red, Green, and Blue). Shape and Size You can inspect an array using the .shape attribute, which returns a tuple representing the length of each dimension. Indexing, Slicing, and Reshaping Because NumPy arrays are based on the same logic as Python lists, zero-based indexing and slicing apply here, but they are extended to handle multiple dimensions. …

3. Data Manipulation with Pandas

From Raw Tables to Actionable Insights Imagine you have just been handed a folder containing three different files: a CSV of customer orders, an Excel spreadsheet of product prices, and a text file of shipping logs. Your boss asks a simple question: "Which product category generated the most revenue in the Northeast region last quarter?" If you tried to answer this using standard Python lists or dictionaries, you would spend hours writing nested loops just to align the rows and calculate the sums. This is where Pandas comes in. Pandas is the industry-standard library for data manipulation in Python. It takes the power of NumPy arrays and adds "labels" to the data, allowing you to treat your data like a high-powered version of an Excel spreadsheet. Instead of tracking index numbers, you can refer to data by column names (like "Revenue" or "Date"), making your code readable and your analysis efficient. To get started, you must first import the library. By convention, it is imported with the alias pd: --- The Building Blocks: Series and DataFrames Before we can manipulate data, we need to understand the two primary objects Pandas uses to store information. The Series A Series is a one-dimensional array. You can think of it as a single column in a table. Unlike a NumPy array, a Series has an index, which is a label for each row. In this example, "Apple", "Banana", and "Cherry" are the indices. This allows you to retrieve a value using the label rather than the position. The DataFrame A DataFrame is a two-dimensional data structure. It is essentially a collection of Series objects that share the same index. In simpler terms, it is a table with rows and columns. Here, the "Product", "Price", and "Quantity" columns are individual Series, and the df object holds them all together in a tabular format. --- Loading Data from External Sources In the real world, you rarely type your data into a script. You load it from files. Pandas provides "reader" functions that convert external files into DataFrames automatically. Reading CSV Files CSV (Comma Separated Values) files are the most common format in data science because they are plain text and universal. Reading Excel Files Excel files (.xlsx) are common in corporate environments. Pandas can handle these, though it often requires an additional helper library like openpyxl installed in your environment. --- Data Selection, Filtering, and Sorting Once your data is loaded, you need to isolate the specific pieces of information relevant to your question. Selecting Columns and Rows To select a single column, you can use the column name in square brackets: To select multiple columns, pass a list of names: Filtering Data (Boolean Indexing) …

4. Exploratory Data Analysis (EDA)

The Detective Work of Data Science Imagine you are handed a spreadsheet containing 10,000 rows of sales data from a global retail chain. Your boss asks a simple question: "Why did our profits dip in October?" You could immediately start building a complex machine learning model to predict future profits, but that would be like trying to solve a crime without first looking at the evidence. You don't know if there are missing values in the "Profit" column, if a few massive returns skewed the average, or if a specific region suffered a logistics collapse. This is where Exploratory Data Analysis (EDA) comes in. EDA is the process of "interrogating" your data. It is the detective work of data science—the phase where you summarize the main characteristics of a dataset, find patterns, spot anomalies, and check assumptions before applying any formal modeling. The goal of EDA is not to prove a point, but to discover what the data is trying to tell you. Understanding Data Distribution with Descriptive Statistics Before we can find "why" something happened, we need to know "what" we are looking at. We do this using descriptive statistics, which are brief coefficients that summarize a given data set. Measures of Central Tendency Central tendency tells us where the "middle" of the data lies. Using the Pandas skills you learned in the previous chapter, you can calculate these quickly. Mean: The arithmetic average. You calculate it by summing all values and dividing by the count. While common, the mean is sensitive to extreme values (outliers). Median: The middle value when the data is sorted from lowest to highest. If you have an even number of observations, it is the average of the two middle numbers. The median is "robust," meaning it isn't easily swayed by a few massive or tiny numbers. Mode: The value that appears most frequently in a dataset. This is particularly useful for categorical data (e.g., knowing that "Blue" is the most common shirt color sold). Measures of Dispersion (Spread) Knowing the average isn't enough. If the average temperature of a city is 70°F, it could mean every day is 70°F, or it could mean half the days are 0°F and half are 140°F. Dispersion tells us how "spread out" the data is. Standard Deviation: This measures the average distance of each data point from the mean. A low standard deviation means the data is clustered closely around the average; a high standard deviation means the data is spread wide. Variance: The square of the standard deviation. While less intuitive to visualize than standard deviation, it is mathematically fundamental to many advanced statistical tests. Range: The difference between the maximum and minimum values. Interquartile Range …

5. Data Visualization

The Power of a Single Image Imagine you are presenting a report to a manager. You have a Pandas DataFrame containing 10,000 rows of sales data across 50 different cities over three years. You want to show that sales in the Northeast region have been steadily declining while the Southwest is booming. You could show them a table of averages, or a long list of summaries. But the manager would have to spend several minutes mentally calculating trends and comparing numbers. Instead, you show them a single line chart. In three seconds, the manager sees one line sloping down and another curving up. The conclusion is instant and undeniable. This is the core of Data Visualization: the process of translating raw numbers into a visual context—such as a map or graph—to make data easier for the human brain to understand. While Exploratory Data Analysis (EDA) helps you find the patterns, visualization is how you communicate those patterns to others (and yourself). The Python Visualization Ecosystem Python doesn't have a single "drawing tool." Instead, it uses libraries. For beginners, two libraries dominate the landscape: Matplotlib and Seaborn. Matplotlib: The Foundation Matplotlib is the "grandfather" of Python visualization. It is a low-level library, meaning it gives you total control over every single pixel on the screen. If you want to change the thickness of a tick mark on the axis or place a text label at a specific coordinate, Matplotlib is the tool. Seaborn: The High-Level Wrapper Seaborn is built on top of Matplotlib. It is a high-level library, meaning it provides "shortcuts" to create complex, aesthetically pleasing charts with much less code. Seaborn is specifically designed to work seamlessly with Pandas DataFrames. The general rule of thumb: Use Seaborn for fast, beautiful statistical plots, and use Matplotlib for fine-tuning the final details. --- Getting Started: The Basic Setup To begin, you need to import both libraries. By convention, the community uses specific aliases to keep the code concise. In this code, plt becomes our shorthand for the Matplotlib plotting functions, and sns is the shorthand for Seaborn. --- Basic Plots: Visualizing Single Variables When you start with a dataset, your first goal is often to understand the "shape" of a single column of data. This is called univariate analysis. Line Charts: Tracking Trends Line charts connect individual data points with a line. They are the gold standard for showing trends over time (time-series data). Example: Website Traffic Suppose you have a list of daily visitors to a blog over one week. plt.plot(): Creates the line. The marker='o' argument adds dots to the specific data points. plt.grid(True): Adds a background grid to make the values easier to read. Bar Charts: Comparing Categories …

6. Introduction to Linear Algebra and Statistics

Why Math Matters for Data Science Imagine you are building a recommendation system for a streaming service. You have thousands of users and thousands of movies. To suggest a movie, the computer doesn't "watch" the film; instead, it represents the movie as a list of numbers (genres, length, release year) and the user as a similar list (preferences). The process of calculating how "close" a user's preferences are to a movie's attributes is not a magic trick—it is Linear Algebra. Similarly, imagine you see a sudden spike in user cancellations. Is this a random fluke, or is there a systemic problem with your latest app update? To answer this, you can't just look at the average; you need to know if the change is "statistically significant." This is where Statistics comes in. While Python libraries like NumPy and Pandas handle the heavy lifting, understanding the underlying math allows you to choose the right tools, debug your models, and interpret your results without relying on guesswork. --- Descriptive Statistics: Summarizing Data Before we can predict the future, we must describe the present. Descriptive statistics provide a way to condense a large dataset (like a Pandas DataFrame) into a few meaningful numbers. Central Tendency: The Mean The Mean is the most common measure of "center." It is the arithmetic average, calculated by summing all values and dividing by the total count of values. When to use it: When your data is distributed evenly without extreme outliers. The limitation: The mean is highly sensitive to outliers. For example, if nine people earn \$50,000 a year and one person earns \$1,000,000, the mean income is \$145,000—a number that doesn't accurately describe anyone in the group. Dispersion: Variance and Standard Deviation Knowing the average isn't enough. Consider two cities where the average temperature is 70°F. In City A, every day is between 68°F and 72°F. In City B, it swings between 20°F and 120°F. The mean is the same, but the "experience" of the data is entirely different. This difference is called Dispersion. Variance measures how far each number in the set is from the mean. To calculate it, you subtract the mean from each value, square the result (to ensure negative numbers don't cancel out positive ones), and average those squares. Standard Deviation is simply the square root of the variance. We use it more often than variance because it returns the value to the original units of the data. Low Standard Deviation: Data points are clustered tightly around the mean (City A). High Standard Deviation: Data points are spread far apart (City B). --- Probability Distributions A Probability Distribution is a mathematical function that describes the likelihood of obtaining the possible values that …

7. Supervised Learning: Regression

Predicting the Future: The Power of Regression Imagine you are a real estate agent. A client walks in and asks, "How much should I list my house for?" To answer this, you don't guess randomly. You look at the square footage, the number of bedrooms, and the prices of similar houses sold in the neighborhood recently. You are instinctively performing Regression. At its core, regression is a way to quantify the relationship between variables to predict a continuous numerical value. Unlike predicting a category (like "spam" or "not spam"), regression predicts a quantity—like a price, a temperature, or a stock value. This is a form of Supervised Learning. In supervised learning, we provide the algorithm with a dataset that contains both the input data and the correct answers (called labels). The model learns the pattern connecting the two, allowing it to predict the label for new, unseen data. Understanding Linear Regression Linear Regression is the simplest and most widely used regression algorithm. It assumes that the relationship between the input (independent variable) and the output (dependent variable) can be represented by a straight line. Simple Linear Regression Simple Linear Regression occurs when we use a single independent variable to predict a dependent variable. Drawing from your knowledge of Introduction to Linear Algebra and Statistics, remember the equation for a straight line: $y = mx + b$ In data science, we tweak the notation slightly, but the logic remains the same: $\hat{y} = \beta0 + \beta1x$ $\hat{y}$ (y-hat): The predicted value (Dependent Variable). $x$: The input feature (Independent Variable). $\beta0$ (Intercept): The value of $y$ when $x$ is zero. $\beta1$ (Coefficient/Slope): The amount $y$ is expected to increase (or decrease) for every one-unit increase in $x$. Example: Predicting a student's test score based on the number of hours they studied. If the model finds that $\beta0 = 50$ and $\beta1 = 5$, the equation is $\text{Score} = 50 + 5(\text{Hours})$. A student studying 0 hours is predicted to score 50; for every hour studied, the score increases by 5 points. Multiple Linear Regression In the real world, a single variable is rarely enough. A house price isn't just based on square footage; it depends on the neighborhood, the age of the home, and the number of bathrooms. Multiple Linear Regression uses two or more independent variables to predict one dependent variable: $\hat{y} = \beta0 + \beta1x1 + \beta2x2 + ... + \betanxn$ The model assigns a coefficient ($\beta$) to each feature, indicating how much that specific feature contributes to the final prediction while holding other features constant. Preparing Data for Regression Before we feed data into a model, we must organize it. Using Data Manipulation with Pandas, you are already familiar …

8. Supervised Learning: Classification

The Yes/No Question: What is Classification? Imagine you are building a system for a bank to flag fraudulent credit card transactions. For every transaction that occurs, the bank needs a definitive answer: Is this transaction fraudulent or legitimate? Unlike the problems we solved in Supervised Learning: Regression, where we predicted a continuous number (like a house price or a temperature), we are no longer looking for "how much." We are looking for "which category." This is the essence of Classification. Classification is a type of supervised learning where the goal is to predict a categorical label (also called a discrete label) for a given input. If you are predicting whether an email is "Spam" or "Not Spam," whether a tumor is "Malignant" or "Benign," or whether a customer will "Churn" or "Stay," you are performing classification. Binary vs. Multi-class Classification Depending on the number of categories, classification is generally split into two types: 1. Binary Classification: The target has only two possible outcomes (e.g., Yes/No, True/False, 0/1). 2. Multi-class Classification: The target has three or more possible outcomes (e.g., predicting if an image is a "Cat," "Dog," or "Bird"). For the remainder of this chapter, we will focus primarily on binary classification to build a strong foundation. --- Logistic Regression: The Baseline Classifier Despite its name, Logistic Regression is not used for regression (predicting numbers); it is used for classification. In Supervised Learning: Regression, we learned about Linear Regression, which fits a straight line to data. However, a straight line is problematic for classification because it can predict values like 1.5 or -2.0, which make no sense when our only options are 0 (No) or 1 (Yes). The Sigmoid Function To fix this, Logistic Regression takes the output of a linear equation and passes it through a mathematical function called the Sigmoid Function. The Sigmoid function acts as a "squashing" mechanism. No matter how large or small the input number is, the Sigmoid function forces the output to fall strictly between 0 and 1. In data science, we interpret this output as a probability. For example, if the model outputs 0.85, it means there is an 85% probability that the data point belongs to the "Positive" class (Class 1). Setting a Decision Threshold Since a probability (0.85) is not a category ("Fraud"), we apply a Decision Threshold. The most common threshold is 0.5: If Probability $\ge$ 0.5 $\rightarrow$ Class 1 (Positive) If Probability $<$ 0.5 $\rightarrow$ Class 0 (Negative) Implementing Logistic Regression with Scikit-Learn To implement this in Python, we use the scikit-learn library. Assume we have a Pandas DataFrame df with a feature creditscore and a target isfraud. --- Decision Trees: Mimicking Human Logic While Logistic Regression uses a …

9. Unsupervised Learning: Clustering

The Mystery of the Unlabeled Dataset Imagine you are a marketing analyst for a global music streaming service. Your boss hands you a dataset containing the listening habits of 10,000 users: how many songs they skip, their average volume level, the time of day they listen, and the genres they prefer. There is one problem: the data has no labels. You don’t have a "Churn" column or a "Premium User" category. You simply have a massive table of numbers. Your goal is to figure out if there are natural "types" of listeners—perhaps "The Late-Night Lo-Fi Studiers" or "The High-Energy Gym Goers"—so the marketing team can create targeted playlists for them. Up until now, we have focused on Supervised Learning. In Regression and Classification, we had a "teacher" (the target label) telling the model exactly what to predict. But in the real world, we often have data without answers. This is where Unsupervised Learning comes in. Instead of predicting a known value, we ask the computer to find hidden patterns, structures, or groupings within the data on its own. The most common form of this is Clustering. What is Clustering? Clustering is the process of grouping a set of data points so that points in the same group (called a cluster) are more similar to each other than to those in other groups. If you think back to the Exploratory Data Analysis (EDA) and Data Visualization chapters, you might have used scatter plots to spot visual clumps of data. Clustering is essentially the mathematical automation of that process. It allows us to group data in two, ten, or even a hundred dimensions—far more than our eyes can see. The Concept of Similarity To group things, the computer needs a way to measure "similarity." In data science, similarity is usually defined as distance. If we represent two users as points on a graph, the physical distance between those points represents how different they are. The most common measurement is Euclidean Distance (the "straight-line" distance between two points), which you may recall from your study of Linear Algebra and Statistics. K-Means Clustering K-Means is the most widely used clustering algorithm. The "K" stands for the number of clusters you want the algorithm to find. How K-Means Works K-Means follows a simple, iterative process to find the center of each cluster: 1. Initialization: The algorithm randomly picks $K$ points in the data space to act as the initial centroids (the center point of a cluster). 2. Assignment: Every data point is assigned to the nearest centroid based on Euclidean distance. 3. Update: The algorithm calculates the average (the mean) of all points assigned to each centroid. The centroid then moves to this new average …

10. Model Optimization and Validation

The "Perfect" Model Trap Imagine you are training a machine learning model to predict whether a loan applicant will default. You feed your model a dataset of 1,000 past customers. After some tweaking, you achieve 99% accuracy. You are thrilled—until you deploy the model to the real world. Suddenly, the accuracy plummets to 60%. What happened? Your model didn't actually learn how to identify "risky borrowers"; it simply memorized the specific names and patterns of the 1,000 people in your training set. It became a master of the past but is useless for the future. This is the central challenge of data science: Generalization. The goal is not to perform perfectly on the data you already have, but to perform reliably on data the model has never seen before. This chapter focuses on the tools we use to ensure our models are robust, fair, and optimized for the real world. --- Understanding Overfitting and Underfitting Before we can optimize a model, we must understand why it fails. Most model failures fall into two categories: Underfitting and Overfitting. Underfitting: The Over-Simplifier Underfitting occurs when a model is too simple to capture the underlying pattern of the data. It’s like trying to describe a complex coastline using only a straight line. An underfit model performs poorly on both the training data and new data. This usually happens because: The model is too simple (e.g., using a linear regression for data that follows a curve). The model wasn't trained long enough. The features provided aren't descriptive enough to explain the outcome. Overfitting: The Over-Memorizer Overfitting occurs when a model learns the "noise" (random fluctuations) in the training data rather than the actual signal. It is the equivalent of a student who memorizes the exact answers to a practice exam but doesn't understand the concepts; they get 100% on the practice test but fail the actual exam. An overfit model performs exceptionally well on training data but poorly on new, unseen data. This usually happens because: The model is too complex (e.g., a decision tree with too many branches). The training dataset is too small. The model was trained for too many iterations. The Bias-Variance Tradeoff To understand these concepts technically, we use two terms: Bias and Variance. 1. Bias is the error resulting from overly simplistic assumptions. High bias leads to underfitting. 2. Variance is the error resulting from overly complex models that are too sensitive to small fluctuations in the training set. High variance leads to overfitting. The goal of model optimization is to find the "sweet spot" where both bias and variance are minimized, resulting in a model that generalizes well. --- Feature Scaling: Preparing the Ground Many machine learning algorithms …

Continue learning