Free Programming learning guide
Learn Python for Data Analysis: Beginner's Guide
Learn Python for Data Analysis: Beginner's Guide — a free beginner-level guide covering how to learn python for data analysis. Learn with clear...
What you will learn
1. Python Foundations
Why Python for Data Analysis? Imagine you are a data analyst at a rapidly growing e-commerce company. Your manager hands you a CSV file containing 50,000 customer transactions from the last quarter and asks a simple question: "What was our average order value, and which day of the week was most profitable?" If you opened that file in a standard spreadsheet application, you might scroll for minutes just to get a sense of the data. If your computer has limited memory, it might freeze or crash trying to load all 50,000 rows at once. Python changes this equation entirely. Instead of manually clicking through menus, you write a few lines of precise, repeatable instructions. In a fraction of a second, Python reads the file, calculates the average, groups the data by day, and prints the answer. Once you write that script, you can run it on next quarter’s data with a single click. Python is a high-level, general-purpose programming language. "High-level" means its syntax (the rules for how the code is written) is designed to be easily read and understood by humans, abstracting away the complex machine-level instructions happening behind the scenes. While it was originally created in the early 1990s as a language for general software development, it has since become the undisputed lingua franca—the common language—of data analysis and data science. This is largely due to its massive ecosystem of specialized "libraries" (pre-written code bundles you can plug into your own work) designed specifically for manipulating numbers and datasets. However, before you can harness these powerful data tools, you must understand the foundational building blocks of the language itself. Just as you cannot write a novel without knowing the alphabet and basic grammar, you cannot analyze a dataset without understanding Python's basic syntax, variables, and data types. Setting Up Your Local Coding Environment Before you can write your first line of Python, you need a place to write it. To run Python code on your computer, you need two things: the Python interpreter and a code editor. The Python interpreter is the software that reads your code and translates it into instructions your computer's processor can execute. You can download the latest version of Python for free from the official website, python.org. When you install it, ensure you check the box that says "Add Python to PATH" (or "Add python.exe to PATH" on Windows). This step is crucial—it tells your computer's operating system exactly where to find the Python software so you can run it from anywhere on your machine. Once Python is installed, you need a code editor. While you can write Python in a basic text editor like Notepad or TextEdit, it is highly inefficient. Instead, …
2. Control Flow and Logic
Making Decisions with Conditional Statements Imagine you are analyzing a dataset of customer purchases. For every single row of data, you need to categorize the customer as either "High Value" or "Standard" based on whether they spent more than $500. If you had to do this manually for 10,000 rows, you would be reading a number, making a decision, and typing a label, over and over again. This is where control flow comes in. Control flow is the order in which individual statements, instructions, or function calls are executed in a script. By default, a Python program runs from top to bottom, line by line. But as a data analyst, you need your code to make decisions, repeat actions, and skip certain steps based on the data it encounters. The most fundamental tool for controlling this flow is the conditional statement. In Python, we use if, elif, and else statements to give our programs logical decision-making capabilities. Boolean Logic and Comparison Operators Before we write our first if statement, we need to understand how Python evaluates "truth." In the Python Foundations chapter, we introduced data types like the integer, float, and string. There is another critical data type for logic: the boolean. A boolean can only hold one of two values: True or False. To generate boolean values, we use comparison operators. You are already familiar with operators like Addition (+) and Subtraction (-) from the previous module. Comparison operators compare two operands and return a boolean result. == (Equal to): Checks if two values are exactly the same. != (Not equal to): Checks if two values are different. (Greater than): Checks if the left operand is larger than the right. < (Less than): Checks if the left operand is smaller than the right. = (Greater than or equal to) <= (Less than or equal to) Notice that the equality operator is == (a double equals sign), not =. Remember that the single = is the assignment operator, used to assign a value to a variable. Using a single = inside a logical check is one of the most common beginner mistakes! Writing if, elif, and else Statements An if statement evaluates a condition. If that condition is True, the block of code underneath it runs. If it is False, Python skips that block entirely. Python relies on indentation (spaces at the beginning of a line) to define blocks of code. While other programming languages use brackets or parentheses, Python uses whitespace. By convention, you should use four spaces for indentation. Let’s look at a basic example using a variable representing a customer's purchase amount: When you run this code, the Python interpreter checks if 750 is greater than 500. …
3. Core Data Structures
Moving Beyond Single Variables Imagine you are preparing to analyze a dataset containing the daily closing stock prices of Apple over the last month. Based on what we covered in Python Foundations, your first instinct might be to create a separate variable for each day: priceday1 = 150.25, priceday2 = 151.10, priceday3 = 149.80, and so on. For thirty days, this is tedious. For thirty thousand days, it is impossible. Data analysis inherently involves working with collections of data, not just isolated individual values. To handle this, Python provides built-in data structures—specialized containers that organize, store, and manage multiple items in a single variable. In this chapter, we will explore the three most fundamental data structures you will use for data analysis: lists, tuples, and dictionaries. Lists: Ordered and Changeable Collections A list is Python’s most versatile data structure. It allows you to store an ordered collection of items. In programming, "ordered" means that the items have a specific, predictable sequence. If you put three items into a list in a certain order, they will remain in that exact order every time you look at the list. You create a list by placing your items inside square brackets [], separated by commas. Recall from Python Foundations that a variable holds a specific data type like an integer or a string. A list can hold any combination of these data types, even at the same time. Accessing Elements via Indexing Because lists are ordered, you can retrieve a specific item using its position, called an index. Python uses zero-based indexing, meaning the first item is at index 0, the second is at index 1, and so on. You access an element by writing the list’s name followed by the index in square brackets. You can also count backward from the end of the list using negative indices. An index of -1 gives you the last item, -2 gives the second-to-last, and so forth. This is incredibly useful when you have a long list and simply want the most recent entry. Adding and Removing Elements Unlike some data structures we will discuss later, lists are mutable. This means you can change their contents—adding, removing, or modifying items—after the list has been created. To change an existing item, you assign a new value to a specific index using the assignment operator =: To add new items to a list, you will frequently use two built-in methods (functions attached to an object using a dot): append(): Adds a single item to the very end of the list. insert(): Adds an item at a specific index, shifting the rest of the list to the right. To remove items, you have a few options depending on …
4. Introduction to NumPy
Why NumPy? The Need for Numerical Speed Imagine you are an analyst at an e-commerce company. You have a spreadsheet containing the prices of 100,000 products, and your manager asks you to apply a 10% discount to every single item. If you relied only on the core Python concepts covered in earlier chapters, your first instinct might be to use a Python list. You learned in Core Data Structures that lists can hold multiple items. However, to apply that 10% discount, you would need to write a loop that visits each price one by one, multiplies it by 0.90, and appends it to a new list. Python lists are incredibly flexible—they can hold a mix of integers, floats, and strings—but that flexibility comes at a cost. Behind the scenes, a Python list stores pointers to memory addresses rather than the raw data itself. When you ask Python to do math on 100,000 list items, it has to check the data type of every single item before performing the calculation. This overhead makes Python lists notoriously slow for heavy mathematical tasks. This is where NumPy (short for Numerical Python) steps in. NumPy is a third-party Python library that provides a specialized data structure called an array. Unlike a standard Python list, a NumPy array is strictly homogeneous—meaning it can only hold data of the exact same type (e.g., all floats or all integers). Because the library knows exactly what type of data to expect, it can store the raw numbers side-by-side in contiguous blocks of computer memory. This allows NumPy to perform mathematical operations on massive datasets in a fraction of a second, using highly optimized C code under the hood. For data analysis, NumPy is the foundational engine upon which almost all other Python data tools are built. Installing and Importing NumPy Because NumPy is a third-party library, it does not come pre-installed with the standard Python interpreter. You need to install it yourself. Whether you are using Visual Studio Code (VS Code), IDLE, or another Integrated Development Environment (IDEs), the installation process is the same. You will use Python’s package manager, pip, which runs in your terminal or command prompt. Installing the Library Open your terminal and type the following command: Press Enter, and pip will download and install the library onto your system. If you are using a Jupyter Notebook environment, you can run this same command in a code cell by adding an exclamation mark at the front: !pip install numpy. Importing the Library Once installed, NumPy is not automatically available every time you open a Python script. You must bring it into your current working environment using the import statement. By convention, the Python community imports …
5. Pandas Fundamentals
Why Pandas? Imagine you are hired to analyze the sales data for a chain of local coffee shops. The company sends you a spreadsheet containing 50,000 rows of transactions. Each row represents a single purchase, with columns for the date, the store location, the drink type, the size, and the price. If you only had Python's built-in Core Data Structures, like lists and dictionaries, calculating the average price of a large latte in Seattle would require writing complex loops, managing counters, and keeping track of multiple indices. It would be slow, tedious, and prone to errors. This is where Pandas comes in. Pandas is an open-source Python library specifically designed for data manipulation and analysis. It takes the rigid, grid-like structure of a spreadsheet and turns it into a programmable Python object, allowing you to filter, slice, and summarize massive datasets with single lines of code. The name "Pandas" is derived from "Panel Data," an econometrics term for multidimensional structured data sets. In the previous chapter on Introduction to NumPy, you learned how to perform fast mathematical operations on arrays of numbers. While NumPy is incredibly powerful for numerical computations, it struggles when you have mixed data types—like combining text (store locations) and numbers (prices) in the same dataset. Pandas bridges this gap, providing high-level tools to handle tabular data seamlessly. Installing and Importing Pandas Because Pandas is a third-party library, it does not come pre-installed with the standard Python interpreter. You need to install it yourself. Installation If you are using a standard Python environment, you can install Pandas using Python's package manager, pip. Open your terminal or command prompt and run: If you are using the Anaconda distribution (a popular Python distribution for data science), Pandas is already pre-installed. Note: Pandas is built on top of NumPy. When you install Pandas, it automatically installs NumPy as a dependency if you don't already have it. Importing the Library Once installed, you need to bring Pandas into your current script or Integrated Development Environment (IDE) (like Visual Studio Code (VS Code) or IDLE). In the Python community, it is a near-universal convention to import Pandas using the alias pd. This practice keeps your code concise. Every time you want to use a Pandas function, you simply type pd instead of the full word pandas. From this point forward in your script, the pd variable gives you access to the entire Pandas library. The Core Data Structures: Series and DataFrames Pandas revolves around two primary data structures: the Series and the DataFrame. To understand how Pandas works, you must understand the relationship between these two objects. The Pandas Series: A Single Column A Series is a one-dimensional array capable of holding …
6. Importing and Exporting Data
Bridging the Gap Between Files and DataFrames Imagine you are hired to analyze sales data for a local bookstore. The owner hands you a USB drive containing a spreadsheet with five years of daily transactions. You know how to build a Pandas DataFrame from scratch using Python dictionaries, as we explored in Pandas Fundamentals. But manually typing thousands of rows of sales data into your Python script would take months. Real-world data analysis requires a bridge between the files sitting on your computer and the Python environment where you will analyze them. That bridge is built using a set of built-in Pandas functions designed to read external files and translate them into DataFrames. Once your analysis is complete, you need a way to export those findings back into a file format that non-programmers—like the bookstore owner—can open in their preferred software. This chapter focuses on that lifecycle: bringing data in, selecting only what you need, and sending it back out. Understanding Common Data File Formats Before we write any code, it helps to understand how data is stored outside of Python. While there are dozens of file formats in the wild, two dominate the data analysis landscape: CSV and Excel. The CSV Format CSV stands for Comma-Separated Values. It is the most common format for storing tabular data (data organized into rows and columns). A CSV file is essentially a plain text file where: - Each line represents a single row of data. - Commas separate the individual values (columns) within that line. - The first line usually contains the column headers. If you were to open a CSV file in a plain text editor (like Notepad or TextEdit), it might look like this: Because CSVs are just plain text, they are universally compatible. Almost every data tool in existence can read them. They also take up very little storage space. The Excel Format Excel files (typically ending in .xlsx or .xls) are created by Microsoft Excel. Unlike CSVs, Excel files are not plain text. They are complex archives that can store multiple sheets of data, mathematical formulas, text formatting (like bold or colored cells), and charts. While Excel is incredibly popular in the business world, its complexity means we need specific tools to extract the raw data from it into Python. Reading Data from CSV Files The Pandas library provides a powerful function called readcsv() to import CSV files. It reads the file, parses the commas, and automatically constructs a DataFrame for you. To use it, you must first ensure Pandas is imported. By convention, we import Pandas with the alias pd. Handling Different Delimiters While commas are the standard, you will occasionally encounter text files separated by other …
7. Data Cleaning and Preparation
The Messy Reality of Raw Data Imagine you are handed a spreadsheet containing ten years of customer orders for an online store. You open it, eager to analyze purchasing trends, only to find a chaotic landscape. Some rows are missing the customer's age. One column meant to hold prices has random text entries like "TBD" mixed in with the numbers. Another column lists dates in a format your software doesn't recognize. Worst of all, the exact same transaction appears three separate times in different parts of the file. This is the messy reality of raw data. In the previous modules, we successfully imported data into Pandas DataFrames. But before we can perform accurate analysis, we must address these imperfections. In the world of data analysis, there is a universal rule: garbage in, garbage out. If you feed messy, incorrect data into your analysis, your results will be messy and incorrect. Data cleaning is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset. It is often said that data analysts spend 80% of their time cleaning data and only 20% actually analyzing it. Let's learn how to tame that mess using Pandas. Identifying Missing Values Missing data is a fact of life. Sometimes a survey respondent skips a question, a sensor fails to record a reading, or a database simply loses a value during a transfer. In Python, the NumPy library represents these missing numerical values as NaN, which stands for "Not a Number." When you load data into a Pandas DataFrame, Pandas automatically converts empty cells or placeholder text (like "NA" or "NULL") into NaN. But before we can fix missing values, we have to find them. The isnull() Function Pandas provides a built-in function called isnull() to help us locate missing data. When you call isnull() on a DataFrame, it returns a DataFrame of the exact same shape, but instead of showing your data, it shows True or False. A True means the value is missing (NaN), and a False means the value is present. Let's look at a practical example. Imagine we have a DataFrame named orders containing customer data: Output: Looking at a massive grid of True and False values is not very efficient for large datasets. You would quickly lose track of where the missing values actually are. Combining isnull() with sum() To get a bird's-eye view of our missing data, we can chain the isnull() function with the sum() function. In Python, the boolean value True is mathematically treated as 1, and False is treated as 0. If we apply sum() to a column of True/False values, Pandas will add them up. The total will equal the …
8. Data Transformation and Manipulation
Why Reshape Your Data? Imagine you are an analyst for an online electronics retailer. You have just finished importing a dataset of 50,000 customer transactions and cleaning up the missing values. But before you can answer your manager’s question—which premium products are driving the most revenue among our loyal customers?—you need to isolate the relevant transactions, calculate the total price paid, and arrange the results from highest to lowest. Raw, clean data is rarely in the exact format you need for analysis. Data transformation is the process of changing the structure, format, or values of your data to make it suitable for a specific analytical task. In this chapter, we will use the Pandas library to filter rows, sort data, calculate new metrics, and rename columns. By the end, you will be able to take a cleaned DataFrame and mold it to answer specific business questions. Filtering Rows Based on Conditions When you want to look at a specific subset of your data—such as transactions from a single country, or customers over a certain age—you need to filter your DataFrame. In Pandas, filtering works by creating a Boolean mask. A Boolean mask is simply a Series of True and False values that corresponds to the rows in your DataFrame. When you pass this mask back into your DataFrame using square brackets, Pandas keeps the rows marked True and discards the rows marked False. Single Conditions Let's look at a practical example. Imagine we have a DataFrame named salesdf representing our electronics transactions, which includes a column called Category. To find only the transactions where the product category is "Laptops", we use the equality operator (==) to create the mask, and then apply it: You can also write this in a single, more concise line of code, which is the standard practice among Python data analysts: Multiple Conditions Often, a single condition isn't enough. You might want to find transactions for "Laptops" that also cost more than $1,000. To combine multiple conditions, we use Python's logical operators. Recall from the Control Flow and Logic chapter that Python uses and, or, and not to combine conditions. However, when working with Pandas DataFrames, we must use special bitwise logical operators: & (ampersand) replaces and | (pipe) replaces or Crucial Rule: When combining multiple conditions, you must wrap each individual condition in parentheses. If you forget the parentheses, Python will get confused by the order of operations and throw a SyntaxError or a ValueError. Let's filter for laptops costing more than $1,000: If we wanted to find transactions that are either Laptops or Smartphones, we would use the pipe operator: Sorting DataFrames Once you have the right subset of data, you usually want to …
9. Data Aggregation and Grouping
Why Summarize Data? Imagine you are handed a spreadsheet containing 50,000 individual sales transactions from a national electronics retailer. Each row represents a single purchase: the date, the store location, the product category, and the revenue generated. If someone asks you, "What was our total revenue for laptops in New York last month?", looking through the raw data row by row is impossible. In Pandas Fundamentals, we learned how to view and filter individual rows. In Data Transformation and Manipulation, we learned how to create new columns and sort data. But neither of those skills helps us summarize 50,000 rows into a single, readable number. To answer questions about total revenue, average sales, or item counts, we need to aggregate our data. Aggregation is the process of taking many data points and combining them into a single summary value. A common example of an aggregation function is sum, which adds numbers together, or mean, which calculates the average. To aggregate data effectively, we usually split it into groups first. For instance, before we can calculate the total revenue, we must group the data by store location and product category. This "split-apply-combine" workflow is one of the most powerful features of the pandas library. The groupby Method The core tool for grouping data in pandas is the groupby method. To understand how it works, let's look at a small, simplified dataset representing a few days of sales at a coffee shop. If we want to know how many total cups of each product were sold over these three days, we need to group the data by the Product column. Grouping by a Single Column When you apply the groupby method to a DataFrame, you create a DataFrameGroupBy object. This is a special intermediate object. It doesn't display data like a normal DataFrame; instead, it holds the instructions for how the data is split. Output: Think of this object as a dictionary where the keys are the unique categories (in this case, "Espresso" and "Latte") and the values are the corresponding rows. The Split-Apply-Combine Workflow To actually see the summarized data, we need to apply an aggregation function to this grouped object. This completes a three-step process: 1. Split: pandas breaks the DataFrame into separate groups based on the unique values in the Product column (one group for Espresso, one for Latte). 2. Apply: We apply an aggregation function—like sum—to the CupsSold column within each separate group. 3. Combine: pandas takes those individual summary results and combines them back into a new, summarized DataFrame or Series. Output: Notice the syntax: df.groupby('Product')['CupsSold'].sum(). We first group by the category, then select the specific numeric column we want to summarize using bracket notation, and finally …
10. Data Visualization Basics
Why Visualize Data? Imagine you are handed a spreadsheet containing 50,000 rows of daily temperature readings from a weather station over the last century. If you look at the raw numbers in a Pandas DataFrame, your brain can process maybe the first 20 rows before the numbers start blurring together. You could use the Data Aggregation and Grouping techniques from the previous chapter to calculate the average temperature for each decade. You might even find that the average temperature in the 2010s was 1.2 degrees higher than in the 1910s. But if you plot those 50,000 data points on a single graph, the upward trend of global warming becomes instantly, viscerally apparent. A visualization communicates patterns, trends, and outliers in milliseconds—far faster than a table of numbers ever could. Data visualization is the graphical representation of information and data. By using visual elements like charts, graphs, and maps, we can make complex data accessible and understandable. In this module, we will learn how to create these visual representations using two of Python's most popular libraries: Matplotlib and Seaborn. Introducing Matplotlib and Seaborn Throughout this book, we have relied on libraries like NumPy and Pandas to handle our data. To visualize that data, we need new tools. Matplotlib is the foundational plotting library in Python. It is incredibly versatile, meaning you can customize almost every single pixel of your chart. However, because it is so powerful, it can sometimes require a lot of code to make a chart look exactly how you want. Seaborn is a library built on top of Matplotlib. It acts as a wrapper, meaning it uses Matplotlib under the hood but provides a simpler interface. Seaborn is specifically designed for statistical data visualization and comes with beautiful default themes right out of the box. A common workflow for Python data analysts is to use Matplotlib for basic, highly customized charts, and Seaborn for complex statistical graphics that look great with minimal code. Before we can draw anything, we need to import these libraries into our Python script. By convention, Matplotlib's main plotting module is imported as plt, and Seaborn is imported as sns. Note: If you are running this code locally in an IDE like VS Code and get a "ModuleNotFoundError," you will need to install these libraries by running pip install matplotlib seaborn in your terminal. Creating Line Charts with Matplotlib A line chart connects a series of data points with a continuous line. It is the best choice for visualizing data over time (often called a time series) because it clearly shows how a value changes. To create a basic line chart in Matplotlib, we use the plt.plot() function. We pass our x-axis values as …
11. Exploratory Data Analysis Project
The End-to-End Analysis Journey Imagine you are handed a spreadsheet containing 50,000 rows of online product reviews. The columns include dates, star ratings, text comments, and locations. On their own, these rows are just text and numbers. But somewhere inside that spreadsheet is a story: perhaps customers in certain regions are consistently unhappy with shipping times, or maybe a specific product feature is driving five-star ratings. Finding that story is the goal of Exploratory Data Analysis (EDA). EDA is the process of examining a dataset to uncover its main characteristics, often using visual methods. It is how you transform raw, messy data into actionable insights. Throughout this book, you have built an impressive toolkit. You learned Python Foundations and Control Flow and Logic to write instructions. You explored Core Data Structures and the Introduction to NumPy to store and compute numerical data. You mastered Pandas Fundamentals to work with tabular data, and you learned how to bring that data into your environment through Importing and Exporting Data. You then learned how to fix messy data through Data Cleaning and Preparation, reshape it via Data Transformation and Manipulation, summarize it using Data Aggregation and Grouping, and finally, illustrate it using Data Visualization Basics. Now, in this final module, we are going to stitch all of those isolated skills together. We will walk through a complete, end-to-end EDA project, from the initial spark of a question to a final, communicable insight. Formulating a Guiding Question Raw data without a purpose is just noise. Before you write a single line of Pandas code, you need to know what you are trying to achieve. In data analysis, this is called formulating a guiding question. A guiding question gives your analysis direction. Without it, you might find yourself creating dozens of charts and calculating endless statistics without ever answering anything useful. A good guiding question is specific, measurable, and answerable with the data you have. Broad vs. Specific Questions Imagine you are working with a dataset of e-commerce sales. You might start with a broad curiosity: “How are our sales doing?” While this is a natural starting point, it is too vague to guide an analysis. Does "how" refer to total revenue, number of items sold, or profit margins? Are we looking at this month, this year, or all time? To make it actionable, refine it into a specific question: “Did total revenue from electronics increase in the fourth quarter of 2023 compared to the third quarter?” This refined question tells you exactly what to do: 1. You need to filter the data for the "electronics" category. 2. You need to filter the dates for Q3 and Q4 of 2023. 3. You need to aggregate …
Continue learning
- Intermediate Python Projects for Portfolio BuildingIntermediate Python Projects for Portfolio Building — a free intermediate-level guide covering intermediate python projects for portfolio building....
- Python Programming for Beginners: Zero to Real ProjectsPython Programming for Beginners: Zero to Real Projects — a free beginner-level guide covering learn python programming from zero to real projects....
- Python Programming for Beginners: From Zero to Real ProjectsPython Programming for Beginners: From Zero to Real Projects — a free beginner-level guide covering learn python programming from zero to real...
- Python for Beginners: From Zero to Real-World ProjectsPython for Beginners: From Zero to Real-World Projects — a free beginner-level guide covering learn python programming from zero to real projects....