Free Science learning guide
Machine Learning with Python: Practical Basics
Machine Learning with Python: Practical Basics — a free intermediate-level guide covering basics of machine learning with python. Learn with clear...
What you will learn
1. Data Preprocessing and Feature Engineering
The "Garbage In" Reality Imagine deploying a churn prediction model that achieves 95% accuracy during development, only to watch it fail spectacularly in production. The culprit isn't the algorithm; it's the data. The training set contained duplicate rows, the categorical variables were improperly encoded causing the model to treat zip codes as continuous integers, and a missing value flag inadvertently became the strongest predictor for churn. Machine learning models are mathematical functions—garbage in, garbage out. While algorithms capture the headlines, the reality is that data preprocessing and feature engineering dictate the ceiling of your model's performance. A well-preprocessed dataset fed into a simple logistic regression model will almost always outperform a poorly prepared dataset fed into a state-of-the-art ensemble. Transforming raw, messy data into a model-ready format involves four critical steps: handling anomalies and missing values, encoding categorical variables, scaling numerical features, and strategically splitting your data. Handling Missing Values and Encoding Categoricals Real-world data is incomplete and messy. Before a model can mathematically optimize an objective function, the data must exist in a complete, numeric matrix. Strategies for Missing Data Missing values typically fall into three categories: Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR). Understanding the mechanism dictates your strategy. For intermediate workflows, dropping missing values (df.dropna()) is rarely the first choice, as it discards valuable information. Instead, we rely on imputation. Using scikit-learn's SimpleImputer allows for robust, pipeline-compatible imputation: Numerical features: Impute with the mean (sensitive to outliers) or median (robust to outliers). Categorical features: Impute with the mode (most frequent) or a constant placeholder like "Missing". For more advanced scenarios, multivariate imputation (like IterativeImputer in scikit-learn) models each feature with missing values as a function of other features, yielding highly accurate imputations at the cost of computational complexity. Encoding Categorical Variables Machine learning models require numerical input. Categorical variables must be transformed, but the method depends entirely on the cardinality of the feature and the algorithm being used. 1. Ordinal Encoding: Used when categories have an intrinsic order (e.g., "Low", "Medium", "High"). Scikit-learn's OrdinalEncoder maps categories to integers (0, 1, 2). 2. One-Hot Encoding: Used for nominal variables with low cardinality (e.g., "Red", "Green", "Blue"). It creates a binary sparse matrix. 3. Target Encoding: Used for high-cardinality nominal variables (e.g., zip codes, product IDs). This replaces the category with the mean of the target variable for that category. Caution: This causes severe data leakage if not strictly handled within cross-validation folds. Setting handleunknown='ignore' ensures that if the model encounters a new platform in production (e.g., "Tablet"), it will output all zeros for the one-hot encoded columns rather than throwing an error. Scaling and Normalizing Numerical Features Distance-based algorithms (like …
2. Linear Models and Regularization
The Linear Foundation Imagine you are predicting the price of a house. You have features like square footage, number of bedrooms, and distance to the nearest school. If you assume that each feature contributes independently and additively to the final price—a constant price per square foot, a fixed premium for each bedroom, a specific discount per mile from the school—you are already thinking in linear terms. Linear models are the workhorses of machine learning. While they might lack the flashy predictive power of complex algorithms, they are fast, interpretable, and serve as the essential baseline for both regression and classification tasks. In Module 1, we meticulously engineered our features, applied Standardization (StandardScaler) to handle differing scales, and established our Training Set: and Validation Set: while strictly adhering to The Golden Rule: of preventing data leakage. Now, we put those pristine features to work by training linear models and using regularization to keep them robust. Linear Regression and the Cost Function At its core, a linear regression model fits a straight line (or hyperplane in multiple dimensions) through the data. The model is defined by weights (coefficients) assigned to each feature, plus an intercept (bias). The goal of the algorithm is to find the optimal weights that minimize the difference between the predicted values and the actual target values. This is achieved using a cost function—typically Ordinary Least Squares (OLS)—which calculates the sum of squared errors between predictions and actuals. The algorithm iteratively adjusts the weights to minimize this error. Let’s implement a baseline linear regression model using scikit-learn. For this scenario, we will predict a continuous target variable based on a set of preprocessed numerical and one-hot encoded categorical features. Mathematical Assumptions and Limitations Linear regression is mathematically elegant, but its simplicity imposes strict assumptions. If your data violates these, the model's performance and interpretability will degrade: Linearity: It assumes a linear relationship between features and the target. If the true relationship is highly non-linear, the model will underfit. Independence: Observations must be independent of one another. Homoscedasticity: The variance of the residuals (errors) should remain constant across all predicted values. If errors fan out as predictions increase, the model's confidence intervals become unreliable. Multicollinearity: Features should not be highly correlated with one another. If you included both "square footage" and "square meters" as features, the model struggles to assign weights appropriately, leading to unstable coefficients. Logistic Regression for Classification Despite its name, logistic regression is the standard baseline model for binary classification tasks. Instead of fitting a straight line through data points, it fits an S-shaped logistic function (a sigmoid) that bounds the output between 0 and 1, representing the probability of an instance belonging to the positive class. …
3. Tree-Based Models and Ensembles
The Limits of Linearity Imagine you are building a model to predict housing prices. You have a feature representing the size of the house in square feet. A Linear Model assumes that the relationship between square footage and price is constant: every additional square foot adds the exact same amount of value to the house, whether you are expanding a 500-square-foot apartment or a 5,000-square-foot mansion. In reality, relationships are rarely this rigid. The value of an additional square foot might be massive for a tiny apartment, but negligible for a sprawling estate. The price might also jump discontinuously based on location—crossing a specific school district boundary could instantly add $50,000 to the value, regardless of square footage. Linear models, even with Binning or polynomial feature engineering, struggle to capture these abrupt, non-linear shifts and complex feature interactions naturally. Tree-based models solve this by asking a series of "if-then" questions. Instead of fitting a global line or plane, they recursively partition the feature space into distinct regions, allowing them to naturally capture step-functions, thresholds, and complex interactions without manual feature engineering. Decision Trees: Structure and Splitting A Decision Tree is a supervised learning algorithm that partitions data into subsets based on feature values. It consists of nodes (where a feature is evaluated) and branches (the outcome of the evaluation, leading to the next node). The process starts at the root node and ends at a leaf node, which contains the final prediction. How Trees Split To build a tree, the algorithm searches through all features and their possible split points to find the one that results in the "purest" resulting subsets. Purity is measured differently depending on the task: 1. Classification: Trees use Gini Impurity or Entropy. A perfectly pure node is one where all samples belong to a single class. The algorithm chooses the split that minimizes the impurity of the child nodes. 2. Regression: Trees use Mean Squared Error (MSE) or Mean Absolute Error (MAE). The split chosen is the one that minimizes the variance (or absolute deviation) of the target variable within the child nodes. Because the algorithm evaluates every feature at every possible threshold, trees require no feature scaling. A StandardScaler or MinMaxScaler has absolutely no effect on a tree's performance, making them highly robust to outliers and varied feature distributions. The Overfitting Problem Left unconstrained, a decision tree will continue to split until every leaf node contains exactly one sample (or samples with identical target values). This results in zero training error. However, as we established in earlier chapters, a model with zero training error is almost certainly overfitting. It has memorized the training data, capturing the underlying signal and the random noise. To prevent …
4. Unsupervised Learning Techniques
Discovering Structure in the Void Imagine you are handed a massive dataset of customer transactions for a newly acquired e-commerce platform. There are no labels. No one has flagged these customers as "high value," "churn risk," or "bargain hunter." Your task is not to predict a specific target variable, but to understand the landscape of the customer base so the marketing team can design targeted campaigns. In the previous chapters, we explored Linear Models and Tree-Based Models to predict outcomes—whether predicting continuous numbers or classifying records into known categories. These were supervised learning tasks, guided by labeled Training Sets and Test Sets. Now, we remove the training wheels. Unsupervised learning drops the target variable entirely. Our goal shifts from prediction to discovery: grouping similar entities together (clustering) and simplifying complex, high-dimensional data into digestible forms (dimensionality reduction). Clustering: Grouping Similar Data Points Clustering algorithms mathematically segment data points into groups where intra-group similarity is high, and inter-group similarity is low. Because we lack a ground truth to validate against, clustering relies heavily on understanding the geometry of our data and the assumptions of our algorithms. K-Means Clustering K-Means is a centroid-based clustering algorithm. It assumes that clusters are convex (spherical) and roughly equally sized. The algorithm works iteratively: 1. You specify the number of clusters, $k$. 2. The algorithm randomly drops $k$ initial centroids into the feature space. 3. Every data point is assigned to its nearest centroid. 4. The centroid is recalculated as the mean of all points assigned to it. 5. Steps 3 and 4 repeat until the centroids stop moving (convergence). Because K-Means relies on distance calculations, it is highly sensitive to the scale of your features. You must apply Standardization (StandardScaler) or Normalization (MinMaxScaler) to your numerical features before feeding the data into the algorithm. Without scaling, a feature measured in thousands will completely dominate a feature measured in decimals. Let’s look at a practical application using Python and scikit-learn: In this scenario, K-Means will group the low-income/low-spending customers together, high-income/high-spending together, and middle-income/middle-spending together. However, K-Means has a critical vulnerability: it does not handle outliers well. Because centroids are calculated using the mean, a few extreme outliers can pull the centroid far away from the true center of a cluster. If your data contains extreme anomalies that you couldn't handle via Binning or RobustScaler during preprocessing, K-Means might produce distorted clusters. DBSCAN: Density-Based Clustering DBSCAN (Density-Based Spatial Clustering of Applications with Noise) takes a fundamentally different approach. Instead of requiring you to predefine the number of clusters, DBSCAN groups together points that are packed closely together, marking points in low-density regions as outliers (noise). DBSCAN relies on two parameters: - eps: The maximum distance between …
5. Model Evaluation and Hyperparameter Tuning
The Illusion of Accuracy Imagine you’ve built a classifier to detect a rare but fatal disease that affects 1% of the population. You train a logistic regression model on your data, test it, and achieve an astonishing 99% accuracy. Excited, you present the results to a clinical partner, only to be met with a blank stare. They point out a simple, humiliating flaw: a model that simply predicts "healthy" for every single patient—ignoring all features—would also achieve 99% accuracy. This scenario illustrates why raw accuracy is a dangerous metric for imbalanced datasets. Up to this point, we’ve focused heavily on how to build models—fitting linear regressions, tuning regularized models, and growing random forests. But building a model is only half the battle. Without a rigorous framework to evaluate performance and systematically optimize parameters, we are flying blind. Advanced Evaluation Metrics To properly evaluate models, we must match the metric to the business problem. Relying on accuracy obscures the nuances of model performance, particularly in classification and regression tasks where errors have different costs. Classification Metrics: Beyond Accuracy For classification, especially with imbalanced classes, we rely on the Confusion Matrix as our foundation. It breaks down predictions into four categories: - True Positives (TP): Correctly predicted positive cases. - True Negatives (TN): Correctly predicted negative cases. - False Positives (FP): Negative cases incorrectly predicted as positive (Type I error). - False Negatives (FN): Positive cases incorrectly predicted as negative (Type II error). From this matrix, we derive two fundamental metrics: 1. Precision: $TP / (TP + FP)$. When the model predicts positive, how often is it correct? High precision minimizes false alarms. 2. Recall (Sensitivity): $TP / (TP + FN)$. Out of all actual positive cases, how many did the model find? High recall minimizes missed detections. In our medical scenario, recall is paramount—missing a diseased patient (FN) is far worse than a false alarm (FP). Conversely, in an email spam filter, precision is prioritized—sending a crucial work email to the spam folder (FP) is worse than letting a single spam email into the inbox (FN). Because improving precision often hurts recall (and vice versa), we use the F1-score to find a balance. The F1-score is the harmonic mean of precision and recall: $$F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$$ The harmonic mean punishes extreme values. If your recall is 0.0, your F1-score is 0.0, regardless of how high your precision is. This makes F1 an excellent single-number metric for imbalanced classification. The ROC Curve and AUC While F1-score evaluates a model at a specific probability threshold (usually 0.5), the Receiver Operating Characteristic (ROC) curve evaluates the model across all possible thresholds. The ROC curve plots the True Positive …
6. Machine Learning Pipelines and Model Persistence
The Leakage Problem Imagine you've just spent two weeks perfecting a churn prediction model. You meticulously handled Missing Completely at Random (MCAR) and Missing at Random (MAR) patterns using multivariate imputation, applied One-Hot Encoding to your categorical features, used StandardScaler to standardize your numerical data, and tuned a Gradient Boosting ensemble to achieve an impressive 92% accuracy on your Validation Set. But when you deploy the model to production, its performance tanks. What happened? The culprit is almost always data leakage. During development, it is easy to accidentally fit your scaler or imputer on the entire dataset before performing cross-validation. The model effectively "peeks" at the validation data's distribution during preprocessing, leading to overly optimistic metrics that crumble when faced with truly unseen data. The solution to this, and to the general chaos of managing multiple preprocessing and modeling steps, is the scikit-learn Pipeline. Constructing Scikit-learn Pipelines A Pipeline chains multiple data transformations and a final estimator into a single object. By bundling your workflow, you guarantee that the exact same preprocessing steps applied to your Training Set are applied to new data during inference, eliminating the risk of manual synchronization errors. Basic Pipeline Mechanics A Pipeline is instantiated with a list of (name, transformer/estimator) tuples. Every step except the last must be a transformer (having fit and transform methods), and the last step must be an estimator (having a fit method). When you call numericalpipeline.fit(Xtrain, ytrain), the pipeline sequentially fits the imputer, transforms the training data, passes it to the scaler, fits the scaler, transforms the data, and finally fits the classifier. Calling numericalpipeline.predict(Xtest) automatically routes the new data through the same imputer and scaler before generating predictions. Orchestrating Complex Workflows with ColumnTransformer Real-world datasets are messy, containing a mix of numerical, categorical, and ordinal features. As established in earlier chapters, different feature types require different engineering approaches. Applying One-Hot Encoding to a continuous numerical variable, or StandardScaler to a nominal category, is a recipe for disaster. The ColumnTransformer allows you to apply different transformations to specific subsets of columns in parallel, combining the results into a single output array. Combining ColumnTransformer and Pipeline The most powerful reproducible workflows nest Pipeline objects inside a ColumnTransformer, or vice versa. Let's look at a realistic scenario: predicting housing prices based on numerical features, categorical features, and ordinal features. Why this architecture shines: - Safety: Data leakage is structurally prevented. The SimpleImputer medians and StandardScaler means are learned only from the training folds during cross-validation. - Reproducibility: The entire workflow—from mode imputation to ensemble predictions—is encapsulated in a single variable (fullpipeline). - Tunability: You can pass the entire pipeline into GridSearchCV or RandomizedSearchCV. To tune a hyperparameter inside the pipeline, you use …
7. Introduction to Neural Networks with Keras
From Linear Models to Neural Networks Imagine you are trying to predict customer churn. You have meticulously cleaned the data, handled missing values using multivariate imputation, and applied One-Hot Encoding to the categorical features. You start with a Logistic Regression model, but the decision boundary is strictly linear. You move to a Random Forest, which captures non-linear interactions beautifully but behaves like a black box. What if you could build a model that learns complex, non-linear relationships directly from the data, composing features in hierarchical layers? This is the domain of neural networks. In the previous modules, we built a strong foundation in scikit-learn, exploring everything from Regularization to Machine Learning Pipelines. Now, we transition to deep learning. While scikit-learn provides some basic neural network implementations (like MLPClassifier), the industry standard for building neural networks in Python is Keras, the high-level API for the TensorFlow library. Keras allows us to define, compile, and train complex neural networks with just a few lines of intuitive code. Defining Architectures with the Keras Sequential API The simplest way to build a neural network in Keras is using the Sequential API. As the name implies, this API allows you to build a model layer by layer, sequentially. A standard feedforward neural network (also known as a Multi-Layer Perceptron) consists of three main parts: 1. Input Layer: Implicitly defined by the shape of your training data. 2. Hidden Layers: The core of the network where mathematical transformations occur. 3. Output Layer: Produces the final prediction. Let’s look at a concrete scenario: predicting housing prices based on 8 numerical features (like in the California Housing dataset). We will build a regression model. Understanding the Code Dense: This defines a fully connected layer, meaning every neuron in the previous layer is connected to every neuron in the current layer. units: The number of neurons in the layer. Choosing the right number is a hyperparameter tuning problem. A common strategy is to start with a power of 2 (e.g., 64, 128) and taper down in subsequent layers. inputshape: Only the first layer needs this. It tells the network the dimensionality of the input data. Here, (8,) represents a 1D array of 8 features. activation: The function applied to the output of the layer. We will explore this next. Configuring Activation Functions, Optimizers, and Loss Functions Once the architecture is defined, the model must be compiled. Compilation configures the model for training by specifying three critical components: the optimizer, the loss function, and (optionally) metrics. Activation Functions Activation functions introduce non-linearity into the network, allowing it to learn complex patterns. Without them, a deep neural network would simply collapse into a single linear transformation, offering no advantage over the …
Continue learning
- Master Real-Time Satellite Image Processing With Python To Track DeforestationMaster Real-Time Satellite Image Processing With Python To Track Deforestation — a free intermediate-level guide covering master real-time satellite...
- Build A Scalable, AI-Driven Platform For Real-Time Urban Noise Pollution Mapping And Mitigation StrategiesBuild A Scalable, AI-Driven Platform For Real-Time Urban Noise Pollution Mapping And Mitigation Strategies — a free intermediate-level guide covering...
- Build And Deploy A Real-Time AI Sign Language Translation App With Python And OpenCVBuild And Deploy A Real-Time AI Sign Language Translation App With Python And OpenCV — a free intermediate-level guide covering build and deploy a...
- Build A Local AI-Powered Semantic Search Engine For Your Personal Video ArchiveBuild A Local AI-Powered Semantic Search Engine For Your Personal Video Archive — a free advanced-level guide covering build a local ai-powered...