Free Programming learning guide

Intermediate Python Projects for Portfolio Building

Name: Intermediate Python Projects for Portfolio Building
Author: Tanmay Kalbande

Intermediate Python Projects for Portfolio Building — a free intermediate-level guide covering intermediate python projects for portfolio building....

130 min read14 chaptersintermediate

What you will learn

Setting Up a Professional Python Development Environment
Building a RESTful API with FastAPI
Creating a Web Scraper with BeautifulSoup and Scrapy
Developing a Data Processing Pipeline with Pandas
Building a Command Line Tool with Click or Typer
Creating a Data Visualization Dashboard with Plotly Dash
Implementing a Task Scheduler with Celery and Redis
Developing a Machine Learning Model with Scikit-learn
Building a Real-time Chat Application with WebSockets
Creating a Document Processing System with Python
Developing a Multi-threaded Web Crawler
Creating a Password Manager with Encryption
Building a Real-time Data Processing System
Deploying Python Applications to Cloud Platforms

1. Setting Up a Professional Python Development Environment

Why a Professional Setup Matters Before Your First Line of Code You’ve probably heard the saying: "A craftsman is only as good as their tools." In software development, this rings especially true. A messy development environment leads to inconsistent code, hard-to-debug issues, and friction in collaboration. Worse, it can silently undermine your portfolio projects before you even write a single function. Consider this scenario: You’re building a FastAPI application for your portfolio, and it’s time to share the code. You’ve written great endpoints, but the project lacks a requirements.txt that works across environments. A teammate clones it, runs pip install -r requirements.txt, and the app crashes because one dependency isn’t pinned correctly. Or worse—your formatter and linter aren’t set up, so the code is inconsistent, and reviewers notice style issues more than your logic. This isn’t hypothetical. It happens to every intermediate Python developer who skips environment setup in favor of "just getting it working." A professional setup isn’t just about aesthetics—it’s about reproducibility, maintainability, and respect for future you and your collaborators. We’re going to build a development environment that scales from your first script to a complex portfolio project. You’ll learn how to isolate dependencies, enforce code quality automatically, and structure your project like a professional. By the end of this chapter, you’ll have a repeatable blueprint you can apply to every project in this book—and beyond. --- Setting Up the Foundation: Python and Tooling Before we configure anything, ensure you have a modern Python installation and the right tools. Install Python Properly Use Python 3.10 or higher. Avoid system-wide installations if you’re on Linux or macOS—use a version manager. - Windows: Download from python.org or use Chocolatey: - macOS: Use pyenv to avoid conflicts: - Linux: Use pyenv as well: ⚠️ Note: On Linux, you may need to install build dependencies: Verify installation: --- Install Core Tooling You’ll need three foundational tools: 1. Poetry – for dependency management and packaging 2. Git – for version control 3. Pre-commit – for automated code quality checks Install them globally (they’re tools, not project-specific): 💡 Why Poetry? Unlike pip + virtualenv, Poetry handles dependency resolution, virtual environments, and project metadata (like pyproject.toml) in one place. It’s the modern standard for professional Python projects. ✅ Use --user to avoid messing with system Python. You can also use pipx for better isolation: --- Creating a Project with Poetry Let’s scaffold your first project using Poetry. This will generate a professional structure with a pyproject.toml file—Python’s modern alternative to setup.py. Initialize the Project The --src flag creates a src/ layout, which is the recommended structure for serious projects. It keeps your package code separate from tests and scripts. Your directory should look like: …

2. Building a RESTful API with FastAPI

From Zero to Production: Building a FastAPI Service That Stands Up to Real Use Imagine you’re at a hackathon. You’ve spent the weekend building a prototype that scrapes GitHub trending repos, enriches the data with language popularity, and serves it through a React dashboard. The prototype works—until 500 users hit your single /trending endpoint simultaneously and your Flask app collapses under the load. The dashboard freezes. The judges walk away. The VPs from the sponsor company ask, “Can you make this resilient?” That’s the moment you realize: prototypes are for learning, production systems are for scale. FastAPI isn’t just a framework—it’s your on-ramp from prototype to production. It combines Python’s async power with automatic OpenAPI docs, Pydantic validation, and SQLAlchemy integration, letting you ship a RESTful API that handles thousands of requests per second without boilerplate sprawl. This chapter walks you through building a production-grade FastAPI service from scratch—one that uses Alembic for migrations, includes pagination, filtering, and sorting, and ships with pytest coverage. We’ll focus on a realistic scenario: a public API for tracking open-source contributions. We’ll call it “ContribTrack.” It will let users: - Register and authenticate - Submit GitHub usernames - Fetch aggregated contribution stats (commits, PRs, issues) - Filter, paginate, and sort results - Export data as CSV By the end, you’ll have a FastAPI app that looks and feels like a real SaaS backend—not a tutorial toy. --- Setting Up the Project with Poetry We’ll start where Chapter 1 left off. If you haven’t already, initialize a new Poetry project: This gives us: - fastapi with Starlette under the hood - uvicorn as the ASGI server (async-ready) - sqlalchemy for ORM - alembic for migrations - pytest and httpx for async testing - python-jose for JWT auth - passlib and bcrypt for password hashing - python-multipart for form/file uploads We’ll structure the project as: This structure separates concerns: - api/v1 contains all route handlers - schemas holds Pydantic models for requests/responses - core holds app configuration and security utilities - db/models contains SQLAlchemy models - services isolates business logic from routes - tests uses pytest fixtures and async httpx client --- Defining the Data Model with SQLAlchemy Let’s define two core models: User and Contribution. In src/contribtrack/db/models/user.py: In src/contribtrack/db/models/contribution.py: The base.py file simply defines the Base class: --- Configuring the Database Engine and Session We’ll use async SQLAlchemy with asyncpg for PostgreSQL. Add to pyproject.toml: In src/contribtrack/db/session.py: Note: settings.DATABASEURL should be defined in core/config.py as: Use environment variables in production—never commit secrets. --- Creating Alembic Migrations Initialize Alembic: Edit migrations/env.py to use our async engine and models: Create the first migration: This creates the users and contributions tables in PostgreSQL. --- Defining Pydantic Schemas for …

3. Creating a Web Scraper with BeautifulSoup and Scrapy

From Static Pages to Dynamic Content: Building Robust Web Scrapers The first time you scrape a website that loads its content dynamically via JavaScript, your BeautifulSoup script will return an empty result set. That moment when your scraper fails silently against a modern site is the same moment developers decide whether to pivot to more advanced tools—or abandon the project entirely. This chapter will turn that failure into a learning opportunity by showing you how to handle both static and dynamic content while keeping your scrapers ethical, maintainable, and scalable. You’ll move beyond simple HTML parsing into real-world challenges like rate limiting, user-agent rotation, and storage optimization—skills that directly transfer to professional data collection pipelines. Whether you're extracting product details from an e-commerce site, monitoring job postings, or archiving news articles, the techniques here will help you build scrapers that work today and adapt to future site changes. --- Scraping Static Sites with BeautifulSoup BeautifulSoup excels at parsing HTML and XML documents, making it ideal for static sites where content is embedded directly in the page source. Unlike dynamic sites that rely on JavaScript to render content, static sites serve fully formed HTML that your scraper can process immediately. Setting Up the Environment Ensure you have the required packages installed: These tools give you: - BeautifulSoup: Parse and navigate HTML/XML - Requests: Fetch web pages - lxml: A fast HTML/XML parser backend for BeautifulSoup Create a new directory for this project and initialize it with Poetry: Note: Use the same project structure introduced in Setting Up a Professional Python Development Environment—separate src/ for source code, tests/ for tests, and pyproject.toml with proper linting and type checking. --- A Real-World Example: Scraping Book Listings Let’s scrape a static book listing page. We’ll use Books to Scrape, a safe, legal sandbox for practicing scraping. Step 1: Fetch the Page Always check the response status. A 403 or 429 means you’ve been blocked. Step 2: Extract Book Data Each book is in an article with class productpod. We’ll extract: - Title - Price - Availability - Rating Use .select() and CSS selectors for precision. Avoid brittle class names—prefer semantic or structural patterns. Step 3: Store the Data This gives you a clean CSV ready for analysis or visualization in later modules. --- Handling JavaScript-Rendered Content with Selenium Static sites are rare today. Modern sites often load content via JavaScript after the initial HTML loads. BeautifulSoup can’t execute JavaScript, so we turn to Selenium, a browser automation tool. Selenium controls a real browser (e.g., Chrome, Firefox), allowing you to scrape pages that render dynamically. Installing Selenium webdriver-manager auto-downloads the correct browser driver. --- Example: Scraping a Dynamic Job Board Let’s scrape a job listing …

4. Developing a Data Processing Pipeline with Pandas

From Messy Spreadsheets to Clean Insights: Building a Real-World Data Pipeline Imagine inheriting a dataset that looks like this: a collection of CSV files from three different departments, each with inconsistent date formats, missing values, and a mix of numeric and textual data in the same column. The timestamps? Some are in ISO format, others as "October 15, 2023" or "15/10/23". The product IDs? They started with letters but were later changed to just digits. And don’t get started on the “Notes” column, where analysts pasted everything from emails to emojis. This isn’t hypothetical—it’s the daily reality of data practitioners. The difference between a failed analytics project and a successful one often comes down to how well you clean and structure the data before analysis. This chapter isn’t about basic filtering or simple cleaning tricks. It’s about building a robust, reusable, and production-grade data processing pipeline using pandas—one that can handle real-world mess, scale across multiple datasets, and be trusted to produce consistent, validated outputs. You’ll learn how to turn chaotic spreadsheets and fragmented logs into a clean, structured dataset ready for modeling, visualization, or API consumption. You’ve already set up a professional Python environment using Poetry, Black, isort, and pre-commit hooks in earlier chapters. These tools aren’t just for code—they’re essential for maintaining clean, version-controlled data pipelines. Just as you wouldn’t deploy a FastAPI app without type hints and linting, you shouldn’t process data without validation and reproducibility. --- Designing a Data Pipeline That Scales A data pipeline isn’t just a script you run once—it’s a repeatable process that transforms raw data into clean, structured information. Think of it like a factory assembly line: raw materials go in, quality checks happen at each stage, and only validated products move to the next step. Before writing code, ask: What does "clean" mean for this dataset? The definition depends entirely on the use case. A dataset for a machine learning model might tolerate more missing values than one used in financial reporting. A dashboard might require consistent date ranges, while a backend API might need normalized identifiers. Let’s define a pipeline architecture using three core stages: 1. Ingestion: Load data from multiple sources (CSV, Excel, JSON, APIs) 2. Transformation: Clean, normalize, aggregate, and validate 3. Validation & Export: Ensure quality and save to a standard format This modular structure lets you test and debug each part independently. It also makes the pipeline reusable—swap out the input file, and the same cleaning logic applies. --- Example 1: Cleaning a Sales Log with Multiple Inconsistencies Let’s walk through a real-world example: a sales log from an e-commerce platform that’s been manually maintained for years. The file has 5,000 rows and several known issues: …

5. Building a Command Line Tool with Click or Typer

Why Command Line Tools Matter in Python Development Consider the tools you use daily: git, curl, docker, pip. These aren't just utilities—they're force multipliers. They allow developers to express complex operations in a few keystrokes, automate workflows, and build systems that integrate seamlessly into larger ecosystems. When building Python applications, the command line interface (CLI) often serves as the primary entry point for users, from fellow developers to DevOps engineers. A well-designed CLI tool doesn't just save time—it reduces cognitive load. Instead of navigating through a web interface or sifting through configuration files, users express intent directly: mytool analyze --input data.csv --output report.pdf. This declarative style aligns with the Unix philosophy: "Make each program do one thing well," and "Expect the output of every program to become the input to another." In this chapter, you'll build a professional-grade CLI tool using Click or Typer, two of Python’s most powerful libraries for creating maintainable, user-friendly command line interfaces. You’ll move beyond basic argument parsing into subcommands, validation, progress tracking, and even distribution—all while learning patterns that scale from small scripts to tools used by thousands. --- Setting the Stage: The "FileSync CLI" Project To ground your learning, let’s build a real-world CLI tool called FileSync CLI. This tool helps developers synchronize files between directories, with support for filtering, logging, and progress reporting. It’s inspired by tools like rsync, but tailored for Python developers working in local or CI environments. You’ll implement features such as: - Syncing files from a source to a target directory - Filtering by file type or pattern - Dry-run mode for safety - Verbose and colorful output - Subcommands for common operations (e.g., filesync diff, filesync clean) - Input validation and graceful error handling By the end of this chapter, you’ll have a fully functional tool that can be installed via pip install filesync-cli, with a professional structure and polished user experience. --- Choosing Between Click and Typer Both Click and Typer are built on top of argparse but offer modern, Pythonic APIs with decorators and type hints. The key difference lies in design philosophy: | Feature | Click | Typer | |--------|-------|-------| | API Style | Decorator-based | Decorator-based with strong use of type hints | | Python Version | Python 2.7+ | Python 3.6+ | | Learning Curve | Moderate | Lower (for type-hint users) | | Best For | Complex, nested CLIs | Modern, clean CLIs with strong typing | | Type Safety | Limited | Strong (built on pydantic) | | Subcommands | Supported | First-class support | | Dependency | Standalone | Requires click, rich, pydantic | Since this book assumes Python 3.10+, Typer is the natural choice for its modern, maintainable …

6. Creating a Data Visualization Dashboard with Plotly Dash

A Dashboard That Turns Data Into Decisions Imagine a product manager who must decide whether to double‑down on a new feature. She pulls a CSV of daily sign‑ups, queries a PostgreSQL table for churn metrics, and checks a public API for competitor pricing. With three separate tools—Excel, a SQL client, and a browser tab—she spends an hour stitching together a story. Now picture the same manager opening a single web page that instantly shows a line chart of sign‑ups, a bar chart of churn by cohort, and a live sparkline of competitor prices, all filtered by a date range picker. She slides the picker, the charts update in real time, and a summary card flashes the projected revenue impact. That one page is a Plotly Dash dashboard. In this chapter you’ll build exactly that kind of interactive, data‑driven interface, learning how to: Assemble a minimal Dash app with core components. Wire interactivity using callbacks and state. Pull data from CSV files, SQL databases, and external APIs. Apply professional styling with Dash Bootstrap Components (DBC). Deploy the finished product to a cloud platform, ready for a portfolio showcase. --- Project Skeleton – Leveraging the Environment You Already Built Your development workflow from the Setting Up a Professional Python Development Environment chapter already includes: Poetry for dependency management. A src/ package layout (src/dashboard/). Pre‑commit hooks that run Black, flake8, and isort. Create a new project inside the same workspace: The resulting directory should look like: Tip: Keep the same src/ convention you used for the FastAPI project. It makes the repository instantly recognizable to recruiters and aligns with modern Python packaging best practices. --- Core Dash Building Blocks Dash apps are essentially Flask servers that render a React front‑end. The two pillars you’ll work with are: | Pillar | Role | |--------|------| | Layout | Declares the static structure (HTML‑like components). | | Callbacks | Connects user actions (inputs) to updates (outputs). | 1. Defining the Layout Create src/dashboard/layout.py: Dash Bootstrap Components give you ready‑made containers, rows, and columns that follow the Bootstrap grid system. Each component gets a unique id. Those IDs become the hooks for callbacks. 2. Bootstrapping the App src/dashboard/app.py ties everything together: suppresscallbackexceptions=True lets you define callbacks that reference components that may be generated dynamically later. Running python -m src.dashboard.app launches the demo at http://127.0.0.1:8050. --- Wiring Interactivity – Callbacks, Inputs, Outputs, and State Dash callbacks are Python functions decorated with @app.callback. The decorator lists Inputs, Outputs, and optionally State (values that do not trigger the callback but are needed for computation). 1. Simple Callback Example Add to src/dashboard/callbacks.py: What’s happening? 1. Inputs (startdate, enddate, region-filter) fire the callback whenever the user changes them. 2. The function loads …

7. Implementing a Task Scheduler with Celery and Redis

Why Do You Need a Scheduler When Your API Is Already Fast? Imagine a SaaS platform that ingests user‑uploaded CSV files, validates the data, and sends a confirmation email. The upload endpoint returns instantly, but the heavy lifting—parsing, validation, and email dispatch—can take several minutes. If you push those steps into the request‑response cycle, users experience time‑outs and the server’s concurrency suffers. A task scheduler solves this problem by off‑loading work to background workers, letting the API stay responsive while guaranteeing that jobs run reliably, on schedule, and can be retried when they fail. In this chapter you’ll wire up Celery with Redis as a broker, define periodic tasks with Celery Beat, add robust retry logic, monitor everything through Flower, and scale the system across multiple workers—all within the project layout you already built for the FastAPI chapter. --- 1. Project Set‑up Revisited Your repository already follows the production‑grade layout introduced earlier: Add a new package for task‑related code: 1.1 Install Celery and Redis Client with Poetry Tip: If you prefer a separate virtual environment for workers, you can create a dev extra in pyproject.toml and install with poetry install -E dev. The rest of the book uses the same environment for the API and workers, keeping things simple for a portfolio project. 1.2 Run a Local Redis Instance - Docker: docker run -d -p 6379:6379 redis:7-alpine - macOS (Homebrew): brew install redis && brew services start redis - Windows (Chocolatey): choco install redis-64 Confirm connectivity: --- 2. Wiring Celery to Redis 2.1 Create the Celery Application tasks/celeryapp.py Why reuse settings? Your FastAPI config already stores the Redis URL (see chapter Building a RESTful API with FastAPI). This keeps a single source of truth for environment‑specific values such as credentials or hostnames. 2.2 Declare a Simple Task tasks/tasks.py Key points - bind=True gives the task access to its own request (self), needed for retries. - maxretries limits the number of attempts. - defaultretrydelay is the base delay; we override it with exponential back‑off inside the except block. 2.3 Hook the Task into FastAPI Add an endpoint that enqueues the job: The endpoint returns instantly with the Celery task ID, which the client can later use to query status (covered in a later chapter on monitoring). --- 3. Scheduling Recurring Jobs with Celery Beat Many background jobs are periodic: cleaning stale sessions, sending daily digests, or pruning old files. Celery Beat is a lightweight scheduler that reads a dictionary of intervals and launches tasks accordingly. 3.1 Define the Beat Schedule tasks/beatschedule.py 3.2 Implement the Scheduled Tasks 3.3 Run Celery Beat Celery Beat will now emit the two tasks at the configured intervals. You can combine Beat with the worker process (see …

8. Developing a Machine Learning Model with Scikit-learn

From Raw CSV to Deployable API: A Hands‑On Scikit‑learn Journey A small real‑estate startup wants to give its agents instant price estimates for residential listings. All they have is a CSV file containing historic sales (square footage, number of bedrooms, neighborhood, year built, etc.). Your task: turn that raw file into a REST endpoint that returns a price prediction in under a minute. The steps below walk you through exactly that—starting with data cleaning, moving through feature engineering, model training, hyper‑parameter tuning, and finally persisting the model and wiring it up to the FastAPI service you built in Chapter 2. --- 1. Project Layout – Leveraging the Poetry Template Reuse the production‑grade repository structure introduced earlier: Why this layout? It isolates data handling, model logic, and API code, making each piece testable and version‑controlled—exactly the pattern you practiced with pre-commit and static typing in earlier chapters. --- 2. Loading & Splitting the Data Tip: Use pyright (static type checker from Chapter 1) to verify the signatures above. --- 3. Preprocessing & Feature Engineering 3.1 Handling Missing Values 3.2 Encoding Categorical Variables 3.3 Scaling Numerical Features 3.4 Creating New Features Why these features? - Age captures depreciation. - Price per sqft normalizes the target, useful for linear models. - Bed‑bath ratio often correlates with luxury vs. efficiency. 3.5 Building a Unified ColumnTransformer All preprocessing now lives inside a single, reusable object that can be dropped into any scikit‑learn pipeline. --- 4. Model Selection – Trying a Few Algorithms 4.1 Quick Evaluation with Cross‑Validation Running evaluate() on the three models usually reveals that tree‑based ensembles (RF, GB) beat the plain linear regression on this dataset, but the linear model is still valuable as a baseline. --- 5. Hyper‑Parameter Tuning with GridSearchCV 5.1 Defining a Search Space 5.2 Running the Grid Search The returned bestestimator is a fully fitted pipeline ready for persistence. Tip: Keep the scoring metric aligned with the business goal. If agents care about “average error” more than “squared error”, switch to negmeanabsoluteerror. --- 6. Persisting the Trained Model 6.1 Using joblib (recommended) joblib is faster for large NumPy arrays than pickle and handles compression out of the box. 6.2 Verifying the Saved Model Run this after loading to ensure the pipeline still works end‑to‑end. --- 7. Exposing the Model via a FastAPI Endpoint Recall: Chapter 2 showed you how to spin up a FastAPI service. Now we’ll plug the persisted model into that service. 7.1 Endpoint Code Key points - The Listing schema enforces input validation (type, range). - addfeatures is reused from the training module to guarantee identical transformations. - The model is loaded once at import time, keeping request latency low. 7.2 Running & Testing Test with …

9. Building a Real-time Chat Application with WebSockets

Why Real‑time Chat Matters Imagine a support desk where customers can type a question and see an agent’s reply instantly, without the page reloading. Or a collaborative coding platform where teammates discuss a bug while watching each other’s edits in real time. The magic behind these experiences is bidirectional, low‑latency communication—exactly what WebSockets provide. Unlike traditional HTTP request/response cycles, a WebSocket connection stays open, allowing the server to push data to the client the moment something happens. Building a real‑time chat app is an excellent way to showcase asynchronous programming, connection management, and secure, scalable design—all attractive skills for a Python portfolio. --- Choosing the Right WebSocket Stack | Option | Pros | Cons | |--------|------|------| | FastAPI + WebSocket (built‑in) | • Seamlessly integrates with existing FastAPI routes <br• Same dependency injection, validation, and docs <br• Leverages starlette’s mature ASGI implementation | • Slightly more boilerplate for pure‑WebSocket logic | | websockets library | • Minimalist API focused solely on WebSockets <br• Good for learning the protocol fundamentals | • Separate server from any HTTP API you already have <br• Requires manual routing for authentication, static files, etc. | For this chapter we’ll use FastAPI because the book already introduced it in Chapter 2, and the same project can expose both REST endpoints (e.g., for user signup) and WebSocket routes side‑by‑side. The approach, however, works just as well with websockets if you prefer a leaner stack. --- Project Scaffold Leverage the Poetry‑based template introduced earlier: Tip: Run poetry install to create an isolated environment, then poetry run uvicorn app.main:app --reload to start the development server. This mirrors the workflow from the earlier “production‑ready structure” chapter. --- Implementing the WebSocket Server Connection lifecycle FastAPI treats a WebSocket route like any other endpoint, but the handler receives a WebSocket object instead of a Request. The typical flow: 1. Accept – tells the client the handshake succeeded. 2. Authenticate – we’ll validate a JWT (see the auth chapter). 3. Register – add the socket to a manager that tracks users per room. 4. Loop – await websocket.receivetext() blocks until a message arrives. 5. Cleanup – on WebSocketDisconnect remove the socket from the manager. Managing connections with a ConnectionManager A central class keeps track of every open socket, allowing us to broadcast efficiently: The manager is singleton‑like: instantiate it once in app/ws/init.py and import it wherever needed. --- Broadcasting Messages With the manager in place, broadcasting becomes a one‑liner: Real‑world example 1 – Global lobby If roomid is "lobby" for every user, the broadcast call distributes each message to all connected clients, creating a classic group chat. Real‑world example 2 – Private chat When users join a room named after a unique conversation …

10. Creating a Document Processing System with Python

Why Document Processing Matters Imagine a small law firm that receives dozens of contracts, court filings, and scanned letters every week. Every new document must be searchable, its key clauses indexed, and any deadlines flagged—otherwise the firm risks missing critical dates. Manually opening each PDF, copying text into a spreadsheet, and hunting for a due‑date is a recipe for error and lost billable hours. A document processing system built with Python can automate the entire workflow: Pull raw text from PDFs and Word files. Run OCR on scanned images to unlock hidden text. Extract dates, contract numbers, and parties using regex/NLP. Store the results in a searchable database behind a FastAPI endpoint. The following sections walk you through a production‑ready implementation that you can add to your portfolio and extend for any domain that works with documents. --- Project Architecture Overview Leverage the production‑grade structure introduced when we built the FastAPI API earlier: Poetry continues to manage dependencies and virtual environments. FastAPI provides the HTTP layer for uploading documents and searching results. SQLite with FTS5 (or Elasticsearch for larger deployments) offers full‑text search. All code lives under src/docproc/, keeping the public API clean and the internal modules isolated—a pattern you already adopted in the earlier FastAPI chapter. --- Setting Up Dependencies Tesseract OCR is an external binary, not a Python package. Install it per OS: | OS | Command | |----|---------| | macOS | brew install tesseract | | Ubuntu/Debian | sudo apt-get install tesseract-ocr | | Windows | Download the installer from the official repo and add its bin directory to PATH. | Add a pre‑commit hook to enforce Black, flake8, and isort (already set up in the repo template) so the new modules stay clean. --- PDF Extraction Strategies Choosing the Right Tool | Library | Strength | When to Prefer | |---------|----------|----------------| | PyPDF2 | Fast, works on most PDFs, simple API | Text‑only PDFs, no need for layout details | | pdfplumber | Retains layout, can extract tables and exact coordinates | PDFs with complex columns, tables, or mixed text/image content | Both libraries are pure Python, so they play nicely with Poetry. Example: Extracting Text from a Multi‑Page PDF Usage The function returns a clean string ready for downstream regex/NLP processing. --- Word Document Handling with python-docx Reading, Modifying, and Saving Example: Pulling Out Tables from an Invoice You can now feed the raw text or the tabular data into the same extraction pipeline used for PDFs. --- OCR for Scanned Documents Installing Tesseract Make sure the tesseract command is reachable from the shell (tesseract -v should print the version). Using pytesseract Tip: Tesseract performs best on 300 dpi images. If you encounter low‑quality scans, …

11. Developing a Multi-threaded Web Crawler

Why Crawl When You Can Conquer Imagine you need to monitor the latest articles on dozens of tech blogs, extract their headlines, and feed them into a recommendation engine that updates every hour. Manually visiting each site is impossible; a well‑behaved, high‑performance crawler can do the heavy lifting while staying polite to the servers it visits. This chapter walks you through building exactly that: a multi‑threaded web crawler that respects robots.txt, throttles itself appropriately, avoids duplicate work, and persists the harvested data in a database ready for downstream analytics. --- Project Scaffold – Leveraging What You Already Built If you followed the earlier chapters on FastAPI, Poetry, and the production‑grade project layout, you already have a solid foundation: - Poetry for dependency management (poetry init → poetry add requests beautifulsoup4 sqlalchemy aiosqlite) - A src/ package with clear separation (crawler/, models/, utils/) - Pre‑commit hooks (black, flake8, isort) ensuring clean code Create a new module under src/crawler/ called engine.py. This will house the core crawling logic, while src/models/ will contain the SQLAlchemy ORM definitions. Keeping the structure consistent with previous chapters makes the codebase instantly familiar and ready for future expansion (e.g., exposing a FastAPI endpoint that triggers a crawl). --- 1. The Bare‑Bones Crawler Before we add concurrency, let’s implement a simple, single‑threaded crawler that fetches a page and extracts links. What this snippet demonstrates - Requests for HTTP handling. - BeautifulSoup for parsing HTML. - A frontier list that drives breadth‑first traversal. - Simple duplicate avoidance via the visited set. Run it to see the crawler traverse a single domain. The next sections will make it polite, fast, and persistent. --- 2. Playing Nice – robots.txt and Crawl Delays Webmasters publish a robots.txt file to declare which parts of a site are off‑limits to crawlers. Ignoring it can get your IP banned or, worse, land you in legal trouble. Python ships with urllib.robotparser, a lightweight parser that we can wrap in a utility class. Key points - Caching: We keep a parser per host to avoid repeated network calls. - Thread safety: A simple Lock protects the shared dictionaries. - Default delay: If the site omits Crawl‑delay, we fall back to a courteous 1 second pause. Integrate the guard into the crawler: Now the crawler automatically backs off according to each site’s policy, dramatically reducing the risk of being blocked. --- 3. Detecting Duplicates – Normalization & Bloom Filters A naive set works for small crawls, but as the frontier grows into the hundreds of thousands of URLs, memory consumption spikes. Two techniques keep duplicate detection lightweight: 1. URL Normalization – strip query parameters that don’t affect content (e.g., ?ref=twitter), sort remaining parameters, and lowercase the scheme/host. 2. …

12. Creating a Password Manager with Encryption

A Breach You Could Have Prevented You walk into a coffee shop, pull out your laptop, and open a spreadsheet that contains every login you’ve ever created—plain‑text passwords, email addresses, and security‑question answers. A barista glances over, spots the file, and you both know exactly what could happen next. If you had a password manager that stored those credentials encrypted with a master password derived from a strong key‑stretching algorithm, the data would be useless to anyone without that one secret. Building such a tool yourself not only eliminates the human‑error risk of reusing passwords, it also gives you full control over the cryptographic choices and the data lifecycle. Below we’ll walk through a complete, production‑ready password manager written in Python 3.10+. The design leans on the tooling and best‑practice foundations you’ve already set up in earlier chapters (Poetry for dependency management, Black & isort for formatting, type hints for static analysis, etc.), and it adds a secure CLI, strong password generation, strength checking, encrypted SQLite storage, and optional sharing and recovery features. --- 1. Threat Model & Design Goals | Threat | Mitigation | |--------|------------| | Database theft – attacker gains raw file | Encrypt the entire payload with AES‑256‑GCM; store only the ciphertext and a nonce. | | Master‑password brute‑force | Derive the encryption key with a memory‑hard KDF (Argon2id) or a proven PBKDF2 implementation with high iteration count. | | In‑memory snooping | Zero out sensitive variables after use (del, ctypes.memset). | | Credential sharing abuse | Encrypt shared entries with a public‑key (RSA‑OAEP) so only the intended recipient can decrypt. | | Loss of master password | Offer a recovery token generated from a secret‑sharing scheme (Shamir) that can reconstruct the master key. | The manager will therefore: 1. Prompt for a master password each session. 2. Derive a symmetric key using Argon2id (fallback to PBKDF2 if Argon2 not available). 3. Store credentials in an SQLite database, with each row encrypted individually. 4. Provide a Typer‑based CLI for CRUD operations, password generation, and strength feedback. 5. Include optional sharing and recovery commands. --- 2. Project Skeleton Leverage the Poetry template you adopted in Setting Up a Professional Python Development Environment: Directory layout (trimmed for brevity): All modules are type‑annotated and formatted with Black; static analysis with mypy ensures the cryptographic API is used correctly. --- 3. Cryptographic Core 3.1 Key Derivation Why Argon2id? It deliberately consumes both CPU and RAM, making large‑scale brute‑force attempts expensive. The fallback to PBKDF2 with 200 k iterations still offers substantial work factor for most attackers. 3.2 Symmetric Encryption AES‑GCM provides confidentiality and integrity in a single primitive, eliminating the need for separate MAC handling. --- 4. Data Model & Encrypted …

13. Building a Real-time Data Processing System

The Real‑World Hook: Detecting Anomalous Transactions as They Happen Imagine a fintech startup that must flag potentially fraudulent credit‑card transactions within seconds of their arrival. A delayed batch job is not an option; the business needs a pipeline that ingests each event, enriches it, runs a sliding‑window risk score, and pushes the result to a live dashboard for analysts. Building such a pipeline from scratch is a perfect way to showcase a real‑time data processing system and, more importantly, to add a compelling project to your portfolio. The remainder of this chapter walks you through exactly that pipeline—using either Apache Kafka or RabbitMQ as the transport layer, processing streams with pure Python, performing windowed aggregations, visualising the flow live, and preparing the whole stack for cloud deployment. --- 1. Choosing and Bootstrapping the Message Broker Both Kafka and RabbitMQ are battle‑tested, but they differ in semantics: | Feature | Kafka | RabbitMQ | |---------|-------|----------| | Model | Append‑only log, partitioned topics | Queue‑based, exchanges & routing keys | | Ordering | Guarantees per‑partition order | No built‑in ordering (can be enforced with single consumer) | | Retention | Time‑ or size‑based retention, replay possible | Messages removed once acknowledged | | Throughput | Very high (hundreds of k msgs/s) | Good for moderate loads, easier to set up for small projects | For a portfolio project, Docker Compose gives you a one‑click environment for either broker. Below is a minimal docker-compose.yml that spins up both options; you’ll comment out the service you don’t need. Tip: The project’s pyproject.toml (see the Why Poetry? chapter) already includes docker as a dev dependency, so you can run poetry run docker compose up -d from the repository root. 1.1 Creating Topics / Queues Both commands are idempotent; running them repeatedly won’t cause errors. --- 2. Producing a Stream of Sample Data A realistic data generator helps you test the whole pipeline. The following script uses confluent-kafka for Kafka and pika for RabbitMQ. It emits JSON‑encoded transaction records every 200 ms. Run it with poetry run python -m src.producer. The script demonstrates production‑ready patterns you already saw in the FastAPI chapter—environment‑driven configuration, graceful shutdown (add a signal handler), and JSON serialization. --- 3. Consuming and Processing the Stream Processing can be done with several libraries, but for a pure‑Python approach that still offers windowed semantics, Faust (Kafka Streams) is ideal. For RabbitMQ, we’ll build a lightweight async consumer using aio-pika and manually manage windows. 3.1 Faust‑Based Processor (Kafka) Run the app with poetry run python -m src.processorfaust. Faust automatically handles stateful windows, checkpointing, and graceful restarts—concepts you touched on when building the Task Scheduler with Celery. 3.2 Async Consumer with Manual Windows (RabbitMQ) The RabbitMQ …

14. Deploying Python Applications to Cloud Platforms

From Local Prototype to Global Service You’ve just finished the FastAPI‑based news aggregator that scrapes headlines with the multi‑threaded crawler from Chapter 11, stores results in PostgreSQL, and offers a clean JSON API. Locally it runs with poetry run uvicorn app.main:app. Now a client asks: “Can you make this run 24/7, handle spikes, and give us logs in real time?” The answer lies in three tightly coupled steps: 1. Package the code in a reproducible container 2. Push the container to a cloud service that can scale 3. Wire the whole thing into an automated CI/CD pipeline with monitoring The sections below walk you through each step, using the same codebase you built earlier. By the end you’ll be able to ship any of the projects from previous chapters—FastAPI APIs, password‑manager back‑ends, or real‑time processing pipelines—into production with a single command. --- 1. Containerizing Python Applications with Docker 1.1 Why Docker Matters for Python Isolation: No more “works on my machine” surprises—every dependency lives inside the image. Portability: The same image runs on your laptop, an EC2 instance, or a Cloud Run service. Speed: Layers are cached, so rebuilds after a small code change are near‑instant. 1.2 Building a Minimal, Multi‑Stage Dockerfile Because you already use Poetry for dependency management, the Dockerfile can stay lean: Key points Multi‑stage build strips out build‑time packages (gcc, curl) from the final image, reducing size from ~600 MB to <150 MB. Poetry’s virtualenvs.create false installs directly into the global site‑packages, avoiding an extra layer of indirection. gunicorn + uvicorn gives you a robust production server with graceful reloads and configurable worker counts. 1.3 Local Build & Test Visit http://localhost:8000/docs—the same OpenAPI UI you generated in Chapter 2—to verify the container works. --- 2. Choosing a Cloud Platform 2.1 Quick Comparison | Feature | AWS (ECS/Fargate) | GCP (Cloud Run) | Azure (Container Apps) | |-----------------------------|-------------------|-----------------|------------------------| | Pricing model | Pay‑per‑vCPU‑second | Pay‑per‑request + CPU‑seconds | Pay‑per‑vCPU‑second | | Managed DNS | Route 53 | Cloud DNS | Azure DNS | | Built‑in logging | CloudWatch Logs | Cloud Logging | Azure Monitor Logs | | Serverless container | Fargate (no servers) | Cloud Run (fully serverless) | Container Apps (serverless) | | Native CI/CD integration| CodeBuild + CodePipeline | Cloud Build | Azure DevOps Pipelines | | Best for | Complex networking, VPC‑level control | Simple HTTP services, quick scaling | Tight integration with Microsoft ecosystem | All three providers support Docker images stored in a container registry (ECR, Artifact Registry, or ACR). Pick the one that matches your existing cloud footprint; the steps below focus on AWS because it offers the most granular control for multi‑service architectures (e.g., a FastAPI front‑end plus a …