Pustakam Library

Free Programming learning guide

Intermediate Python Automation Scripts for Beginners

Intermediate Python Automation Scripts for Beginners — a free intermediate-level guide covering intermediate python automation scripts for beginners....

103 min read11 chaptersintermediate

What you will learn

  1. 1. Automation Environment Setup
  2. 2. OS Interaction and File System Automation
  3. 3. Web Scraping with Requests & BeautifulSoup
  4. 4. Browser Automation with Selenium
  5. 5. Scheduling and Running Scripts
  6. 6. Data Handling: CSV, Excel, JSON, and Databases
  7. 7. Email and Messaging Automation
  8. 8. Logging, Error Handling, and Debugging
  9. 9. Command‑Line Interfaces and Argument Parsing
  10. 10. Packaging, Distribution, and Deployment
  11. 11. Best Practices and Maintainability

1. 1. Automation Environment Setup

Why a Clean Automation Playground Matters Imagine you’ve just written a script that logs into a web portal, pulls a daily report, and emails it to the team. The next morning, the script crashes because a new version of pandas introduced a breaking change. Or worse, the script works on your laptop but fails on the CI server because the server still runs Python 3.7 while you developed with 3.11. A well‑crafted environment isolates your code from such “dependency drift” and gives you the confidence to iterate quickly. This chapter shows you how to build that playground from the ground up, using the two most popular tools—venv and conda—and how to equip your editor with the right linting and debugging aids. --- 1. Choosing Between venv and conda Both tools create isolated Python runtimes, but they differ in scope and workflow. | Feature | venv (standard library) | conda (Anaconda/Miniconda) | |---|---|---| | Scope | Pure Python packages (no external binaries) | Python and non‑Python binaries (e.g., SQLite, OpenSSL) | | Installation | Comes with any Python ≥3.3 | Separate installer (conda command) | | Cross‑platform binary handling | Relies on pip wheels; may need system libs | Handles compiled libraries automatically | | Size | Small (few MB) | Larger (tens of MB) due to bundled packages | | Command syntax | python -m venv <dir | conda create -n <name python=3.x | | Typical use case | Simple scripts, pure‑Python automation | Projects that need compiled libs (e.g., Selenium with ChromeDriver, pandas with C extensions) | Rule of thumb - Use venv when you only need pure‑Python packages and want the lightest footprint. - Reach for conda when you anticipate heavy scientific packages, need a specific version of a system library, or want a single command to set up the whole stack. You can even mix them: create a conda environment, then run python -m venv inside it to keep a pure‑Python layer for your project’s own code. --- 2. Setting Up a venv‑Based Project 2.1 Creating the Environment 2.2 Activating the Environment | OS | Command | |---|---| | Linux / macOS | source .venv/bin/activate | | Windows (cmd) | .venv\Scripts\activate.bat | | Windows (PowerShell) | .venv\Scripts\Activate.ps1 | When activated, your shell prompt typically shows the environment name, e.g., (.venv) $. All subsequent python and pip commands now operate inside this sandbox. 2.3 Installing Automation Libraries Tip: Pin exact versions once your script works, then freeze them: Later, anyone can reproduce the environment with: 2.4 A Mini‑Project: Daily Report Bot (venv) Create fetchreport.py: Running python fetchreport.py inside the activated venv will execute the whole workflow without touching the global Python installation. --- 3. Setting Up a conda‑Based …

2. 2. OS Interaction and File System Automation

Quick Scenario: The Daily Report Drop Imagine you are the “automation hero” for a small analytics team. Every night a third‑party system drops a ZIP file into a shared folder, containing a CSV of raw transactions, a PDF of a summary, and a binary log file. Your job is to: 1. Detect the new drop as soon as it appears. 2. Unpack the archive into a date‑stamped sub‑directory. 3. Move the original ZIP to an “archive” folder, deleting any files older than 30 days. 4. Parse the CSV, add a processing timestamp, and store a cleaned version in a separate folder. 5. Run a shell command that generates a PDF report from the CSV and capture any errors. All of this must run on Linux, macOS, and Windows without manual intervention. The rest of this chapter shows you, step by step, how to build that pipeline using only Python’s standard library—pathlib, os, csv, and subprocess. Why focus on the standard library? It’s always available, has no external dependencies, and works the same way across platforms. Later chapters will introduce powerful third‑party tools, but mastering the built‑ins gives you a solid foundation and keeps your scripts lightweight. --- Working with Paths: pathlib vs. os When to Reach for pathlib pathlib treats filesystem paths as objects rather than plain strings. This object‑oriented approach eliminates many of the string‑manipulation bugs that plague os.path code. Key benefits: - Auto‑normalization (/ works on Windows too). - Method chaining (path / "subdir" / "file.txt"). - Rich API (path.iterdir(), path.isfile(), path.stat()). Common File‑System Tasks | Task | pathlib | os (for reference) | |------|-----------|----------------------| | Create a directory (including parents) | path.mkdir(parents=True, existok=True) | os.makedirs(path, existok=True) | | List all files in a folder | for p in path.iterdir(): ... | for f in os.listdir(path): ... | | Rename / move a file | src.rename(dst) | os.rename(src, dst) | | Delete a file | path.unlink() | os.remove(path) | | Delete an empty directory | path.rmdir() | os.rmdir(path) | | Delete a non‑empty tree | shutil.rmtree(path) | shutil.rmtree(path) | Tip: When you need a quick one‑liner and you’re already using pathlib, prefer its methods. Reserve os for cases where you need low‑level access (e.g., os.stat flags). Example 1: Organizing Incoming Drops Why it works everywhere: Path.glob respects the underlying OS’s case‑sensitivity rules, and replace handles both Windows and POSIX rename semantics. --- Reading and Writing Files Efficiently Text Files: The open Context Manager Key points: - Use UTF‑8 explicitly; it avoids hidden locale issues on Windows. - shutil.copyfileobj streams data in chunks, reducing memory pressure for large files. - The temporary‑file‑then‑replace pattern prevents data loss if the script crashes mid‑write. CSV Files: The csv Module While pandas (introduced …

3. 3. Web Scraping with Requests & BeautifulSoup

Why Requests + BeautifulSoup Still Matter Imagine you need to pull the latest exchange rates from a financial portal that publishes a simple HTML table every morning. A quick glance tells you the data lives behind a static page—no JavaScript rendering, no login wizard, just a clean <table element. In a handful of lines you can retrieve the page, parse the table, and dump the numbers into a CSV for downstream analysis. Even with the rise of headless browsers and APIs, Requests combined with BeautifulSoup remains the most lightweight, transparent, and controllable toolset for such “read‑only” sites. It gives you full visibility into the HTTP conversation, lets you tweak headers or payloads on the fly, and keeps dependencies tiny—perfect for the intermediate automation scripts you’ll be building throughout this book. --- 1. Making HTTP Requests with Requests 1.1 The Basics: GET and POST requests.get(url, params=None, kwargs) sends a GET request. requests.post(url, data=None, json=None, kwargs) sends a POST request. Both return a Response object with attributes you’ll use throughout the chapter (statuscode, text, content, headers, cookies). 1.2 Custom Headers – Pretending to Be a Browser Many sites block “non‑browser” traffic by inspecting the User‑Agent header. Adding a realistic UA string is often enough to get past the first gate. Tip: Keep a short list of common UA strings in a JSON file and rotate them (see Section 6). 1.3 Query Parameters When a site expects search terms or pagination indexes in the URL, pass them via the params argument. Requests builds the query string for you and takes care of proper encoding. 1.4 POSTing Form Data vs. JSON Form‑encoded data (application/x-www-form-urlencoded) is the default when you pass a dict to data=. JSON payloads (application/json) are sent by using the json= argument. 1.5 Session Objects – Persisting Cookies & Connection Pool If you need to maintain a login session or reuse TCP connections, wrap your calls in a requests.Session. The session automatically stores cookies and reuses the underlying socket, improving speed and reducing overhead. Why it matters: Many sites issue a session cookie after the first request. Using a session prevents you from having to manually extract and resend that cookie on each call. --- 2. Parsing HTML with BeautifulSoup 2.1 Creating a Soup Object The parser choice (html.parser, lxml, html5lib) trades speed for tolerance. For most static pages, html.parser is sufficient and avoids extra dependencies. 2.2 Navigating the Parse Tree find(tag, attrs) returns the first match. findall(tag, attrs) returns a list of all matches. Both accept any HTML attribute as a keyword argument (id='main', href=True, etc.). 2.3 CSS Selectors – The select Family When you already know the CSS path to an element, selectone and select are concise. selectone('header h1') returns …

4. 4. Browser Automation with Selenium

1. Why Selenium? Imagine you need to test a new signup flow that populates a user’s dashboard via AJAX, or you must pull the latest price from a site that only renders after a button click. The requests + BeautifulSoup combo works great for static HTML, but once JavaScript starts shaping the page, you need a real browser. Selenium drives Chrome or Firefox just like a human would, giving you access to the live DOM, the ability to click, type, and wait for asynchronous updates. Quick win: With a few lines of code you can log into a portal, download a PDF, and capture a screenshot—all without ever leaving your editor. --- 2. Installing and Configuring WebDrivers 2.1 Choose the right driver | Browser | Driver name | Recommended version | |---------|------------|----------------------| | Chrome | chromedriver | Match the major version of Chrome | | Firefox | geckodriver | Works with any recent Firefox release | The Automation Environment Setup chapter already walked you through creating a virtual environment (venv or conda). Keep that environment active while you install Selenium: 2.2 Automatic driver management Manually downloading drivers is error‑prone. The webdriver-manager package handles version matching and path resolution: Sample factory function Tip: Store the driver instance in a context manager (with getdriver() as driver:) to guarantee proper shutdown—a pattern you’ll reuse throughout the book. --- 3. Managing the Driver Lifecycle 3.1 Start‑up and teardown Using the context manager: 3.2 Implicit vs. explicit waits Implicit wait tells Selenium to poll the DOM for a given time before throwing NoSuchElementException. It applies globally. Explicit wait targets a specific condition, such as element visibility or clickability. It’s more precise and avoids unnecessary delays. Best practice: Prefer explicit waits for dynamic pages; keep implicit waits short (1–2 s) to prevent hidden deadlocks. --- 4. Locating Elements Selenium offers several locator strategies. The most reliable are: | Strategy | Syntax example | |----------|----------------| | By.ID | driver.findelement(By.ID, "username") | | By.NAME | driver.findelement(By.NAME, "email") | | By.XPATH | driver.findelement(By.XPATH, "//button[text()='Submit']") | | By.CSSSELECTOR | driver.findelement(By.CSSSELECTOR, "input[type='password']") | | By.CLASSNAME | driver.findelement(By.CLASSNAME, "alert") | Pro tip: Start with By.ID or By.NAME. If those aren’t available, fall back to a concise CSS selector before resorting to brittle XPaths. 4.1 Example: Filling a login form --- 5. Interacting with Forms and Buttons 5.1 Text input and selection sendkeys() mimics typing. For <select elements, Selenium provides the Select helper. 5.2 Clicking, double‑clicking, and right‑clicking --- 6. Handling Alerts, Pop‑ups, and Frames 6.1 JavaScript alerts 6.2 Modal dialogs (HTML‑based) Treat them like any other element—just locate inside the dialog’s container and interact. 6.3 Switching frames --- 7. Waiting for JavaScript‑Generated Content Modern sites often load data via XHR after the …

5. 5. Scheduling and Running Scripts

When a Script Becomes a Daily Assistant Imagine you have built a Selenium crawler that logs into a supplier portal, downloads the latest inventory CSV, and stores it in a PostgreSQL table. The code works perfectly when you run it manually, but every morning at 3 AM the portal updates its data. Manually remembering to launch the script is error‑prone, and a missed run means stale inventory for the whole business day. The solution? Let the operating system launch your Python script automatically, exactly when you need it. Whether you’re on Linux, macOS, or Windows, the built‑in schedulers—cron and Task Scheduler—can turn a one‑off script into a reliable, unattended service. This chapter shows you how to write correct cron expressions, configure crontab entries, create Windows Task Scheduler tasks (both via the GUI and the command line), and use the lightweight schedule library for in‑script timing when a full OS scheduler isn’t required. --- 1. Cron on Linux and macOS 1.1 The Anatomy of a Cron Expression A cron line consists of six space‑separated fields: | Field | Meaning | Accepted values | |-------|---------|-----------------| | Minute | 0‑59 | 0‑59, /5 | | Hour | 0‑23 (24‑hour clock) | 0‑23, /2 | | Day of month | 1‑31 | 1‑31, ? | | Month | 1‑12 (or names) | 1‑12, Jan‑Dec | | Day of week | 0‑7 (Sun=0 or 7) | 0‑7, Mon‑Fri | | Command | The program to execute | any shell command | A classic “run at 2 AM every day” looks like: Special strings (@reboot, @daily, @weekly, @monthly, @yearly) are shortcuts for common schedules and improve readability. 1.2 Editing the Crontab The per‑user crontab is edited with: The editor defaults to vim or nano depending on the $EDITOR variable. Each line you add is a separate job; comments begin with . Tip: Keep a single crontab file under version control (e.g., ~/repo/cron/cron.txt) and install it with: That way you can track changes alongside your Python source. 1.3 Environment Gotchas Cron runs with a minimal environment. Variables you rely on in an interactive shell—PATH, PYTHONPATH, virtual‑env activation—are not automatically available. Explicit paths: Use absolute paths for the interpreter and any binaries. Activating a venv: Prefix the command with the activation script, e.g.: Logging: Redirect stdout and stderr to a file (as shown above) to capture traceback information. 1.4 Practical Example – Scheduling a Selenium Scraper Your Selenium script (inventoryscraper.py) lives in the ~/projects/inventory directory and uses the selenium and pandas packages installed in a virtual environment venv. To run it every weekday at 02:30 AM: 1. Create a wrapper script (runscraper.sh) that ensures the venv is active and sets a sensible HOME: Make it executable: chmod +x …

6. 6. Data Handling: CSV, Excel, JSON, and Databases

A Real‑World Trigger: When a Quarterly Report Needs a Make‑over Imagine you’ve been asked to deliver a quarterly performance report every Monday. The raw data arrives as a CSV export from the sales system, the finance team expects a polished Excel dashboard, and the senior leadership wants a JSON snapshot for their internal API. Manually juggling these formats wastes time and introduces errors. By mastering CSV, Excel, JSON, and SQLite handling, you can script the entire workflow—clean the data, archive it, and generate the exact outputs each stakeholder needs, all with a single Python run. --- 1. CSV Files – The Workhorse of Tabular Data 1.1 The csv Module: Quick‑and‑Dirty Reads The built‑in csv library is perfect for lightweight, line‑by‑line processing when you don’t need the full power of pandas. Why use DictReader? - Columns are accessed by name (row['price']) instead of index. - No hidden type conversion—everything comes in as a string, giving you full control over parsing. 1.2 Pandas for CSV: Load, Clean, Aggregate When the dataset grows beyond a few thousand rows, pandas shines. It automatically infers data types, handles missing values, and provides vectorised operations. Key pandas tricks for automation | Situation | pandas tip | |----------------------------------------|------------| | Large files ( 100 MB) | pd.readcsv(..., chunksize=50000) and iterate | | Need only a subset of columns | usecols=['orderid','price'] | | Preserve original column order when writing | df.tocsv(..., index=False, columns=originalorder) | 1.3 Practical Example – Cleaning a Sales Dump A nightly job receives salesraw.csv. The script below demonstrates a full pipeline: Outcome: a tidy CSV (salesclean.csv) ready for further analysis, and an in‑memory monthlytotals series you can feed into a report generator. --- 2. Excel Workbooks – When Presentation Matters 2.1 openpyxl: Direct Cell‑Level Manipulation openpyxl lets you modify existing workbooks, create formulas, and style cells without loading the entire file into memory. When to reach for openpyxl - Updating a template that already contains charts, images, or macros. - Adding or removing rows/columns without disturbing existing formatting. 2.2 Pandas + Excel: The Best of Both Worlds If you’re comfortable with pandas, you can read/write Excel files with a single line. Under the hood, pandas uses openpyxl (for .xlsx) or xlrd (for older .xls). 2.3 Practical Example – Updating a KPI Dashboard Suppose the finance team maintains a quarterly KPI workbook (kpitemplate.xlsx). Every week you need to inject the latest sales totals. Result: a ready‑to‑share Excel file with fresh numbers, while preserving any embedded charts or corporate branding. --- 3. JSON – The lingua franca for APIs and Configurations 3.1 The json Module: Load and Dump Tip: json.dump(..., ensureascii=False) keeps Unicode characters intact, useful for international data. 3.2 Dealing with Nested Structures Often API responses contain …

7. 7. Email and Messaging Automation

Sending Emails with SMTP – From “Hello World” to Attachments When a script finishes a nightly data‑dump, the next logical step is often “mail it to the team”. Python’s built‑in smtplib together with the email.mime hierarchy makes that possible without any external dependencies – a perfect match for the lightweight scripts you’ve already built in earlier chapters. 1. The Minimal “Hello‑World” Email Why the EmailMessage class? It automatically handles proper MIME headers, line‑breaks, and Unicode encoding, sparing you the low‑level MIMEText gymnastics you might have seen in older tutorials. 2. Adding Attachments – CSV, PDFs, Images Most real‑world notifications need a file. The addattachment method accepts raw bytes, so you can keep the data in memory (e.g. a pandas.DataFrame turned into CSV) without writing a temporary file. Tips that save headaches | Situation | Tip | |-----------|-----| | Gmail blocks “less secure apps” | Use an App Password (Google Account → Security → App passwords). | | Corporate SMTP requires a domain‑wide certificate | Pass context=ssl.createdefaultcontext() to SMTPSSL instead of starttls(). | | You need to attach a PDF generated on the fly | Use io.BytesIO to hold the PDF bytes, then call msg.addattachment(pdfbytes.getvalue(), maintype='application', subtype='pdf', filename='report.pdf'). | 3. Bulk Mail with a CSV List When the recipient list lives in a spreadsheet (a common scenario after a data‑cleaning step), you can reuse the CSV handling from Chapter 6. --- Automating Outlook & Exchange Many enterprises standardise on Microsoft Outlook or Exchange. Two popular routes exist: 1. win32com – works only on Windows, leverages the locally installed Outlook client. 2. Microsoft Graph API – REST‑based, works cross‑platform, requires an Azure AD app registration. Both approaches can be called from the same script, letting you pick the best tool for the deployment environment. 1. Outlook Automation via win32com.client Prerequisite – The pywin32 package is already available in the virtual environment you set up in Chapter 1. Real‑world usage Gotchas Outlook must be installed and configured for the user who runs the script. On headless servers (e.g., a CI runner) win32com will raise a COMError. In those cases, switch to the Graph route. 2. Microsoft Graph – A Cross‑Platform Solution The Graph API speaks JSON over HTTPS, so you’ll use requests (already in your toolbox). The heavy lifting is authentication; the easiest pattern for scripts is client credentials flow using a service principal. 2.1. Register an Azure AD Application 1. Azure portal → App registrations → New registration. 2. Set Supported account types to Accounts in this organizational directory only. 3. Under Certificates & secrets, create a client secret. Store it securely (e.g., in a .env file, loaded with python‑dotenv). 4. Grant the app Mail.Send permission under API permissions → Application permissions. …

8. 8. Logging, Error Handling, and Debugging

A Crash‑Course in Getting Real‑World Automation Scripts to “Talk” Imagine a nightly Python job that: 1. Launches a Selenium‑driven browser to pull a daily sales dashboard. 2. Parses the HTML with BeautifulSoup, builds a pandas DataFrame, and writes a CSV. 3. Sends the file to the sales team via the email‑automation routine you built in Chapter 7. 4. Is scheduled with cron (Linux/macOS) or Task Scheduler (Windows). At 02:00 AM the script aborts. The browser window disappears, the CSV is half‑written, and the sales team never receives the report. You discover the failure only when a colleague asks why the numbers are missing. Why did the script fail silently? Why was there no trace of the exception? Why didn’t the driver close cleanly, leaving a stray Chrome process? The answer lies in three tightly‑coupled practices: Robust logging – a permanent, searchable record of what the script did and why it stopped. Graceful exception handling – catching errors at the right level, cleaning up resources, and providing useful feedback. Effective debugging – using interactive tools and strategic log statements to reproduce and fix the problem quickly. The following sections walk you through configuring a production‑ready logging pipeline, designing exception‑aware code, and debugging with pdb, IDE breakpoints, and logging‑centric techniques. All examples build on the Selenium, email, and scheduling foundations you already have. --- 1. Logging Foundations for Automation 1.1 Why a Rotating File Log Is a Must‑Have Persistence – Console output disappears after the script ends; a file log survives for post‑mortem analysis. Size control – Long‑running jobs can generate megabytes of output. Rotating handlers automatically truncate old logs, preventing disk‑full crashes. Level‑based filtering – Separate debug chatter from error alerts, and route each to its own file if needed. Pro tip: In a scheduled environment you’ll rarely have a live terminal to watch print() statements. Treat the log file as the primary UI for your automation. 1.2 The Logging Module in a Nutshell logging.getLogger(name) – Retrieve (or create) a named logger. logger.setLevel(level) – The lowest severity the logger will handle. Handlers (StreamHandler, FileHandler, RotatingFileHandler, TimedRotatingFileHandler) – Direct log records to destinations. Formatters – Shape the textual (or JSON) representation of each record. 1.3 A Minimal Rotating‑File Setup The script now writes a compact, timestamped log that rolls over automatically. You can swap RotatingFileHandler for TimedRotatingFileHandler if you prefer daily or hourly rotation: 1.4 Controlling Verbosity with Log Levels | Level | Numeric | Typical Use | |-------|---------|-------------| | DEBUG | 10 | Fine‑grained diagnostic data (e.g., HTTP request bodies). | | INFO | 20 | High‑level progress (“Login successful”, “CSV written”). | | WARNING | 30 | Recoverable oddities (“Missing optional column”). | | ERROR | 40 | Failed operation …

9. 9. Command‑Line Interfaces and Argument Parsing

From Script to Super‑Tool: Turning a One‑Off Automation into a Reusable CLI Imagine you have a script that scrapes a product catalog, exports the data to a CSV, and emails the report to the sales team. You run it manually from your IDE, tweaking file paths and dates each time. Now a colleague asks for the same report but for a different date range and wants the output in JSON instead of CSV. Every time you edit the script, you risk breaking the logic you just verified. A command‑line interface (CLI) solves this problem. By exposing positional and optional arguments, adding sub‑commands for related tasks, and providing a clear help message, you transform a fragile notebook into a robust, shareable tool that can be invoked from any terminal—whether you schedule it with cron, launch it from a Windows Task Scheduler, or call it from another automation script. Below we’ll walk through building such a CLI with the two most popular Python libraries: argparse (in the standard library) and Click (a third‑party, decorator‑driven alternative). You’ll learn how to: Define positional and optional arguments, and group them into logical sub‑commands. Validate inputs (types, ranges, file existence) and surface helpful error messages. Generate automatic help and usage strings that guide users. Package your script as a console entry point so mytool can be run from anywhere on the system. The concepts build directly on earlier chapters—e.g., the Selenium script from Chapter 4 already uses a configuration file; we’ll now expose those configuration options on the command line. Logging set up in Chapter 8 will be reused for consistent diagnostics. --- 1. The Anatomy of a Good CLI A well‑designed CLI follows a predictable pattern that users (and your future self) can rely on: | Element | Description | Typical Placement | |---------|-------------|-------------------| | Program name | The executable that appears in usage strings. | Defined by the entry point or script filename. | | Positional arguments | Required inputs identified solely by order. | First after the program name (e.g., inputfile). | | Optional arguments | Flags or options prefixed with -/--. | Anywhere after the program name; can appear in any order. | | Sub‑commands | Independent actions grouped under the same top‑level program (e.g., upload, download). | Appear as the first positional argument after the program name. | | Help & version flags | -h/--help and -V/--version are auto‑generated by most parsers. | Global; available for every command or sub‑command. | | Error handling | Clear messages when required arguments are missing or malformed. | Triggered by the parser before your code runs. | Tip: Keep the interface descriptive yet concise. Users should be able to type mytool --help and instantly …

10. 10. Packaging, Distribution, and Deployment

Why Package Automation Scripts at All? You’ve spent weeks stitching together a script that logs into a web portal, scrapes a table, massages the data with pandas, and emails the results. When the script sits on a single laptop, you can run it manually or schedule it with cron (Chapter 5). But as the codebase grows, you’ll want to: Share the script with teammates without sending zip files or copy‑pasting code. Version the functionality so you can roll back if a change breaks the workflow. Reuse common helpers (e.g., a logging wrapper from Chapter 8) across several projects. Deploy the automation to cloud services where a dedicated VM isn’t justified. Packaging turns a collection of .py files into a first‑class Python library that can be installed with pip, bundled into a Docker image, or uploaded to a private PyPI server for internal consumption. The rest of this chapter shows you how to get from a working script to a reusable, distributable artifact and finally to a cloud‑native deployment. --- 1. Structuring a Reusable Automation Package A clean directory layout does most of the heavy lifting for you. Below is a minimal but production‑ready skeleton for a script called reporter that pulls data from a website (Chapter 4) and emails a CSV summary (Chapter 7). Keep runtime code inside the inner reporter/ directory. Place tests alongside the package but outside the importable namespace. Include a CLI (cli.py) that uses argparse (Chapter 9) and declares a console script entry point (see pyproject.toml). 1.1 Choosing Between setup.py and pyproject.toml Historically, setup.py was the sole way to declare a package’s metadata and build requirements. Modern tooling (PEP 517/518) prefers a static pyproject.toml that can be read without executing arbitrary code—safer for CI pipelines and private repositories. | Feature | setup.py | pyproject.toml | |-----------------------------|----------------------------------------|------------------------------------------| | Executable build script | Yes (arbitrary Python) | No (declarative) | | Build backend selection | Implicit (setuptools) | Explicit ([build-system]) | | Editable install (pip -e)| Works out‑of‑the‑box | Works, but requires setuptools backend | | Future‑proofing | Legacy, still supported | Recommended by the Python community | Recommendation: start with a pyproject.toml that uses setuptools as the build backend. You can still add a thin setup.cfg for static metadata if you prefer to keep the TOML tidy. Minimal pyproject.toml dependencies list the runtime packages you already imported in earlier chapters. optional‑dependencies let downstream users install only what they need (pip install reporter[aws]). The scripts entry creates a console command called reporter that invokes cli.main(). When you run python -m build (install build via pip install build), the command produces a wheel (.whl) and a source distribution (.tar.gz). Those artifacts are what you’ll upload to a private PyPI …

11. 11. Best Practices and Maintainability

A Real‑World Wake‑Up Call Imagine you’ve built a nightly script that: Scrapes a public web page with Selenium (Chapter 4) Transforms the data into a CSV using pandas (Chapter 6) Emails the report to stakeholders (Chapter 7) Is scheduled with cron on Linux and Task Scheduler on Windows (Chapter 5) It runs flawlessly for weeks—until a colleague adds a new column to the source page. The script crashes, the error lands in the log you set up (Chapter 8), and you spend an hour hunting down the problem. The root cause? The new column broke an implicit assumption in a helper function that had no tests, no docstring, and a line that violated PEP 8. If the script had been written with coding standards, automated tests, and proper documentation, the break would have been caught before deployment, and fixing it would have been a matter of updating a test or a docstring rather than debugging in production. The following sections give you the tools to turn that “nightmare” into a predictable, maintainable automation pipeline. --- 1. Coding Standards – Writing Clean, Consistent Code 1.1 Follow PEP 8, the de‑facto style guide | Area | Recommendation | Why it matters | |------|----------------|----------------| | Indentation | 4 spaces, never tabs | Guarantees that the same block is interpreted the same way on any editor. | | Line length | ≤ 79 characters (or 99 if you enable max-line-length = 99 in flake8) | Keeps code readable on terminals and in side‑by‑side diffs. | | Naming | snakecase for functions/variables, PascalCase for classes | Makes intent obvious; IDEs can auto‑complete correctly. | | Imports | Grouped as standard → third‑party → local, one per line, sorted alphabetically | Prevents circular imports and clarifies dependencies. | | Trailing whitespace | None | Diff noise is eliminated, making code reviews cleaner. | Tip: Use the black formatter (installed with pip install black) to automatically enforce most of these rules. Run it as part of your pre‑commit hook (see Chapter 10). 1.2 Linters – The First Line of Defense flake8 – Checks PEP 8 compliance, unused imports, and more. pylint – Provides a richer score and catches potential bugs (e.g., undefined variables). Integrating linters into your workflow 1. IDE integration – VS Code, PyCharm, and even vim have plugins that display lint warnings as you type. 2. Pre‑commit hook – Add a .pre-commit-config.yaml: Then run pre-commit install. Every git commit now runs flake8 first. 3. CI pipeline – In a GitHub Actions workflow, include: Failing lint checks block merges, ensuring the main branch stays clean. 1.3 Common Pitfalls and Quick Fixes Unused imports – flake8 flags them as F401. Remove them or prefix with if they’re …

Continue learning