Python for Software Engineering Bootcamp

This lecture guides writing the first service test for the user service, covering delete user, async testing with a test decorator, and introducing dependency injection via initialization parameters.

Learn to write unit tests that assert specific exceptions are raised, such as user not found, and build robust tests that cover edge cases and proper exception handling.

Learn how to send http requests in python using the requests library, including get and post examples, headers, data vs json, and synchronous versus asynchronous approaches.

Compare synchronous and asynchronous requests, install and use an asynchronous http library, and manage an http client session with async/await to avoid blocking your program.

Learn to test synchronous http requests in python by mocking external services with the responses library. Create a get_user function and simulate endpoint responses without real network calls.

Learn to test asynchronous requests by mocking predefined responses, using a configurable base url with endpoint prefix and user id, and asserting a 200 status and expected json payload.

Assess test coverage to see how much of your code is executed by tests, interpret coverage percentages, and uncover edge-case gaps while planning tests for every function and class.

Learn how to integrate mypy into your Python testing workflow, fix type notation errors, and ensure functions return the correct objects while validating with unit and integration tests.

Learn to move, rename, and copy files and directories using shell commands, use absolute and relative paths, copy contents with dot notation, and transfer data between machines.

Learn how to set and inspect environment variables with export and echo, organize them in local, staging, and production files, and load them with source to switch environments.

Learn to use history to review past commands, then pipe output into grep for targeted searches—even handling case sensitivity with the minus flag—and redirect results to a file.

Learn to run Python scripts from the command line, install Python with sudo apt, use which to check Python existence, and run test.py with Python 3.

Learn how to run executables from the terminal, create shell scripts, and modify file permissions with chmod, exploring user, group, and other rights and numeric permission codes.

Send requests from the terminal using curl to test APIs with get, post, put, and delete, including authorization headers and JSON data with proper content-type.

Learn how Docker creates isolated containers to provide consistent environments for Python applications, addressing dependency and version mismatches, and enabling easy deployment across machines.

Create and customize a makefile to shorten recurring docker compose commands. Define phony targets like start, stop, and unit tests, and enforce tab-based syntax.

Set up a local PostgreSQL server on Mac using the PostgreSQL app, start the server, and access the default three databases on your machine.

Connect to your local databases on mac with postico, set localhost on port 5432, create and delete databases, switch between template1 and Postgres, and write queries in the editor.

Explore using the SQL editor CLI to connect to a Postgres server via the terminal, write queries, and manage databases, with GUI options for flexibility.

Learn to create and drop databases within a data source, test connections to Postgres, and manage multiple databases using create and drop commands with semicolons.

Create and manage schemas to organize tables, drop and recreate schemas and databases as needed, and use introspection to verify the public schema is available.

Learn to create tables in database schemas with create table, define columns and data types, set primary keys, manage schemas, and establish foreign keys referencing related tables.

Alter tables to adapt your database as requirements change, adding, dropping, or modifying columns. See examples of adding a region to user info and changing ID types to accommodate growth.

Learn how to define an enumerated data type for days of the week, order them, and use it in a table to track attendance, emphasizing semicolon terminators to separate queries.

Learn to streamline data insertion through backend pipelines rather than manual insert into statements, and use sql scripts to drop tables, cascade relations, and load data from a GitHub dataset.

Explore filtering a track table with a where clause to select tracks by composer using string literals, and apply limit, and operators =, <, <=, >, >=, to refine results.

Learn how to perform data type conversions by casting between integers and real numbers, using as and :: syntax, casting individual values or full columns to enable floating point divisions.

Trim left, right, or both sides of strings with ltrim, rtrim, and trim. Cast between text and numbers and apply lowercase or uppercase conversions.

Explore string positional information in Python for software engineering by using length, case statements, and position to measure name lengths, bucket sizes, and locate substrings for data analysis.

Master string replacements and the replace function to format timestamps as readable text, switch spaces to T or vice versa, and standardize date formats for databases.

learn to work with dates and times in sql queries by casting timestamps to date, extracting time components, and filtering with now for current utc date.

Learn to use time intervals to compute differences between now and invoice dates, convert intervals, and filter data by relative ranges like the last seven days.

Master inner joins to combine album and artist data using album id and artist id, explore foreign keys, aliases, and selecting only needed columns for efficient queries.

Master how left, right, and full outer joins extend inner joins to include unmatched data, using track and invoice line examples to compare behavior and outcomes.

Explore window functions to compute running totals, counts, and averages while preserving rows. Learn partition by, over, order by, and window aliasing to simplify complex aggregations.

Explore how window functions enable row numbering, ranking, and dense ranking within partitions, using unit price partitions on the invoice line table, with practical examples.

Learn to evaluate query performance with an aggregation on the track table using explain to view the plan and explore execution time across databases.

Update your Python app to read user profiles from a database using a database client, SQLAlchemy queries, left joins, array_agg for liked posts, and paginated results with total counts.

Implement a PostgreSQL insert for creating a user, returning the new user id. Handle conflicts with on conflict do nothing and raise a user already exists exception.

Implement and test a delete statement to remove a user by ID in the database, with a helper execute method and transactional execution to ensure proper database updates.

Refactor the create update flow to use a general update with an asynchronous database client, choosing insert or update based on a user lookup and validating with unit tests.

Learn to test code with unavailable databases by building an async mocking database client and fixtures, asserting awaited calls and arguments, and validating paginated queries.

Understand caching with Redis to speed up reads by storing results in memory as key-value pairs. Learn how expiration times and in-memory operations boost performance and reduce database load.

Implement Redis-backed caching in the application, integrating Postgres and Redis in docker compose, adding prefixes, get/set/delete helpers, and cache-first strategies for read and write operations.

Explore Redis hashes to efficiently group related data and avoid memory explosions from large objects; learn hash operations, existence checks, and subset retrieval for scalable caching.

Explore Redis sets by adding and reading unique values, removing items, and performing intersection and union operations, while understanding efficient data transfer.

Switch from json to Python-specific pickle serialization for Redis cache, storing byte representations and enabling immediate Python object retrieval with proper deserialization and decoding control.

Explore cache flushing: implement a flush db endpoint to clear keys in a specific database, compare with flush all, and consider asynchronous options and potential load risks.

Explore when to compress data before caching using Python snappy, reducing cache size and network transfer while weighing CPU costs for compression and decompression.

Learn to configure mypy for type checking, manage library stubs and missing imports, and enforce strict typing to improve code quality.

Explore isort-based import sorting with a configuration file to ensure consistent multi-line formatting, line-length choices, and trailing commas, integrated with typing checks and linting.

Install Black and run Black check to automatically format code for readability. Reformatting multiple files, then rebuild the Docker image to include the new tool.

Install a linting tool to catch unused imports and anti-patterns, run make check, and align with block formatting 88 line length while ignoring f401 and error codes two or three.

Set up git and GitHub on your machine, create a GitHub account, configure global username and email, and learn to log in with the GitHub CLI for pushing code.

Learn to use the GitHub desktop client as an alternative to the command line, install from desktop.github.com, sign in via preferences or accounts, and review code changes with previews.

Learn how to clone a GitHub repository to your local machine using the terminal or GUI tools (GitHub Desktop or Docker Desktop), choose a path, and open it in PyCharm.

Learn how to undo local and published commits, reset and revert changes, and squash multiple commits into one to clean up history, with force pushes when necessary.

Navigate the pull request workflow to review, merge, and protect code with branch rules, status checks, and peer approvals, while resolving conflicts and rebasing for a clean master history.

Learn to integrate pre commit hooks into your Python project, enabling automatic linting and formatting on every commit. Set up and customize a pre commit workflow and enforce coding standards.

Develop a Postgres insertion worker to write price data to a prices table using a SQLAlchemy engine, environment variables, and a master scheduler with an input queue.

Connect the Postgres master scheduler with symbol and price queues across the wiki worker, Yahoo finance price scheduler, and Postgres worker; ensure proper done signaling and timestamp casting for inserts.

Define and read a YAML pipeline config to declare queues and workers, enabling a modular, configurable data flow from Wikipedia data to Yahoo Finance and PostgreSQL.

Develop a yaml reader to load a pipeline, initialize queues and worker instances, and dynamically import and configure workers with input and output queues using flexible initialization parameters.

Improve a YAML-based data pipeline by coordinating workers and queues, implementing a central monitoring loop, and signaling completion only after all workers finish.

Define local and pipeline environment variables in a .env file and export them for testing on Linux or Mac, including Postgres settings and extraction time.

Master how to prevent race conditions in Python threading by using locks and context managers to safely increment a shared counter across multiple threads.

Split the workload across multiple processes to leverage processing resources, use a queue to pass bucket results, and coordinate with a done signal for inter-process communication.

Explore using multiprocessing pool map with multiple arguments by leveraging partial from font tools to predefine fixed parameters while leaving the changing one for each iteration.

Use star map with multiprocessing to pass multiple varying argument pairs by zipping input lists into (x, y) pairs and applying a function on each pair.

Explore multiprocessing techniques to check how many values in a list fall within specified lower and upper bounds, using a comparison list and star-unpacking for scalable input.

Explore asynchronous programming in Python using asyncio to run a single event loop, define async functions and coroutines, await results, and schedule futures and tasks for concurrent execution.

Explore how async gather enables concurrent execution of independent tasks, like simultaneous api calls, by scheduling coroutines in an event loop, via asyncio, reducing blocking and comparing async versus threading.

Demonstrate asynchronous for loops in Python by yielding values with a generator and awaiting sleeps, showing how the event loop gains control while the loop executes sequentially rather than concurrently.

Compare synchronous requests with an asynchronous http client to fetch url text, and learn about async with, the event loop, and tasks for non-blocking performance.