Concurrent and Parallel Programming in Python

Explore concurrent programming in Python, covering threading, multiprocessing, and async, with insights into CPU utilization, IO-bound vs compute-bound tasks, thread safety, race conditions, and practical tradeoffs.

Master threading in Python by creating and starting threads to run calculations and sleeps, measure timing, synchronize with join, and understand daemon threads and multiprocessing for cpu-bound tasks.

Explore turning threaded programs into reusable classes by implementing squared sum and sleepy worker classes. Manage threading with run, start, and join, and use keyword arguments for flexible initialization.

Explore concurrent programming by building a wiki worker that fetches S&P 500 symbols from Wikipedia, retrieves Yahoo Finance prices, and saves results to a database using requests and BeautifulSoup.

Set up a threaded Yahoo Finance price reader using a wiki worker to fetch S&P 500 symbols and a price worker to scrape prices with requests and lxml XPath.

Systemize symbol processing with a multiprocessing queue and a master scheduler that feeds Yahoo Finance price workers, enabling scalable, thread-safe producers and consumers for price extraction.

build a postgres insertion worker that inserts price data into a postgres prices table (id serial, symbol, price, extracted time) driven by a postgres master scheduler with an input queue.

Integrate the Postgres master scheduler into the main function, connect input and output queues, and coordinate a three-step pipeline with Yahoo Finance prices and downstream done signals.

Define and read a yaml pipelines configuration to orchestrate queues and workers for concurrent python workflows, enabling flexible, reusable data pipelines.

Learn to build a yaml-based pipeline reader that loads a yaml-defined queue of workers, initializes input and output queues, dynamically imports worker classes, and orchestrates concurrent execution.

Implement a yaml-driven concurrent pipeline by integrating a yaml pipeline executor, wiki worker, and master scheduler with queues, enabling scalable, parallel scraping and processing.

Refactor the yaml pipeline executor to run as a main worker thread, monitor progress, and only send done signals after all workers finish across downstream queues.

Clean up the program by configuring an environment via a local .env file, exporting the pipeline location and database variables, and validating with a test run for network bound threading.

Explore locking in Python threading to prevent race conditions when updating a shared counter, using lock acquire and release and a context manager for thread-safe results.