MLOps: MLflow, CI/CD, Feature Stores and Model Monitoring

This course contains the use of artificial intelligence.
Your model is serving. You are now the person who gets paged about it. That is the gap this course is for.

This is the direct sequel to Course 02. That course got a model behind an interface, in a container, on a cluster, with a rollback you could defend. This one automates everything after that moment: the pipeline that retrains it, the registry that promotes it, the store that serves its features, and the monitor that tells you it has quietly gone wrong. Serving, containers, clusters and rollout mechanics are assumed here, not re-taught.

If you are maintaining a model by hand - retraining when somebody complains, from a folder of notebooks with dates in the filenames - you already know it does not scale. What you have probably been told is that you need MLOps, without being told what that consists of. It consists of four things, and this course builds all four from the primitive upward.

It takes a position on tools, out loud, in the first sections. MLflow is the one taught properly: the tracking API, what a run actually is, params against metrics against artefacts, the backing store, and the registry built on top of it. Weights and Biases gets one honest comparison lecture instead of a second shallow tutorial - what hosted collaboration, sweeps and reports buy you, what they cost in account, data residency and per-seat billing, and how to decide. Airflow is taught properly because you can run it; TFX is taught as an architecture and a set of guarantees. Two tools taught badly is worse than one taught well.

Nothing here walks a console. GitHub Actions, a tracking UI, a cloud registry and a monitoring dashboard all get redesigned on somebody else's schedule. You learn the workflow YAML, the API call and the object model, because those have been stable for years and a screenshot of a settings page is obsolete before the course finishes rendering.

The spine of the second half is a dataset you can reproduce: 18 months of scoring traffic for an invented small-business loan approval model, 3,000 rows a month, 54,000 in total, 8 features, trained on the first 3 months. The lender, the applicants and the data are all synthetic and seeded. At month 6 the input distribution moves: PSI against the previous month hits 0.2238, well over the conventional 0.2, the monitor screams, and accuracy falls from 72.09% to 68.78% - a loss of 3.31 points. At month 11 the relationship changes while the inputs do not: PSI reads 0.0087, quieter than the 0.0067 of an ordinary month, the monitor stays green, and accuracy falls to 59.31% - a loss of 9.47 points. An input-distribution monitor is loud about the thing that did not matter and silent about the thing that did, and no function of the inputs could ever have seen month 11. That is a limit, not a gap in your tooling.

So the monitoring section is about thresholds you can defend rather than tests you can install. At 0.2, exactly 1 of the 17 month-on-month comparisons fires, and it is not the one that cost you 9.47 points. Drop to 0.01 and 8 months fire, still catching none of it. Drop far enough to catch month 11 and 15 of 17 months fire, which is an alert nobody reads by the second week. There is no threshold on the inputs that solves this, so the lecture is about what you do instead.

Retraining is measured too, including where it falls short. Left alone, the stale model holds 59.31% on the tail months. Refit on the three months from month 11 it reaches 65.53% - a gain of 6.23 points out of the 9.47 that were lost. Report the shortfall, not just the gain. And a third failure runs the whole time that is neither kind of drift: a feed degrading quietly until 3.63% of rows arrive with no debt-service value, which every distribution test in the course ignores because PSI does not look at nulls.

Two lectures exist because tutorial-grade MLOps courses skip them. One is about cost - the four centres a bill decomposes into, the unit each bills in, which line items are missing from everybody's first estimate, and how to decide whether the automated loop pays for itself. It quotes no prices, deliberately. The other is about who owns the model in production when it is wrong at three in the morning, which is a question about team topology and on-call, not about tooling.

The capstone is a loop that walks the months forward one at a time, seeing only what had happened by then - detect, decide, retrain, validate, promote, roll back - against a written policy, with a log that has to give a reason for every decision it made. A refused challenger is a result. A loop that never refuses anything has not been tested.