Build AI Reviewer Skills for Platforms that Pay (L 3)

The first three steps of rubric design. Start with the decision the data will drive, derive dimensions from observed variation in real responses rather than inventing them, then compress to three to six dimensions using three simple tests.

Choose the right scale, then learn the anchor masterclass: three rules for writing anchors that make a score mean the same thing to every rater. We finish by looking at what a completed rubric looks like, using a worked one-to-four resolution scale.

A rubric has to earn its way into production. You will build a calibration set of annotated examples for every scale point, run the pilot loop, and see how the same rubric adapts for an LLM judge, including the three additions it needs and where it should not be trusted.

Two raters agreeing 80% of the time sounds impressive, but much of that is chance. This lecture shows why percent agreement flatters, introduces chance-corrected measures, and points to what lies beyond kappa when your data has more than two raters or more than two categories.

Low agreement is a symptom, not a verdict. You will learn the three common patterns of disagreement and the repair for each, so you fix the rubric instead of blaming the raters. Then we cover drift: agreement is something you maintain, not something you achieve once.

Expert evaluators name exactly what failed. This lecture builds the first half of a working failure taxonomy: factual errors, reasoning errors, instruction-following failures, and calibration failures. Learn what each looks like in a real response and how to tell them apart.

The second half of the taxonomy: sycophancy, safety failures, style problems, and agentic failures. We then close the loop on how professionals use a taxonomy to derive rubric dimensions, connecting this module back to the design process.

Specialist domains change how you evaluate. For code, correctness is testable, so you run it. For STEM, you verify the derivation, not just the destination. Learn what to check, what to trust, and the traps that can trip you up.

Three extra disciplines apply when evaluating regulated professions. Multilingual evaluation splits into three separate dimensions: fluency, register, and cultural fit. Finally, build your own T-shape: reliable generalist judgement plus one domain where you go deep, and know where your competence ends.

For multimodal responses, grounding is everything: does the answer match what is actually in the image, chart or document? You will see where grounding breaks and how multimodal rubrics anchor on it, so you can catch confident descriptions of things that are not there.

Agentic evaluation asks a different question. Not whether the final answer is good, but whether the trajectory was sound. Learn three disciplines for reviewing an agent’s steps, and how to separate a lucky success from a well-reasoned one and a root cause from a symptom. Level 3 assessment is an option to practice and answer key for all assessments attached.