Detection Engineering Masterclass: Part 2

Learn to use the elastic api with Python to create, update, and delete roles and rules, and perform bulk actions. Push sample roles and custom alerts; upload scripts to GitHub.

Organize scripts into folders, log into Elastic to create and secure an api key (base64 encoded and viewable only once), and store it in a password manager.

Push a sample detection rule to Elastic using the security API with token authentication, posting to the detection engine rules endpoint and validating the result.

Develop a toml to json Python script to convert html files into elastic-ready json by iterating over rules, extracting required fields, handling data types, and escaping quotes.

Develop and test a Python script to fetch alerts by rule ID via a get request to the API, highlighting documentation gaps and fixes.

Leverage custom detections by enforcing rule IDs, validating scripts, converting HTML to JSON, and troubleshooting until all detections push correctly to Elastic.

Update custom detections by using the update_alert.py script and the put method to replace alerts, incorporating a role id in the URL and testing changes by adjusting a value.

Explore GitHub actions for automating document workflows, including HTML validation, elastic automation, metrics generation, and a main and dev branch workflow with git commands.

Learn how GitHub actions automate workflows with triggers like push, pull, and cron. Configure a simple yaml workflow, use the checkout action, and run scripts on an Ubuntu runner.

Upload detection documents and scripts to GitHub, organize folders for detections and development, replace hardcoded keys with GitHub secrets, and set up a GitHub action for HTML validation.

Create a GitHub action to validate detection documentation with validation.py and minor.py, implement error handling with exit codes, and run all detections using structured steps.

Enforce validation by protecting the main branch with status checks, ensuring HTML, miter, and detection document validations pass before merging.

Sync detections from the main branch into elastic using two GitHub actions: one ships all detections, the other updates only changed files on push, with API keys stored in secrets.

Sync updated detections to Elastic using a change files action and an updated alert script. The workflow covers branch management, push on main, and 404 handling to insert new alerts.

Explore metrics for detection engineering using Python and GitHub actions, and build three converters—HTML to CSV, Tamil to Markdown—plus a Mightor Navigator JSON data generator via GitHub actions.

This lecture shows building a toml to csv converter that outputs a csv with date, name, author, risk score, severity, and mitre data like tactic, technique, and sub technique.

Build an HTML to markdown script that outputs three markdown tables of detections by current month, last month, and two months ago, using Python date handling for GitHub rendering.

Convert a toml file into an att&ck navigator json by building a dynamic heatmap layer, counting techniques and sub-techniques, and exporting navigator.json for use in attack navigator.

Create a GitHub action to automate running metrics scripts, configure a personal access token and secrets, push results to GitHub, and add optional cron scheduling.

Learn to create status badges for GitHub actions, copy the badge markdown, and embed it in the README to monitor metrics, updates, and validation at a glance.

Explore next steps in detection engineering, including expanding atomic red team tests, enhancing automation with elastic dashboards and Power BI, and using threat intelligence to evolve detections across networks.