Mastering Splunk: CIM Compliance, Data Models, and Tstats

Explore the shortcomings of static inventories and learn how dynamic inventories in Splunk overcome them. Leverage host and network logs to build a robust dynamic inventory within Splunk.

Relying on static inventories fails in a changing IT landscape, creating blind spots and vulnerability exposure. Learn to observe your network with the tools you already possess.

Build a dynamic inventory with Splunk, drawing on Zeek network logs and host logs for accurate visibility. Leverage data models and stats to customize insights and detect misconfigurations and threats.

Zeek, the open source network security monitoring tool, transforms raw traffic into logs for connection, dns, and http, to build an inventory of ip addresses, mac addresses, services, and protocols.

Building your own inventory transforms operations by enabling faster troubleshooting and proactive maintenance, with immediate access to devices, configurations, and network paths for better compliance and asset visibility.

Explore module two to analyze raw log data, understand the structure of log formats, and identify essential fields for a dynamic inventory using Splunk search commands.

Explore the most common log formats—syslog, JSON, CSV/TSV, and XML—and learn how their formats, delimiters, and fields affect parsing, storage, and data ingestion in Splunk, using a Sysmon example.

Extract key fields like source and destination IP addresses, host names, and MAC addresses to identify devices and classify by type and operating system for a dynamic inventory.

Use splunk search language spl to extract, filter, and organize logs with fields, table, and stats commands, focusing on source ip, dest ip, and hostname.

Explore how to use Splunk by running index equals Zeek queries, filter to the left by source type, and use all-time searches to access Zeek logs.

Learn to use fields, table, and stats on Zeek con logs to extract origin and destination IPs. Switch between verbose, fast, and raw modes with source-type filtering.

Learn to search logs efficiently with Splunk’s metadata command, querying metadata fields like source types and hosts, then refine with index and Zeek logs for fast all-time analysis.

Download the bots v3 data set, transfer it via scp, extract into Splunk, and restart the server to enable CIM compliance, data models, and tstats.

install the Splunk stream add-on for the bots v3 data set to parse logs for your lab. use the bots v3 GitHub page, not Splunk base, as the data changes.

Execute hands on practice in Splunk with boss of the sock bots data, identifying source types, focusing on stream logs, and extracting key fields to inventory assets, users, and events.

Learn how to identify, categorize, and analyze network and security data in Splunk Stream, using DNS source type, stream IP fields, and event logs to inventory hosts and queries.

Explore how the common information model organizes logs into authentication, network traffic, and web endpoints, using standardized fields like source IP, destination IP, URL, and user.

Map raw logs to SIM compliant fields by understanding the source data, tagging sources, mapping fields, transforming data, and testing with SIM validation tools to ensure accuracy.

Install the common information model (CIM) app via settings or Splunk base to enable data models like network traffic and authentication, and review derived, extracted, and calculated fields.

Explore the Splunk common information model and its data models, including network traffic and authentication, with fields and prescribed values explained in the documentation.

Identify and document source types in bots v3 data during lab two, focusing on network traffic, network resolution, dns, and web activity to map to Splunk CIM data models.

Identify source types and data models for stream IP traffic, including DNS and web traffic using HTTP. Explore fields like bytes in/out and source and dest IPs.

Map Zeek con logs to the network traffic data model, extracting ports, proto, service, duration, and bytes to build a network inventory. Enable traffic analysis and anomaly detection.

Map Zeek DNS logs to the network resolution DNS data model and Zeek HTTP logs to the web data model, using query, query type, method, and URL.

Map Zeek smtp logs to the email data model, enrich network inventory, and use data models to detect phishing, analyze email content, and track email flow for policy adherence.

Add authentication logs to the authentication datamodel using Zeek, SSH, RDP, NTLM, Kerberos, and Windows host logs to enrich inventory and reveal unusual login patterns for CIM compliance.

Ingest host logs from PowerShell and bash into the endpoint data model to enrich software inventory and context. Use this model for monitoring and registry key tracking to aid compliance.

Expand beyond the basics by mapping logs to data models, performing event correlation across data models, and enriching with inventory metadata to detect malicious domains from phishing campaigns.

Map logs to data models to unlock faster tstats queries, enabling direct index-field access and scalable, standardized compliance reporting across disparate systems for IT and cybersecurity decisions.

Map a set of fields across three data models in lab three using the bots v3 dataset, including source, destination, ports, bytes, http and dns fields for inventories queries.

Review lab 3 results in Splunk, mapping key sim fields to network traffic, DNS, and web data models for CIM compliance, with source/dest IP and port alignment using bots v3.

Learn field extraction to parse raw log data into usable fields, normalize data for CIM compliance, enrich with lookups and calculated fields, and tag events for actionable insights.

Extract fields from raw logs to normalize data for SIM compliance, enabling searches and analytics via automatic discovery, regex-based extraction, the Splunk field extractor, and props and transforms for deployment.

Demonstrates automatic field extraction in Splunk by parsing a misformatted CSV using comma delimiters, refining fields (IP, hardware, org, room), and saving the results for reuse across apps.

Learn to perform regex-based field extraction on the sysmon log by creating an inline regex for the source ip. Save, reload, and apply the extraction to multiple fields.

Learn how to use the Splunk field extractor to automatically build regex-based field extractions, validate results, and apply extracted fields like user, source port, authentication type, and host name.

Apply regex-based delimiters in props and transforms to parse logs, then package and deploy them as a Splunk app, and avoid production parsing, moving from local to the target environment.

Explore data enrichment in Splunk using lookups, calculated fields, and tags to enhance logs with external data, IP addresses to host names, geo mappings, and organizational context.

Explore hands-on data enrichment with lookups in Splunk, using the lookup editor app to upload CSVs, define fields, and enrich z logs by matching IP addresses to host and location.

Create and validate calculated fields in Splunk using eval, join fields with proper quoting, and publish permanent fields; build a simple risk score from protocol values and set permissions.

Tag icmp data by creating event types and applying icmp tags. Color code red with a priority, then search by tag equals icmp to surface red logs for analysts.

Cover module six by setting up data models for network traffic and network resolution dns. Alias fields and add calculated fields to map zeek logs to the models.

Apply field aliasing to network traffic data by mapping selected fields to Zeek Conn logs, and create the new field alias network traffic beacon.

Validate field names and aliases to ensure orange packets and response packets populate correctly in the network traffic data model; tag data as network and communicate and define event types.

Create a calculated field in Splunk to classify network connections by state, using a case expression to label S0 as a successful handshake and others as failed.

Learn to configure DNS field aliasing in Splunk CIM, mapping DNS fields (query type, TTL, proto, transport, source and dest IP, reply code) and validate propagation for Zeek DNS data.

Troubleshoot aliasing failures in network resolution by creating calculated fields with eval and a case statement when field aliases won’t work.

Continue setting up Splunk field aliases by creating and verifying aliased fields for each source type, then tag data in the data model and run tag queries to confirm propagation.

Alias network traffic fields, including source and destination IPs, for stream IP, DNS, and HTTP; then tag, create event types, and validate with data model queries.

Validate SIM compliance using the CIM with the SIM validator app to test data, ensuring required fields, tags, and event types align for data models and stats.

Download and explore the sa sim gladiator app to validate sim data models and identify essential fields. Use the dictionary view to test CIM-aligned data and correct field values.

Apply calculated field fixes in the Vladiator app to compute bytes in and bytes out, handle missing values, and align vendor product and dest IP fields for CIM compliance.

Validate three data models in the Splunk Gladiator app, follow the instructions, and confirm the models map correctly while exploring the dictionary for field insights.

Walk through lab 5 to analyze network traffic, DNS resolution, and web data in Splunk, assess field coverage, and propose calculated fields and aliases to improve CIM compliance.

Explore data models in Splunk, a structured representation that organizes raw logs into meaningful objects for faster searches and learn to use the SPL data model command to accelerate them.

Explore data models in Splunk, from base objects to child datasets like all traffic and traffic by action, and learn how inherited constraints and meta fields shape searches.

Explore how parent data models and child data sets inherit constraints and tags in Splunk CIM, comparing endpoint versus web data models and how this affects SPL query searches.

Learn how Splunk accelerates searches by turning non-structured data into structured data via data models, enabling acceleration and fast t stats analyses on large datasets.

Accelerate data models in Splunk by running periodic backfill searches, configuring backfill ranges and concurrency, while noting a five-minute lag and no real-time results for health monitoring.

Master the data model SPL command in Splunk by using a pipe to access data model, select data set names, and choose a data model search mode (common, flat, acceleration).

Set the time range to all time, work with the network traffic data model, and compare search and flat options to read fields and preserve true field names.

Explore the SPL datamodel command part 2, detailing strict fields (default false vs true) and how data models, inherited and calculated fields, and lookups shape search results.

Learn how data acceleration duplicates raw data, consuming disk space and CPU with a 2-to-1 storage cost, and how old data rolls off to free space.

In module nine, build a Zeek log inventory using data models and stats, then enrich it with lookups from other logs, keeping it near real time and dynamic.

Learn how to manage time in Splunk queries, using current time or eval time equals now to preserve historical inventory data within a 30-day window and adjust lookup generators.

Build a flexible inventory using conlogs or Zeek logs, using IP addresses as the internal inventory key, and adopt summary indexing with collect commands and script options.

Learn to build an inventory using the collect command to create a summary index from CSV connection data. Run the collect commands to populate and verify the summary connections dataset.

Manually set up your new dataset to generate enriched metadata from weblogs, http logs, and printer logs. Include dce rpc and remote protocols for windows dns, web, ldap, and tlm.

Learn how a Python script automates populating Splunk indexes using a setup_env.sh, REST API calls, and a 1.5 second sleep between commands, with optional manual runs.

Combine source and destination IPs to generate a unified IPs field, expand multi-values, and use stats by IPs to deduplicate and record last seen times for an IP inventory.

Exclude non-RFC 1918 IP addresses by building a network topology lookup of internal ranges (10.0.0.0/8 and 192.168.0.0/16) and using the network subnet list macro to filter internal IPs.

Append daily ip data to a historical lookup in Splunk, dedupe by ip and keep last seen times, then purge entries older than 30 days to build a cim-compliant inventory.

Enrich an IP inventory by defining meta roles through protocol responses, labeling HTTP as a web server and SMTP as a mail server, regardless of software.

Enrich your IP inventory by building and updating metadata lookups from DNS, HTTP, and scanner logs, deduplicating by dest IP and tracking first and last seen with 30-day purges.

Integrate static and analyst-provided inventory using lookup generators and dedicated inventory lookups to populate host, subnet, IP, OS, and location data for enterprise security inventories.

Normalize Splunk data by deduplicating multi-value fields and labeling attributes with yes or no through meta fields (is web server, is file share, is scanner), improving enterprise security readability.

Define and build a pipe-separated category field, with regex to trim the trailing pipe, to classify data by roles like DC, OT, DNS, Windows, and SA for enterprise security queries.

Finalize the metadata inventory by appending part two lookups, using tstats to capture last seen, and purging records older than 30 days, then store the output as a lookup.

Build a modular IP inventory in Splunk by creating metadata, enriching data with lookups and metadata fields, and separating inventory and enrichment steps to simplify troubleshooting and maintenance.

Build a daily IP inventory by querying source and destination IPs, deduplicating, and applying lookups with mvappend and mvexpand to generate web and DNS results.