Splunk Administration - An End-to-End Course[2023]

Discover how UF and Splunk Enterprise monitor logs, configure deployment architectures, and manage indexers, dispatchers, and the Splunk server for effective log analysis.

Learn how Splunk processes data from input through parsing, timestamp extraction, and line breaking to indexing, using a transaction log example. See how indexed events are retrieved via indexers.

Trace how log data flows from an application server to the Splunk indexer, covering inputs, parsing, indexing, event breaking, timestamps, buckets, and search workflows.

Learn how to create an AWS account to install and configure splunk servers, covering signup, payment details, mobile otp verification, activation, and login.

Create an EC2 instance in the management console, select Mumbai, configure a security group and key pair, then connect via ssh on port 22 using the public ip.

Learn to install Splunk Enterprise on Linux, download and sign up, install as a non-root Splunk user, and open port 8000 (and 8089) to access via browser.

Install the universal forwarder on a linux instance by downloading the package, running the installer, and verifying the service with commands, while noting that the forwarder lacks a web UI.

Learn hardware recommendations for Splunk Enterprise and Splunk Firewater, including 12 GB RAM minimum (up to 128 GB), 30–48 CPU cores, and high speed storage with 800+ IOPS.

Explain essential Splunk ports for internal server communication, web login, data intake via forwarders, and syslog, and advocate using an intermediate server with load balancing to prevent data loss.

Configure three key Splunk server settings: disable THP for performance, enable boot-start on reboot, and set ulimit values for open files and processes.

Learn to disable THP, enable boot-start services, and configure ulimit values on systemd-based Linux systems to ensure Splunk runs reliably after reboots.

Learn to restart the Splunk server's internal process (not the OS) within an enterprise installation, and understand the $SPLUNK_HOME context and how to locate the restart command.

Learn about Splunk deployment types, from a single instance handling inputs, indexing, and searching, to distributed and multi-site cluster architectures. Understand indexer clustering, replication, and knowledge object availability across sites.

Explore the non-clustered deployment in Splunk administration, configuring forwarders to send data from application servers to indexes and receive data from flight orders, preparing for a clustered environment.

Learn to download, install, and use the MTPutty tool to log into NWS instances, configure multiple servers by adding them with IP addresses, and rename properties for easy identification.

Configure the Splunk indexer via the web GUI by setting up a receiving port and creating a new index, understanding its role as a logical directory to store logs.

Configure indexer backend by creating custom indexes in the local folder using indexes.conf, including index name, directory, max size, and port settings, while avoiding edits to the default folder.

Configure the Splunk indexer via conf files, create inputs and indexes, restart the Splunk process to apply changes, and verify the index is created and active.

Enable SSL for the Splunk indexers via the web GUI by turning on SSL in server settings, adjusting ports, restarting Splunk, and verifying a secure https connection.

Learn to enable SSL certificate in Splunk by creating a local configuration file, setting enable Splunk Web SSL to 1, and restarting Splunk to verify secure login.

Configure forwarder concepts by mapping transaction logs from application servers to indexes, detailing inputs and outputs, and exploring how load balancing distributes events across indexes.

Configure a Splunk forwarder to send log files to two indexers by editing outputs.conf, specifying indexers and ports, restarting Splunk, and verifying internal logs.

Enable the Splunk forwarder to start on reboot with a single command, and verify status after reboot, using root or a specified non-root user.

Configure inputs that can't fail on the Splunk forwarder to monitor a log in the temp folder, de-conflict settings, and route events to the Splunk index with proper timestamp handling.

Configure search head concepts in Splunk by understanding forwarders and indexers, index data routing, and load management across indexers to enable centralized searching.

Configure the search head in Splunk by enabling SSL on the server, adding indexers and management settings, then log in, verify indexes, and run searches across multiple indexes.

Learn how the forwarder uses indexer acknowledgement to guarantee that 64k data blocks are written to disk, retry blocks after a timeout, and switch to another indexer when needed.

Learn to configure indexer acknowledgment on Splunk forwarders, enable the acknowledgment property in outputs, restart services, and manage forwarder configurations centrally with a deployment server.

Configure a deployment server and connect forwarders as deployment clients to centralize apps and configuration management. Create server classes and forwarder apps, then push them to all forwarders.

Learn to edit an existing Firewater forwarder app, update indexer IP addresses on the deployment server, and push the updated forwarder app to all connected forwarders, validating the push.

Conclude the module by reviewing deployment workflow, including setting up the deployment server, connecting forwarders, and applying app-wide orders, with emphasis on indexing.

Understand the Splunk licensing concept by calculating daily indexed data, applying a 10% buffer, and purchasing the appropriate license for Splunk Enterprise.

Explore Splunk licensing types, including enterprise trial licenses valid for 60 days with a 500 MB per day limit, free licenses, and licensing violations affecting indexing and search.

Add and install a Splunk license file via Settings, switch to an enterprise license master, and view daily volume and expiration details.

Configure and link every Splunk component to a licensed master so indexers, deployment servers, and cluster masters report license usage and data volumes.

Splunk reads configuration files from four locations in priority: edc system local, apps/app name local, apps/app name default, and edc system default, resolving conflicts by applying the highest precedence.

Explore Splunk conf file precedence with an end-to-end example showing system local, app local, and default stanzas, and how disabled flags determine the final inputs.conf configuration.

Use the btool utility to view the final version of Splunk configuration files, such as inputs.conf, by analyzing stanza-level attributes and values from multiple source locations.

Estimate required indexers and search heads using the 100 GB per indexer rule and a per eight users guideline, with minimum hardware; for 2 TB/day, this suggests about 20 indexers.

Discover how data is stored on disk in Splunk, using the US index with transaction_data, and how hot and cold buckets in Splunk home organize storage.

Discover how data rolls from hot buckets to frozen or archived buckets within a Splunk index, including conditions that trigger moving, writable vs read-only states, and restoring archival data.

Learn how Splunk creates a logical index directory on disk, stores data in hot buckets with raw data and index files, and rolls hot buckets into one after restart.

Learn what a Splunk bucket contains, including raw data, index files, and metadata, and see how the index stores unique keywords to speed searches.

See how Splunk data buckets move from hot to one bucket, then cold and frozen, driven by size thresholds, timespan, and restart conditions.

Learn bucket naming conventions in Splunk, including hot and replicated buckets, locality-based identifiers, and time-stamp conventions across standalone and distributed cluster environments.

Calculate index storage by multiplying daily data volume by retention period and compression factor, then divide by indexers; specify max total data sets and frozen time period for distributed environments.