ELASTIC 8: Monitoring with Elasticsearch, Kibana, Beats, APM

Plan your Elasticsearch cluster by defining nodes, indices, and primary and replica shards. Apply production guidelines by using three nodes and hot, warm, and cold nodes for data balance.

Discover how to troubleshoot Elasticsearch memory limits in Docker, resolving bootstrap checks failed caused by too low max virtual memory areas and adjusting Linux or Windows Docker settings.

Remove the setup service from the Docker compose YAML after the cluster starts, keeping the three Elasticsearch services and Kibana, updating S01, and restart the cluster.

Create and configure an ubuntu virtual machine in VirtualBox, including memory and disk settings, ISO boot, and initial setup. Then install Elasticsearch on the VM.

Reusing OS Boxes virtual disks for multiple vms may trigger this id conflict; fix by running VBoxManage to change the virtual hard disk UUID and launch three Ubuntu Elasticsearch nodes.

Install elasticsearch on ubuntu with the debian package, import the gpg key, start and enable the service, and verify the cluster at localhost:9200; then install Kebano for the UI.

Install Kibwana on Ubuntu using the Debian package, add the GPG key, configure apt with https, enroll Kibwana to Elasticsearch with an enrollment token, start Cabana, and open localhost:5601.

Edit Kibana's configuration to set server.host from localhost to 0.0.0.0, restart the Kibana service, and verify access from other hosts on the local network.

Prepare a second VirtualBox node for elasticsearch by naming the VM, allocating memory, using an existing drive, and enabling NAT networking for inter-node communication.

Demonstrates adding a second Elasticsearch node by configuring its yaml, generating enrollment tokens, sshing between nodes, restarting services, and verifying the two-node cluster in Cabana.

Install Elasticsearch on node three, configure discovery seed hosts and master nodes with the other two, restart services, and verify a three-node cluster in Kibana dev tools.

Discover Kibana config paths on Debian/Ubuntu and in Docker, review the Kibana default directory, the config directory with the keystore and kibana.yml, and compare with Elasticsearch.

Enable ssl in kibana by configuring server.ssl.enabled to true, generating self-signed certificates with openssl, adjusting file permissions, restarting kibana, and verifying https on localhost.

Explore Elastic Beats, including metric beat, file beat, and heartbeat, to collect metrics, logs, and uptime checks from sources, and send data to Elasticsearch for visualization in Kibana with alerting.

Install and configure filebeat, enable log sources, connect to Elasticsearch with HTTPS and credentials, and use modules to ingest Elasticsearch logs, then verify in Kibana.

Deploy filebeat in Docker to collect container logs via the Docker module and socket, enrich with Docker metadata, and ship them to Elasticsearch, with Kibana for live log exploration.

Set up heartbeat for uptime monitoring and observability with Elasticsearch and Kibana, configure monitors for HTTP checks, and view data streams, uptime, and results.

Configure index lifecycle management for beats data streams in Elasticsearch, linking templates and policies to transition data from hot to warm or cold storage and delete after retention.

Configure metric, file, and heartbeat dashboards in Kibana by updating beat YAML outputs (Elasticsearch output, https and port 5601) and authentication settings, then run setup dashboards to visualize host data.

Discover Elastic APM, instrument both front-end and back-end apps across languages, send data to Elasticsearch, and visualize latency, transactions, dependencies, and alerts in Kibana.

Explore the Discover tab to search your data across Beats in Elasticsearch, using time ranges, field filters, and saved searches to view documents.

Explore visualizations and dashboards in Elasticsearch and Kibana, using aggregation-based visuals and Lens to monitor CPU usage and file system metrics from metric beat data.

Explore the observability overview, review logs, metrics, APM, and uptime, and check hosts and services like lk01 and Java; note that user experience monitoring and alerts are not yet activated.

Explore how the logs tab streams live Elasticsearch logs, like a tail, highlights keywords with a highlighter, and lets you customize indices and columns while filtering by host name.

View and group multiple hosts by metric types, monitor memory and CPU usage, and switch between Kubernetes ports and Docker containers to observe pods, group by host name, and manage indices.

Explore the uptime tab to review heartbeat pings to Microsoft.com, filter by up, down, or all, and watch red charts indicate down pings, while noting indices and upcoming alerts.

Explore the management tab to configure settings with def tools for Elasticsearch, review integrations and fleet status, and begin stack management configurations for Cabana and Elasticsearch.

Discover how to create users and roles in elasticsearch and kibana, assign fine-grained index privileges, and tailor a non-admin test user for secure access and dashboards.