Applied ELK Stack: Data Insights and Business Metrics With Collective Capability of ElasticSearch, Logstash and Kibana

510 Pages·2017·4.002 MB·English

by  Gurpreet Sachdeva

Applied ELK Stack
Data Insights and Business Metrics With Collective Capability of ElasticSearch, Logstash
and Kibana

n n n
Gurpreet S. Sachdeva
Applied ELK Stack: Actionable Data Insights and Business Metrics with
the collective capability of ElasticSearch, Logstash and Kibana
Gurpreet S. Sachdeva
New Delhi, Delhi
India
ISBN-13 (pbk): 978-1545022146
ISBN-10 (pbk): 1545022143
Copyright © 2017 by Gurpreet S. Sachdeva
This work is subject to copyright. All rights are reserved, whether the whole
or part of the material is concerned, specifically the rights of translation,
reprinting, reuse of illustrations, recitation, broadcasting, reproduction on
microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than
use a trademark symbol with every occurrence of a trademarked name, logo, or
image we use the names, logos, and images only in an editorial fashion and to
the benefit of the trademark owner, with no intention of infringement of the
trademark.
The use in this publication of trade names, trademarks, service marks, and
similar terms, even if they are not identified as such, is not to be taken as an
expression of opinion as to whether or not they are subject to proprietary
rights.
While the advice and information in this book are believed to be true and
accurate at the date of publication, the author cannot accept any legal
responsibility for any errors or omissions that may have been made. The author
makes no warranty, express or implied, with respect to the material contained
herein.
Technical Reviewers: Shyam Seshadri and Amit Singh
Source code and supplementary material referenced by the author in this book
is available to readers on https://github.com/gssachdeva/Applied-ELK-
Stack.
To my mother and father
CHAPTER 1: Introduction to ELK Stack
Log Analysis in Today's World
The ELK Stack
Elasticsearch
Logstash
Kibana
ELK Data Pipeline
ELK Stack Installation
Installing Elasticsearch
Running Elasticsearch
Elasticsearch Configuration and Settings
Environment Variables
System Configuration
Installing Elasticsearch Plugins
Installing Logstash
Running Logstash
Logstash with Elasticsearch Output
Configuring Logstash
Installing Logstash Forwarder
Extending Logstash Functionality
Installing Kibana
Kibana Configuration
Kibana Interface
Summary
Sample Dataset
Data Format
Logstash Configuration
Comments
Configuring for Events
Field References
Conditionals
Metadata
Filtering Events
Shipping Events
Reloading Configuration File
Multiline Event Configuration
Analyzing Events
Data Visualization
Building a Line Chart
Building a Bar Chart
Building a Metric
Building a Data Table
Summary
CHAPTER 3: Extending Logstash
Plugin Management
Download and Installation
Plugin Installation
Updating Plugin
Uninstallation
Plugin Structure
Prerequisite
Basic Structure
Configuration
Setup
Execution
Teardown
Build Custom Plugin
Plugin Packaging
Summary
Ubiquity of Data
Elasticsearch Cluster
Node
Anatomy of a Document
Metadata Information
Index
Type
Id
Shard
Primary and Replica Shard
Elasticsearch API
Cluster Health and Configuration
Index Management
Specify Id
Custom Id
Auto-Generated Id
Document Management
Document Retrieval
Partial Document Retrieval
Document Existence
Multiple Document Retrieval
Document Updates
Updating Documents Partially
Partial Updates with Scripts
Conflicting Updates
Document Creation
Document Deletion
Bulk Operations
Bulk Request Size
Conflict Management
Summary
Search Your Way
Simple Searches
Search Without Parameter
hits
took
shards
timeout
Multi-Index, Multi-Type
Pagination
Search Lite
The _all Field
Query Mashup
Query DSL
Query Clause Construction
Working with Multiple Clauses
Filter and Query
Performance Concerns
Key Filters and Queries
term Filter
terms Filter
range Filter
exists and missing Filters
bool Filter
match_all Query
match Query
multi_match Query