What is Splunk

Splunk is an integrated analytic solution providing throughout and extremely efficient insights across system condition of various scale and purpose. Splunk, due to its distributed architecture allows to analyze massive live data streams of various sources.

Data sources for Splunk:

  • event log – called UNIX / Linux system logs, Windows event log (System, WMI, Perfmon) of various specialized devices (tracers, firewalls, physical security etc.)
  • database – e.g. business apps database or audit trail
  • server-side scripting – regular data upload from SAR, vmstat, iostat, cpustat, netstat or from scripts created for custom IT solutions.
  • any text file – both of known and unknown structure
  • specialized object event and configuration repository – Windows registry, directory services: Active Directory (LDAP)
  • network ports – e.g. any TCP or UDP to which messages are sent (e.g. XML mode)
  • Syslog daemon – Splunk can work in SYSLOG daemon mode (rsyslog)

Approach of using a query language based on models, provides “on the fly” data analysis, without a need for pre-adjustment (normalization, converting to common model). This characteristic of Splunk enables various sources data integration, without costs incurred for initial preparation, as in case of other traditional models (e.g. ETL techniques).

The greatest strength of Splunk is its ability to dynamic data indexing, especially in the context of easy and fast access to information concerning the quality and status of the infrastructure and IT applications. This is possible because of the dedicated, integrated datamart (called ‘index’) and available, defined algorithms of content analysis and model search. Described mechanisms also allowed for generating dedicated, interactive environment of creating queries. Basic features of this environment are:

  • regular expressions, as an effective data search method
  • automatic interpretation of intuitive queries, created by the analysts and administrators ‘live’
  • automatic analysis of data structure
  • dynamic correlation of various source data (no matter its ‘raw’ format)
  • user interaction, based not only on query content, but also on ‘deep insights’ into data

Splunk, in addition to providing intuitive user interface and dynamic data analysis, affords business activity monitoring within the entire infrastructure and deployed applications. It provides creating automatic alerts and periodic reporting based on the defined queries. These reports can be automatically sent e.g. to selected e-mail addresses – e.g. IT support team or the executives, in case of web traffic analysis made for e-business industry. Additionally, managers can use custom dashboards dedicated for them, e.g. for reporting compatibility between IT and SLA elements, or sales levels by products.

Below slide presents the exemplary management dashboard for sales analysis. Splunk dashboards, apart from historical data view, provides real time data updates, which allows extremely rapid overview the changes within the business surroundings and swift response at the management level.

Let’s summarize: Splunk is custom designed solution of Enterprise Intelligence, delivering IT infrastructure throughout analysis, including all technical and business events operated within the systems.