Using the ELK Stack for Business Intelligence

As a web developer, I'm comfortable with computers. I love that a computer does what is written in the code and that's that. If the computer doesn't do what you expect, then it's your fault as the programmer. You don't have to worry that it will do anything except what it's told (at least until AI enters the picture). For my entire career, I have worked within technical departments where the requirements are usually pretty straightforward: build this website; integrate this system; add this feature; fix this bug. Recently, I have joined the Marketing team at Opsview, my first truly customer-facing department. The requirements are more complex now. I have to deal with humans.

Human Behavior

Humans are unpredictable. Humans are irrational. Humans are emotional. Human behavior cannot be debugged like a computer. But in the end, technology is useless if it isn't used by a human. As a software company, we have to understand our customers as fully as possible in order to make a product that they love and to connect with potential new customers.

So, how do we know how our users are behaving? We have many systems that give us slivers of information. Our web logs lets us know how we're doing in terms of web site traffic. Our marketing automation system gives us an idea of how users behave after they register. Our CRM lets us know how well we're doing in the Sales funnel and after users become customers. But, how do we bring that all together and see the full lifecycle of our users?

Using ELK

Enter ELK, which stands for (E)lasticsearch, (L)ogstash and (K)ibana. This is an open-source technology stack that wasn't necessarily designed for this type of use case. Elasticsearch was clearly designed for search, built on top of the Apache Lucene search engine library. Logstash is a parser, which has a massive list of plugins that allow you to import all sorts of data into Elasticsearch. Kibana is the front-end, which lets you search and visualize this data. Put it all together, and you have a way to import all of your data sources into a single, searchable place where you can see your potential and existing customers and how they interact with your company.

Kibana's Discover feature with a simple query showing all free software keys in the last year.

After all, what is a search engine? It's a place where you ask a question and get an answer. Within a customer intelligence system, our company should be able to ask pretty much any question about our users and get an answer. What is the most common path that users follow through our web site before acquiring the product? Where do users get confused or leave the web site? Which industries have the most success with our product? What features are most popular among our customers? Every part of our company should be able to easily get answers to these questions and identify where we are successful, where we can make improvements and most importantly, how our decisions as a company affect our users.

Merging Data

Now the hard part. How do we merge all of that data together? With Elasticsearch, there's no way to join data at query-time, like a relational database. So we start with Logstash and create a configuration file for each data source that we're indexing. As we index these various data sources, we join the data so that we have a single record for each user. This requires having a unique key that all data sources share. Then we have a single user record with data from every source.

Once we have the data in Elasticsearch, then we can start working with Kibana. Kibana searches can be done with queries using the Lucene search syntax or a very pretty GUI that allows you to add and remove filters as you build your search. Once you get the search results that you want, you can add a visualization of that data (charts). Kibana has an amazing set of easy to build charts - bar charts, pie charts, line charts, tables, maps, and on and on. This is really where you can see the power of the ELK stack. This is what turns data into information.

Kibana visualization of website users by location

Kibana's Dashboard feature

The most impressive thing about the ELK stack was the ease in which I was able to install all three pieces of software. It does take a little bit of command line know-how to install it yourself, but the instructions found at https://www.elastic.co/products really do a great job of helping you install on Windows or Linux. Or, you can have your stack managed on the cloud at https://aws.amazon.com/elasticsearch-service or https://www.elastic.co/cloud/as-a-service. 

So, don't let the name fool you. Elasticsearch isn't just a search engine for a web site or application. It has many other uses when applied internally as a business system. If you find that you have data spread throughout your company, give the ELK stack a try and take the power back.

The next blog post in this series will be about moving the ELK stack into production. At this point, the tool should be stable enough for anyone in the company to use and data should be ingested in regular intervals so that users are always getting up to date information. As a business system, it will be monitored to verify that Elasticsearch, Logstash and Kibana are all functioning correctly and that all of the data is current. It just so happens that I work for a monitoring company and we have an Opspack for monitoring Elasticsearch. How lucky is that?