Go Splunking

See the forest *and* the trees

There are more than a few analytics packages out there, and some of them give you pretty decent insight into business operations – called Operational Intelligence. There are a lot of reasons why C-level executives (and even those lower on the food chain) would want that insight into business operations. Our problem (or opportunity) then, is that when we (as UI engineers) are faced with presenting that Operational Intelligence in a dashboard, what do we do?

For this post, I'm going to focus on Splunk for a few reasons. First, it's pretty widely used – most people familiar with business analytics have at least heard of Splunk. Second, building stuff for Splunk is a lot simpler than it seems at first glance, as you'll see if you keep reading this post. Third, I have a personal relationship to it – at least two of the four companies I've worked with in the last five years use it and I've been privileged to review multiple tech books about it over the last five years, which has let me experience it through others.

Two of the most common approaches when developing dashboards or apps for Splunk are using Splunk's Simple XML and using HTML, CSS, and JavaScript on the Splunk server to develop the apps everyone wants. There's another option though, and it's the option I would suggest – develop the app on a completely different server.

Developing the dashboard or app on an non-Splunk server lets the Splunk server do what it's really meant to do – index data and provide a query interface to the indexed data – and lets another server handle user requests and serve up the adaptive, responsive web pages more easily and consistently. In addition to resolving potential performance and maintenance issues, this also enables you to create web pages that comply with all the web accessibility guidelines easily (which is something Splunk isn't great at, to be honest). If the tools you use to develop your Splunk dashboard/app are the same tools you use for your other web apps, there are even more benefits (such as possibilities for code re-use).

Here's a quick and (admittedly) overly simplistic explanation of how to get started and have a basically free server that responds quickly and keeps security in mind.

1. Set up a web server using node.js
   For this you can use something like Express, but you might also develop your own. If you decide to develop your own, feel free to look at my node-experiments github repo at https://github.com/hrobertking/node-experiments. You'll want to use node.js for this (rather than using Apache or Nginx or something else) for reasons that will be revealed in the next step. For the purposes of this blog entry, I'm going to refer to this web server (which should really be dedicated to your Splunk apps) as the Splunk App Portal.
2. Add the Splunk SDK (JavaScript version) to your toolkit
   The Splunk JavaScript SDK requires node.js for server-side scripting. Of course you can install it easily (npm install splunk-sdk) and then use require in your script to access it. As part of adding the SDK to your toolkit, be sure to visit the Splunk>dev JavaScript portal (http://dev.splunk.com/javascript) to see the SDK documentation, including examples.
3. Instantiate a Splunk Service object
   Here's the relatively easy part

   var splunkjs = require('splunk-sdk'),   service = new splunkjs.Service({       host: 'my_splunk_server',       password: my_splunk_password,       port: 'my_splunk_server_port',       scheme: 'https',       username: my_splunk_username,       version: '5.0'     });

   Note that you should not ever code a username and password in a JavaScript file. If you pay careful attention to lines 4 and 7 you'll note that the values are not enclosed in single quotes – they are variables. Put in a little extra work and set the value of those variables from the command line. Using this approach enables your newly created Splunk App Portal to communicate with the Splunk server in a controlled manner. I also recommend establishing a Splunk account specifically for this purpose so that use can be monitored and controlled with greater granularity.
   
   I should point out that if you're using JavaScript to develop apps for Splunk another way, you're most likely adding the config.js file and calling the config method to pass in the username and password. It should go without saying (but won't) – don't hard code the username and password in JavaScript file. If you're doing something like the following, stop and fix it.

   <script src="MY_SPLUNK_FILES/splunkjs/config.js"></script> <script> splunkjs.config({     host: 'my_splunk_server',     port: 'my_splunk_server_port',     scheme: 'https',     authenticate :{       username :'DoNotDoThis',       password :'NoNotEver'     }   }); </script>

4. Call the oneshotSearch method to get your data
   Calling the oneshotSearch method is fairly straightforward, code-wise, anyway.

   service.oneshotSearch(query, params, function(error, results) {     //your rendering instructions go here   });

   Here's where it gets a little tricky. The query argument will be something like 'search index=myindex'. If you have even the slightest experience with Splunk, that query string will look familiar. You can, of course, use all the Splunk functions and so forth to craft your search string, just keep in mind that simpler searches are better. The params argument will be a JavaScript object that contains properties such as earliest_time and output_mode – such as var params={earliest_time:'-4h', latest_time:'now', output_mode:'json'};. I recommend using JSON as the output mode (though there are other modes) because it will be easier to reference when you render the response.
5. Render the response
   This is where the real magic happens. If you're looking up values – let's say you need to convert a location code like ATL or ORD to an actual longitude and latitude – this would be the place to do it. This is also where you will decide how you're going to format the response – whether it be HTML, JSON, XML, or even plain text. If you're going with HTML, you may want to consider using a templating engine (like dust.js or mustache.js). Using a templating engine means your response can be even more flexible, and using something like Adaro and Makara (see http://krakenjs.com/) gives you even more power.

Of course, using this approach (rather than Simple XML) means you'll have to actually write more of the web page(s), but you'd likely have to do most of the writing in any case. As is noted in the fifth step, if you're using dust.js or another templating engine, it's going to be pretty easy in general – and it will likely be even easier if you're rendering the response as JSON, where your Splunk App Portal is acting as a proxy for the Splunk server. Why might you want to do this? For many of the same reasons you might want to control data coming out of your database – most notably, who gets to see what. Separating functionality in this manner also means the systems are simplified, or at least simpler than combined systems. Since complex systems fail in complex ways, we're avoiding that trap as well.

Overall, even though this is an admittedly simplified explanation, this is a relatively easy process – a process that will likely benefit you in terms of both performance and maintenance once you have established the infrastructure to support it.

Since this is a fairly complex topic and I've not given much detailed information, if you have specific questions about the process or how to do something within this model, feel free to add them in the comments below and I'll address them as soon and as completely as I can.

Happy coding.