PuppetConf 2014

This last Tuesday (the 23rd) I took the opportunity to listen in on PuppetConf via live web stream. In particular I listened in on the segment on Sensu which was presented by Tomas Duran, a Yelp employee. During this segment Tomas went over the challenges of monitoring, and how Yelp needed to evolve their own system monitoring in order to meet business needs. He laid out how monitoring stereotypically works in today's work environments and then broke into what Sensu is and how it helps break these stereotypes. The part that I focused in the most was how Yelp is using Sensu. They do not use all the components of Sensu, and have tried to simplify the workflow by only having *standalone checks run on all their clients. This helps reduce complexity when using configuration management tools such as Puppet, and also tightens security. Tomas also talked about using custom variables within checks that handlers would use for notification and escalation using tools such as Pagerduty and Atlassian's JIRA. Also during this segment Tomas presented how Yelp uses a custom 'monitoring_check' puppet module to build and manage all service checks on their nodes, which has become a research point within my project. I would like to utilize their monitoring check module, or something similar in order to make service management easier across all nodes like Tomas presented. 

* A standalone check is like a normal check, but instead of the server pushing a request to the client and running the check, the client runs the check locally and the publishes the response up to the server.

Install/Setup

Installation and Setup

So it occurred to me that I completely forgot to post an abbreviated installation and setup for Sensu, which in my opinion is a pretty important point. So the following is a brief checklist of steps that need to be taken in order to install and setup a basic Sensu framework (CentOS). This information was summarized from http://sensuapp.org/docs/latest/guide.

• To begin you will need two machines for a basic setup. These two machines will be referred to as the 'server' and the 'client'.
• On the 'server' machine, you will need to generate SSL certificates for Sensu and install both RabbitMQ and Redis.
• Installing RabbitMQ will involve installing dependencies, installing RabbitMQ, configuring SSL, and adding user credentials.
• Installing Redis is very simple, a yum install is all that is needed (given the dependencies are already installed).
• On both the 'server' and 'client' systems you will then need to install Sensu and create the appropriate SSL locations and move the previously generated certificate files into place.
• Now you will configure the Sensu connection, Sensu API, and Sensu clients. The Sensu connection and Sensu clients will be setup up on both systems, while the Sensu API is configured on the 'server' machine only. Configuration is done entirely by creating/editing JSON files within /etc/sensu/
• After these steps have been completed (correctly) you should be able to enable and start up the sensu-server, sensu-api, and sensu-client on the 'server' machine; and enable/start the sensu-client on the 'client' machine.
• Dashboard: In previous versions of Sensu a dashboard was provided within the package. However, the more current releases no longer include this feature and the Sensu Installation Guide recommends using Uchiwa as the Sensu dashboard, as it is up to date with the latest Sensu API changes.
• Uchiwa in provided via a yum package and is configured in a JSON file under /etc/sensu/

Spammed!

Oops!

So I was getting the first checks on my sensu client to report back to the server and these checks were controlled via an email handler. I had a ping check running every minute (I thought), that was causing some issues due to permissions and thought I had it resolved by the time I got done working on it. Turns out that the issue persisted and I left the check on, and in reality the check was occurring every second. So when I started up my computer and opened my email and was massacred by around 65,000 emails all saying that the ping check failed (18 hours * 60 minutes * 60 seconds ~ 65,000). 

So the lesson that I learned was make sure to double check that checks are reporting correctly before you leave for the day, and if they are not turn off the handler so your email doesn't get blown up by failed check reports.

Sensu

Overview

According to sensuapp.org, Sensu is

"often described as the “monitoring router”. Essentially, Sensu takes the results of “check” scripts run across many systems, and if certain conditions are met; passes their information to one or more “handlers”. Checks are used, for example, to determine if a service like Apache is up or down. Checks can also be used to collect data, such as MySQL query statistics or Rails application metrics.Handlers take actions, using result information, such as sending an email, messaging a chat room, or adding a data point to a graph."

 Sensu is written in Ruby, can use existing Nagios plugins, is configured entirely in JSON, has a message oriented architecture, using RabbitMQ and JSON payloads, and is available via 'omnibus' packages.

There are only a few dependencies required to run Sensu. The Sensu server and API must be connected to a RabbitMQ instance for message routing and Redis for storing persistent data. The Sensu client will also need access to the RabbitMQ instance in order to get messages to the server. The following diagrams shows a basic workflow for Sensu checks and responses.

 

Project Discovery

Project Ideas

Below is a list of project's that I considered while brainstorming:

• Using Sensu as a system monitoring solution
• Setup a virtualized development environment using packer,virtualbox, and vagrant
• Research continuous integration tools and give detailed analysis of my findings
• Build and implement an ELK stack (Elasticsearch, Logstash, Kibana)
• Working with Ruby to solve a problem common to sysadmins/devops

After considering the implications, extent, and difficulty of each I arrived at the conclusion of using Sensu as a system monitoring solution as my senior project. I will be maintaining my project at my place of employment(Buckle Inc.), as they have some interest in Sensu.