Update

RabbitMQ Failover

So as of this post there is still not a fix for to support multiple broker connections within the Sensu puppet module. I submitted an issue via github and was asked if I could take a look at creating a PR to implement this update, but after looking at the module and talking with my coworker for some insight it seems a considerable amount of work would be needed in order to get this fix in place and would more than likely introduce breaking changes within the module. So for right now RabbitMQ is in the same boat as Redis and Sensu and will need to be load balanced if I want failover and high availability. As of right now load balancing these components has taken a back seat, and implementing checks and handlers has become my new focus.

Checks/Handlers

In order to implement checks/handlers I've decided (with guidance from my supervisor) to borrow from Yelp's sensu_handlers github project to create handlers and also use a similar style as Yelp uses in their puppet_monitoring_check project for checks. This will enable me to create a hiera file with team specific information, and define which team the check applies to, which will then be passed to the handler in order to notification routing and event processing. This also will solve, at least in my opinion, the problem that Sensu Core has with not providing contact routing configurations out of the box. 

Sensu Enterprise Announced

Sensu Enterprise

With the latest update to Sensu (0.16), there has been the added feature of installing/purchasing Sensu Enterprise. Sensu Enterprise hosts a slew of features not provided in the free version (Sensu Core), and some of these features kind of frustrate me that they need to be purchased. One such example is contact routing. This feature, provided only in Sensu Enterprise, allows you to create contact group and detail information of the medium to contact that individual or group and then add that contact to a check. I think of this feature as being a basic necessity when it comes to system monitoring rather than an Enterprise feature. I do think some of the features provided in Enterprise are reasonable though, such as: support for alternate transport brokers, "out of the box" support for third-party integrations, and the support features that are expected in enterprise level software packages (deployment assistance, premium support, etc.). The fact that contact routing is not supported in the Core software, in my personal opinion, just seems like bad business. If contact routing was supported in core it would give a more complete open source system monitoring solution, and give it another positive mark for those considering to migrate from another monitoring software such as Nagios.  

Milestones

Project Milestones

Below is a list of Milestones that I have set forward for my project. I hadn't posted any of this yet because my milestones kept evolving with my project, but now I believe that I have enough done and a good enough handle on things to set my milestones down more officially. This list will continue to change but hopefully I will just be adding details to the incomplete milestones as needed, rather than complete restructuring of this list.

• [DONE] Read Sensu docs and get a feel for what it is and how it works
• sensuapp.org/docs/latest/guide
• [DONE] Setup basic 2 machine Sensu environment manually (1 server and 1 client)
• [DONE] Sensu Server
• RabbitMQ
• Sensu API
• Sensu client
• Sensu server
• Redis
• [DONE] Sensu Client
• Sensu client
• connectivity to RabbitMQ
• [DONE] Add Uchiwa dashboard to server for web UI.
• provided in Sensu repo
• [DONE] Puppet implementation and dev environment setup
• [DONE] Create VMs via foreman
• sensu[1,2,3]
• sensu-server
• sensu-api
• sensu-client
• uchiwa
• sensu-mq[1,2]
• RabbitMQ
• sensu-client
• sensu-redis[1,2,3]
• Redis
• Redis Sentinel
• sensu-client  
• [DONE (tentatively)] Initial Puppet Setup
• Clone sensu puppet module from github
• Clone uchiwa puppet module from github
• Create sensu-server profile/role
• Create sensu-client profile/role
• Create sensu-redis profile/role
• Create sensu-rabbitmq profile/role
• Update existing redis module to allow for sentinel
• [DONE (tentatively)] Initial Hiera Setup
•  Create necessary files for each machine
• Setup and implement Sensu check/handler structure in Puppet/Hiera
• Modify Yelp's sensu_handlers to conform to our needs, and add any that may be missing.
• [DONE] mailer
• hipchat
• pagerduty
• Create custom sensu checks to work with custom handlers (borrow from Yelp's style)
• [DONE] Create package for sensu handlers/plugins so we can manage the installation of these components easily via Puppet (there is an issue filed with sensu to create these packages, but no timeline or projected date has been given)
• Testing/Troubleshooting
• Run puppet (successfully) on all VMs (1st iteration with initial configs is done, need to implement checks/handlers before this can be considered 100% done.)
• [DONE] Ensure necessary connections can be established
• Ensure master selection/failover is working properly for RabbitMQ
• With Sensu 0.16 multiple broker connections now supported (updates to puppet submodule to come soon hopefully).
• Conversation on github here: https://github.com/sensu/sensu-puppet/issues/269
• [DONE] Ensure master selection/failover is working for Redis + Sentinel
• Needs load balanced
• [DONE (tentatively)] Ensure master selection/failover is working for Sensu-servers
• Needs load balanced
• Ensure checks/handlers are working as expected
• Really hammer on handlers to make sure exponential backoff is working, or that handlers aren't blowing up hipchat/email/jira/etc.
• Ensure Uchiwa dashboard is displaying information correctly
• LDAP + Uchiwa 
• Accomplished by using Apache as a proxy for authentification, and turn off Uchiwa native auth.
• Code review(s)
• Clean-up/Refactor Code
• Explore Graphite integrations with Uchiwa
• You can embed graphs within Uchiwa and jazz up the interface more details on this can be found at http://roobert.github.io/
• Explore integrations with Consul?
• Explore creating a Sensu API command line tool via Ruby's OptParse?
• Demo Sensu?

Presentation preparation

• Provide documentation on how to start up Sensu manually, and provide well documented code in Puppet if clarification is needed.
• Presentation (ppt?, oral?, whiteboard outline?, project map?)