Monitoring - Jenkins, Fabric, Python

So... there are lots of monitoring tools out there.
But there always seem to be things that just aren't covered.
We use Zabbix, StatsD, CollectD and MMS.
But still find things that aren't handled the way we want.

Currently working on a API monitoring solution that can:

1. do chains of calls that need to happen in order
2. call into various levels of our stack (we have a lot of redundancy which can mask individual component failures... but we'd obviously still like to know about them)

Planning to have Jenkins call a Fabric Task that will do the real work.  We already have a buncha fabric modules for getting/creating/destroying infrastructure components, so that code can be leveraged to dynamically lookup the infrastructure components that we want to monitor.

The general solution should also be able to support 'app specific' monitoring.
For example:

1. use PyMongo to query various MongoDB values... like if Balancing is enabled.
2. use the Requests python module to query restful endpoints on our Haproxies & ELBs to confirm they are up and healthy.

And cuz this feels like an exceedingly verbose and not visually appealing post, here is a link to something on devops reactions:

 "Before diving into Legacy Code"

haproxy - v1.5 supports SSL

Probly going to get rid of Stud.
Team is reporting that Stud doesn't seem to include the header "X-Forwarded-For".
Hopefully HaProxy will include it.....

• https://serversforhackers.com/haproxy-ssl-termation-pass-through/
• https://www.digitalocean.com/community/tutorials/how-to-implement-ssl-termination-with-haproxy-on-ubuntu-14-04#tutorial_series_3
• http://seanmcgary.com/posts/using-sslhttps-with-haproxy

On a side note, there are 3 headers in that same vein:

1. x-forwarded-for
1. for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer
2. x-forwarded-host
1. for identifying the original host requested by the client in the "Host" HTTP request header
3. x-forwarded-proto
1. for identifying the originating protocol of an HTTP request, since a reverse proxy (load balancer) may communicate with a web server using HTTP even if the request to the reverse proxy is HTTPS

APT - FPM, S3, Reprepro

Wanting Kibana 4 as a package so it can be installed easily by Puppet.
Seems like the elasticsearch folks haven't released it as a package yet cuz its not GA.
Gonna have to fight with FPM to make a package.. and then  reprepro & s3cmd to get the package up onto S3.

http://mirrorer.alioth.debian.org/
http://s3tools.org/s3cmd

http://www.jejik.com/articles/2006/09/setting_up_and_managing_an_apt_repository_with_reprepro/
http://reprepro.googlecode.com/svn-history/r1218/trunk/docs/short-howto

https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html
https://www.debian.org/doc/manuals/developers-reference/beyond-pkging.html

http://www.aptly.info/

ELK - why a msg queue is important

People often wonder why the heck there is a broker. Sometimes its Redis, sometimes AMQ or RabbitMQ.

A good answer is below:
http://stackoverflow.com/questions/26930047/why-redis-amqp-or-0mq-is-needed-along-with-elastic-search-and-logstash

Some good reads about setting up AMQ or Redis:
http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/
http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-redis/