TODO - Play more with Sensu

Esp after reading this guys thoughts:

http://petey5king.github.io/2012/03/30/sensu-a-collectd-replacement.html

TODO - Grafana

Grafana is a frontend for Graphite

http://play.grafana.org/#/dashboard/file/default.json

http://grafana.org/blog/

TODO - ElasticSearch Marvel

I think marvel is to elasticsearch as mms is to mongodb...

http://www.elasticsearch.org/guide/en/marvel/current/

http://grey-boundary.com/elasticsearch-marvel-is-awesome/

Work - VM count and breakdown

2 person team (myself included) responsible for about 700 nodes:

• ~200 production
• ~200 performance
• ~200 integration
• ~50 test
• ~50 dev

Production breakdown:

• APIs
• 14 api x 4 instance x 2 stacks = 112
• 3 api x 8 instances x 2 stack = 48
• Mongo
• 8 shards x 5 replica set members  = 40
• 3 config nodes
• Infrastructure
• metrics (graphite / statsd / collectd)
• logging (logstash / elasticsearch / kibana)

Metrics - StatsD and Graphite

Nice write up:
http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/

Gritty details for Graphite:
https://graphite.readthedocs.org/en/latest/

Nice article about scaling Graphite:
http://grey-boundary.com/the-architecture-of-clustering-graphite/

A billion tools that work with Graphite:
http://graphite.readthedocs.org/en/latest/tools.html

Git - Git Flow revisited

https://www.atlassian.com/git/workflows#!workflow-gitflow

Metrics - Beauty in Metrics

Load of 30ish mongodb servers.

Disk Time Writes for some mongo nodes in an availability zone.

These nodes were randomly assigned a time every hour to backup.

Graph shows six hours so you can see the repetition.

Hadoop - Hive and Pig and Impala

Hive and Pig seem to both be query languages for Hadoop.
Hive seems to have a more familiar SQL style language.
Hive is a framework for performing analytic queries.
Pig uses a new language (pig latin).


http://vision.cloudera.com/impala-v-hive/ 
http://www-01.ibm.com/software/data/infosphere/hadoop/hive/
http://www-01.ibm.com/software/data/infosphere/hadoop/pig/




Other notes:

Hadoop is basically 2 things - a Distributed FileSystem(HDFS) + a Computation or Processing framework(MapReduce).

HBase sits on top of HDFS. HBase provides random read/write access.
HBase similar to Google's BigTable.

Kafka.. eventing & queuing ....somewhat similar to Rabbit

Kibana - Great start for search syntax

http://www.elasticsearch.org/guide/en/kibana/current/working-with-queries-and-filters.html

TechTodo - Storm, Mesos, Chronos

Maybe someday I'll read these in full:

http://storm.incubator.apache.org/

http://mesos.apache.org/


http://airbnb.github.io/chronos/

Linux - netstat almost made my head explode

Created a newer faster metrics server cuz our old metrics server was too heavily loaded.
Updated the DNS entry to point to the new machine.
Kicked a buncha clients so they'd read the updated DNS entry.
ran  'netstat -pant' on the old metrics server and found a TON of IPs still connected to it.
scripted a reverse-dns-lookup against all the IPs.

Turns out there were a buncha hosts from OTHER teams connected to our server !
(they should not be!!!!)
Now I have to go track them all down and tell them to go away.

Linux - one of the best commands ever

sudo du -sh *

Gives you the sizes of all subdirectories.  Awesome.

Puppet, Hiera - Some notes on Hiera

Puppet Class Parameters priority:
  1.. use value that was explicitly set
  2.. check hiera for ClassName::ParameterName
  3.. use default value in class def
  4.. fail compilation with error

Hiera lookups get a copy of ALL variables currently available to puppet.
Hiera can use these in "interpolation tokens"

Hiera lookup functions:
  hiera - priority lookup
  hiera_array - array merge lookup
  hiera_hash - odd


src:  https://docs.puppetlabs.com/hiera/1/puppet.html#hiera-lookup-functions

MongoDB, Metrics - Taste the Rainbow

If that graph doesn't make you say "holy shit", then you just don't know what you are looking at.

ElasticSearch - good resources

good intro
http://joelabrahamsson.com/elasticsearch-101/

elasticsearch guide
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/index.html

elasticsearch reference
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index.html

exploring elasticsearch Beta book
http://exploringelasticsearch.com/overview.html

Cars - Tutto Italiano at Larz Anderson