Consul

Main Stuff
Service Discovery - provide a service or find a service
Health Checking - to avoid unhealthy nodes
Key/Value Store - includes a hierarchical kvs
Multi DataCenter - yay high availability

Architecture
Service provider runs a Consul Agent
Consul Agent talks to a Consul Server
Consul Server can run/should in a cluster.
Recommended one cluster per datacenter.
Don't need to run a Consul Agent to find a service provider.

from http://www.consul.io/intro/

Very nice compare/contrast of Consul vs various other tools.
http://www.consul.io/intro/vs/index.html

Java - Exception Handling

I was recently asked to improve the exception handling & logging on one of our services.
I figured that was a good prompt for me to go re-read some stuff on the topic.

Had ended up here before, but wanted to paste it cuz I still find it to be a great resource:

http://tutorials.jenkov.com/java-exception-handling/exception-wrapping.html
(just one page of many)


Also.. a nice list of antipatterns:
https://today.java.net/article/2006/04/04/exception-handling-antipatterns#antipatterns

ElasticSearch - Just Some Commands

Some GET commands:
http://someFlippingHostname:9200/_aliases?pretty=1
http://someFlippingHostname:9200/_status
http://someFlippingHostname:9200/_stats/docs

Send a DELETE to this and you can tank an index:
http://someFlippingHostname:9200/someFlippinIndexName

Linux - Script for showing size of subdirectories

So our log service is 98% full.
Thing is, iunno anything about our log server.
So I'm trying to figure out where all the junk is.
This is what I came up with:

 
SCRIPT:
for dir in $1/*/
do
    dir=${dir%*/}
    du -sh $1/${dir##*/}
done

USAGE:
./testScript.sh '/home'

OUTPUT:
8.0K    /home/appuser
4.0K    /home/logstash
32K     /home/ubuntu

Metrics - Time to evaluate Metrics by Coda Hale

https://github.com/dropwizard/metrics
http://metrics.codahale.com/getting-started/

INSTRUMENT IT
COLLECT IT
MONITOR IT (zabbix)
AGGREGATE IT (graphite)

Gauges
Counters
Meter
Histogram
Timer
HealthChecks

Reporting via JMX   -> JConsole or VisualVM
Reporting via HTTP
Reporting via Graphite


available on Maven

Humor - DevopsReactions at Tumblr

I wish this thing never ended:
http://devopsreactions.tumblr.com/

MongoDB - ow it burns

We had a replicaset with a member who no longer existed.
Iunno if someone added a host that no longer existed to the replset or if someone straight up deleted the host while it was a member of the replset.
It destroyed our ability to query mongos. period.  rather worry-some.


rs.remove() the bad host
rs.add() a new host
and was able to query mongos again.   yeck.

HAProxy out the ears

Insomnia? Try this:
http://haproxy.1wt.eu/download/1.5/doc/configuration.txt

When you get 503's with "sH" in your HAProxy Logs, maybe its time to make sure your service actually responds in a reasonable time frame.

sH   The "timeout server" struck before the server could return its
          response headers. This is the most common anomaly, indicating too
          long transactions, probably caused by server or database saturation.
          The immediate workaround consists in increasing the "timeout server"
          setting, but it is important to keep in mind that the user experience
          will suffer from these long response times. The only long term
          solution is to fix the application.

Continuous Itegration with Git, Jenkins, Docker

Gym

If you do shoulder flies in front of the dumbell rack at the gym, I Hate You.
Or any other exercise directly in front of the dumbell rack for that matter.

Kibana

So we were recently suspected to be causing a spike in logging into ElasticSearch/Kibana.

Had to prove our innocence.

Step 1.... Open Kibana for the correct environment.  Pick the approximate Time Range where the spike was happening.

Step 2.. Narrow down the Time Range to a smallish portion inside the spike by selecting a range on the graph up top.

Step 3.. Click on HOST then SCORE

 Step 4... Find real culprit
 

Step 5... Beer