Python - Revisiting Python 2.7

Tutorial
https://docs.python.org/2.7/tutorial/index.html

Python Standard Library
https://docs.python.org/2.7/library/index.html#library-index

Language Reference
https://docs.python.org/2.7/reference/index.html

AWS - Disks and MongoDB - The shocking conclusion

Probably not so shocking.

We spun up some new primaries on aws "m3.2xl" instances.
These had enough drive capacity for our data.
Yes, we would have liked to use "i2" instance types, but they weren't available to us (dont ask).

We ran 'mongoperf' on them and here is a sample of the results:
------------------------------------------------------------------------------

EBS

• 986 ops/sec 3 MB/sec
• 1154 ops/sec 4 MB/sec
• 1139 ops/sec 4 MB/sec
• 1268 ops/sec 4 MB/sec
• 1119 ops/sec 4 MB/sec
• 930 ops/sec 3 MB/sec
• 929 ops/sec 3 MB/sec
• 1330 ops/sec 5 MB/sec
• 1341 ops/sec 5 MB/sec
• 946 ops/sec 3 MB/sec
• 892 ops/sec 3 MB/sec
• 1131 ops/sec 4 MB/sec
• 1153 ops/sec 4 MB/sec
• 1073 ops/sec 4 MB/sec
• 1071 ops/sec 4 MB/sec
• 1316 ops/sec 5 MB/sec

SSD (ephemeral)

• 8790 ops/sec 34 MB/sec
• 8879 ops/sec 34 MB/sec
• 8986 ops/sec 35 MB/sec
• 8976 ops/sec 35 MB/sec
• 8109 ops/sec 31 MB/sec
• 8610 ops/sec 33 MB/sec
• 8942 ops/sec 34 MB/sec
• 8574 ops/sec 33 MB/sec
• 8565 ops/sec 33 MB/sec
• 8292 ops/sec 32 MB/sec
• 8357 ops/sec 32 MB/sec
• 8722 ops/sec 34 MB/sec
• 7123 ops/sec 27 MB/sec
• 7863 ops/sec 30 MB/sec

------------------------------------------------------------------------------------------

 

So like.. 7x 8x performance.

 

Needless to say... when we ran our performance tests again, we Crushed our old timings.

With EBS, our slowest 4 API calls were taking (90th percentile):  2.5, 2.4, 2.4 and 2.4 seconds.

With SSD, our slowest 4 API calls were taking (90th percentile):  0.8, 0.8,  0.4 and 0.4 seconds.

 

 

 

Oh.. and why did we think we were I/O bound in the first place?

Cuz IOSTAT said this about our EBS:

 

 

 

And maybe i'll get around to posting some IOSTAT graphs w/the SSD measurements...

 

AWS - Disks and MongoDB - Part 1

So the general consensus that I'm getting is that Ephemeral Storage is generally quicker than EBS storage.

EBS volumes are accessed via network.

Good article about Mongoperf & IOStat:
http://java.dzone.com/articles/tips-check-and-improve-your

Read the conclusions here:
http://victortrac.com/ec2-ephemeral-disks-vs-ebs-volumes-in-raid.html

Note Ephemeral VS EBS comparisons:
http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html


Tools Takeaway:
mongoperf
iostat
hdparm -t
bonnie++
iozone

TODO - Read SICP

http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-4.html#%_toc_start

Java - Lambda

(String first, String second)
     -> Integer.compare(first.length(), second.length())

Just 2 parts:
A lambda expression is simply:
  a block of code,
  together with the specification of any variables that must be passed to the code.

http://www.drdobbs.com/jvm/lambda-expressions-in-java-8/240166764
http://java.dzone.com/articles/why-we-need-lambda-expressions
http://java.dzone.com/articles/why-we-need-lambda-expressions-0

http://stackoverflow.com/questions/3259322/why-use-lambda-functions

Linux - Wait To Run script

Had to wait for a process to finish before I could run a cleanup job.
Got sick of waiting for process to finish so scripted a "wait loop":


while [ $(ps -ef | grep dupl |wc -l) -ne 1 ]
do
echo waiting....
sleep 30
done

duply prd-usw2a-pr-05-prdmdbshard-modb-s02-0001.prv-openclass.com purge --force

Linux - IOSTAT

Ended up in the weeds on this topic in a hurry....

http://www.r71.nl/index.php?option=com_content&view=article&catid=7:technical-docs&id=185:disk-queue-length-vs-disk-latency-times-which-is-best-for-measuring-database-performance&Itemid=50

http://www.pythian.com/blog/basic-io-monitoring-on-linux/

http://blog.jcole.us/2007/05/08/on-iostat-disk-latency-iohist-onward/

http://dom.as/2009/03/11/iostat/

http://www.admin-magazine.com/HPC/Articles/Monitoring-Storage-with-iostat

https://www.igvita.com/2009/06/23/measuring-optimizing-io-performance/

http://linux.die.net/man/1/iostat

Graphite - Resources

We are going to have to scale graphite sooner or later... the following links include info on the topic:
http://www.aosabook.org/en/graphite.html
http://graphite.readthedocs.org/en/latest/carbon-daemons.html



Carbon Daemons:

1. carbon-cache.py
2. carbon-relay.py  - replication and sharding
3. carbon-aggregator.py - for buffering metrics to reduce I/O (data is available later though)

 Config Files:

1. carbon.conf
2. relay-rules.conf - send certain metrics to certain backends (sharding)
3. storage-schemas.conf - defines retention polices. Whisper preallocates these files.
4. storage-aggregation.conf - defines how to aggregate data to lower-precision retentions (dflt is avg)
5. aggregation-rules.conf - allow you to add several metrics together as the come in

Send data to graphite (actually carbon component) via:

1. plain text
2. pickle
3. amqp

AWS - Xen Reboot

So... AWS reboot impacted us a bit.
We had to provision all our instances in 'us east 1c'
Then we had to start adding the use1c haproxies to the ELB so that traffic would start flowing into use1c.  Then we had to disconnect the use1b haproxies so traffic would stop going to use1b.
Then we just waited until AWS finished doing use1b reboots.

Next step is to bring up live nodes in both AZs.   Yay redundancy.