Book - Release It

Part 0

• Preface
• Introduction
  • Aiming for the Right Target
  • Use the Force
  • Quality of Life
  • The Scope of the Challenge
  • A Million Dollars Here, a Million There
  • ePragmatic Architecture

Part I - Stability

• The Exception That Grounded an Airline
  (an anecdotal story setting up problems to discuss solutions for in the next chapter)
  
  • 2.1   The Outage
  • 2.2   Consequences
  • 2.3   Post-mortem
  • 2.4   The Smoking Gun
  • 2.5   An Ounce of Prevention?
• Introducing Stability
  (software must be cynical. it must expect bad things to happen and never be surprised when they do. failures cause loss of business and money and reputation)
  
  • 3.1   Defining Stability
    (transient impulses and/or persistent stress manifest in strain, which could be increased RAM usage, high load, etc) 
  • 3.2   Failure Modes
    (*something* will fail first. that crack in the system can propagate and lead to a failure mode.  Systems always have a variety of failure modes.  Pearson Anecdote:  think subpub...and then usercomposite... and them sms) 
  • 3.3   Cracks Propagate
  • 3.4   Chain of Failure
  • 3.5   Patterns and Antipatterns 
• Stability Antipatterns
  (antipatterns create, accelerate cracks in a system.  this chapter discusses common antipatterns you should avoid.)
  
  • 4.1   Integration Points
    (integration points are the #1 killer of systems.  counter these problems with "Circuit Breaker" and "Decoupling Middleware" patterns.  make a test harness to simulate problems in 3rd party system so you can check your stability.) 
    • Socket Based Protocols
      (refused connections, slow-to-respond-refused-connections, slow reads.  slow responses are worse than no response often)
    • HTTP Protocols
      (jdk offers NO read timeout.  jakarta commons HttpClient does.  always have a timeout.)
    • Vendor APIs
      (little to no control on these... treat them as suspect)
  • 4.2   Chain Reactions
    (problem inherent in an app. one app in a cluster fails, and the others must pickup the slack.. increasing their likelihood of failing. avoid these with "Bulkheads")
  • 4.3   Cascading Failures
    (when problems in one layer cause problems in callers.  think database failure.  avoid these with "Circuit Breaker" and "Timeouts" pattern)
  • 4.4   Users
    (keep as little in sessions as possible to minimize memory consumption. use SoftReferences to large objects.  identify your most expensive transactions. ) 
  • 4.5   Blocked Threads
    (can get into a situation where every thread is blocked waiting for some impossible outcome.  synchronize only when necessary. scrutinize resources pools. use timeouts.) 
  • 4.6   Attacks of Self-Denial
  • 4.7   Scaling Effects
  • 4.8   Unbalanced Capacities
    (make sure a big cluster doesnt overload a smaller cluster it is dependent upon)
  • 4.9   Slow Responses
    (track your own response time.. consider failing-fast when it exceeds your SLA.)
  • 4.10  SLA Inversion
    (dont offer a better SLA than one of your dependencies!)
  • 4.11  Unbounded Result Sets
    (beware a result that is bigger than you can handle.)
• Stability Patterns
  • 5.1   Use Timeouts
    (networks are unreliable.  any resource pool that can block a thread Must have a timeout.   queue work for a slow retry later.)
  • 5.2   Circuit Breaker
    (detect problems, open the circuit.  prevent the operation instead of executing it and having it fail)
  • 5.3   Bulkheads
    (protect critical clients by giving them their own pool of resources.)
  • 5.4   Steady State
    (dont fiddle around with a system.  remember to purge junk data.  cache carefully and remember invalidation.)
  • 5.5   Fail Fast
    (dont waste resources if you are just gonna throw out the result.  dont do useless work)
  • 5.6   Handshaking
    (can support throttling)
  • 5.7   Test Harness
    (emulate out-of-spec failures, stress the caller via slow responses or no responses)
  • 5.8   Decoupling Middleware
• Stability Summary

Part II - Capacity

• Trampled by Your Own Customers
  • 7.1   Countdown and Launch
    (nice anecdotal story about a website crashing) 
  • 7.2   Aiming for QA
    (dont build things to pass QA, build them to run in production) 
  • 7.3   Load Testing
    (tough to know real prod traffic.) 
  • 7.4   Murder by the Masses
    (number of sessions had killed the site.  there were tons of bad clients.) 
  • 7.5   The Testing Gap
    (test with stupid and ill behaved clients!!) 
  • 7.6   Aftermath
    (be able to black-list IPs.  be able to "fail fast" and return a simple error page when overloaded.)
• Introducing Capacity
  • 8.1   Defining Capacity
    (performance:  how fast the app handles one transaction. can be isolated or under load.  throughput: number of transactions the app handles in a timespan.  capacity:  max throughput a system can sustain while maintaining an ok response time.) 
  • 8.2   Constraints
    (one constraint determines capacity. its whatever limiting factors hits its ceiling first.  find your system constraint. use it to plan capacity improvements.)
  • 8.3   Interrelations
  • 8.4   Scalability
     (horizontal scaling vs vertical scaling)
  • 8.5   Myths About Capacity
    (cpu aint cheap.  storage aint cheap.  bandwidth aint cheap.  dont code like a jerk) 
  • 8.6   Summary
    (capacity planning needs monitoring and optimization.) 
• Capacity Antipatterns
  • 9.1   Resource Pool Contention
    (contention for resources can cause your CPU to waste its time waiting.  make your resource pool size equal to the number of threads.) 
  • 9.2   Excessive JSP Fragments
  • 9.3   AJAX Overkill
    (avoid needless requests. make sure each ajax request doesnt create a new session.  minimize response sizes.) 
  • 9.4   Overstaying Sessions
    (set session timeout to one standard deviation past the average think time.  keep keys, not whole objects.) 
  • 9.5   Wasted Space in HTML
    (omit needless characters.  remove whitespace.) 
  • 9.6   The Reload Button 
  • 9.7   Handcrafted SQL
    (dont do it, often it will be a non-indexed query.  test queries against prod-like database sizes.) 
  • 9.8   Database Eutrophication
    (archive and eliminate. dont do transactions and reporting!  create indexes. ) 
  • 9.9   Integration Point Latency
  • 9.10  Cookie Monsters
    (dont trust cookies, minimize cookies as they are sent with each http request.  use cookies for identifies, note entire objects.) 
  • 9.11  Summary
    (stay in school, dont do drugs) 
• Capacity Patterns
  ("we should forget about small efficiences about 97% of the time:  premature optimization is the root of all evil."  balance small gains against complexity.)
  • 10.1  Pool Connections
    (connection pooling size is critical. an undersized pool leads to resource pool contention, oversized can lead to excess stress on DB. tune it. also, ensure timeouts.) 
  • 10.2  Use Caching Carefully
    (monitor hit rates.  if they are low, your caching isnt helping and may be hurting.  dont leave cache size unbounded.  build a flush mechanism.) 
  • 10.3  Precompute Content
    (precompute pieces that change infrequently. serve that piece many times!) 
  • 10.4  Tune the Garbage Collector
    (untuned apps can spend 10% time in garbage collection. tuning can reduce to around 2%.  use jconsole to show heap usage and time in garbage collection.  adjust ratios of relative sizes for generations)
  • 10.5  Summary 

Part III - General Design Issues

• Networking
  • 11.1  Multihomed Servers
    (have admin&monitoring on its own highly secure network.  partition backup traffic onto its own network segment. bind apps to the Non admin network.) 
  • 11.2  Routing
  • 11.3  Virtual IP Addresses
    (can be moved from one NIC to another) 
• Security
  • 12.1  The Principle of Least Privilege
    (process should have the lowest level of privilege needed to accomplish a task.  developers too!  each major app should have its own user. ) 
  • 12.2  Configured Passwords
    (keep passwords separate from  from config files.) 
• Availability
  (do not divorce a Want from its Cost.)
  
  • 13.1  Gathering Availability Requirements
    (with uptime, determine actual cost vs avoided losses.) 
  • 13.2  Documenting Availability Requirements
    (better to define the SLAs in terms of specific features.  see the list of SLA bulletpoints) 
  • 13.3  Load Balancing
    (DNS roundrobin. Reverse Proxy and x-forwarded-for header. hardware like F5 BigIP.) 
  • 13.4  Clustering
    (ok if no other alternatives for redundancy. doesnt really address scalability) 
• Administration
  • 14.1  “Does QA Match Production?”
    (keep apps separated via vms if need be.  one-to-one vs one-to-many.) 
  • 14.2  Configuration Files
    (keep production config away from plumbing config.) 
  • 14.3  Start-up and Shutdown
    (build clean startup sequences and dont accept connections until startup is complete. ) 
  • 14.4  Administrative Interfaces
    (GUIs are intuitive, but suck to script and automate) 

Part IV - Operations

• Phenomenal Cosmic Powers, Itty-Bitty Living Space
  • 16.1  Peak Season
  • 16.2  Baby’s First Christmas
  • 16.3  Taking the Pulse
  • 16.4  Thanksgiving Day
  • 16.5  Black Friday
  • 16.6  Vital Signs
  • 16.7  Diagnostic Tests
  • 16.8  Call in a Specialist
  • 16.9  Compare Treatment Options
  • 16.10 Does the Condition Respond to Treatment?
  • 16.11 Winding Down
• Transparency
  (transparency refers to the qualities that allow people to understand the systems historical trends and present state.  component level visibility is important.)
  
  • 17.1  Perspectives
    (Different people needs different perspectives:  historical trending, predicting the future, present state, instantaneous behavior) 
  • 17.2  Designing for Transparency
    (transparency needs to be built in.) 
  • 17.3  Enabling Technologies
    (blackbox sits outside.  whitebox sits inside and are integrated during development) 
  • 17.4  Logging
    (logs are still invaluable.  'error' or 'severe' should require ops intervention.  make logs human readable both in format and content.) 
  • 17.5  Monitoring Systems
    (want external monitoring in case your main process is hung. ) 
  • 17.6  Standards, De Jure and De Facto
    (SNMP, JMX, CIM) 
  • 17.7  Operations Database
    (for metrics vs production data.) 
  • 17.8  Supporting Processes
    (review problems weekly.  solve the most time consuming.) 
  • 17.9  Summary
    (transparency can be the difference between a system that improves and a system that decays.) 
• Adaptation
  • 18.1  Adaptation Over Time
  • 18.2  Adaptable Software Design
  • 18.3  Adaptable Enterprise Architecture
  • 18.4  Releases Shouldn’t Hurt
  • 18.5  Summary

Been Busy I guess

haven't posted since late november.. guess i'm going to blame the holidays and this wave of reading i've been doing.

i've read (not memorized....)

Continuous Delivery by Jez Humble & David Farley
Release It by Michael Nygard
Clean Code by Robert Marting
How to Win Friends and Influence People by Dale Carnegie

and will be posting some book reports here in the near future i hope.