Python - dependencies - requirements.txt and setup.py

So none of us are particular python gurus.. we're all a buncha hacks..

We had project X which called out module Y and module Z in requirements.txt.
However... module Y also listed module Z in module Y's setup.py.
So we thought... we should remove module Z from project X's requirements.txt since it will be installed (by pip) as a transitive dependency when module Y was installed.

Then I read this post and decided we were wrong:

http://blog.miguelgrinberg.com/post/the-package-dependency-blues

Long story short:  when you are testing your app, its good to lock-down the versions you install and test against.  Thats what requirements.txt allowed us to do....specify a version for module Z.  Whereas if we allowed pip to decide the version of module Z by what was called out in Module Y's setup.py....that developer could end up with a newer version of the module Z which could include breaking changes.

*Module Y's setup.py had something like "module Z >= 1.0.0" and thus the upper version was not restricted.

Puppet - Roles and Profiles

So its been a long time overdue that we made improvements to our Puppet code.

For at least a year we've relied on assigning multiple 'roles' to a node using this "truth enforcer" module from jordan sissel (link).

The biggest problem we had was that we treated each 'role' independent of the others.  Each role lived in its own "IF" statement.  The enforcer would go through and add these roles (and thus their resources) to the catalog.  But, as you know, without specifying resource ordering, puppet could apply the roles in any order it wanted.  So we had cases where tomcat would come up before the mongo client (and that was bad). 

Sadly (we knew it was wrong, we just didnt have a better solution at the time), we put a dependency in our tomcat module to need mongo client to complete first.  It was frustrating to know that we had just tightly coupled our tomcat module to our mongo client module... but nobody had a better solution at the time.

Then I watched this "Doing the Refactor Dance by Gary Larizza" (link).

I realized that what we were calling 'roles' would really be more equivalent to what the more fluent puppet people out there would call 'profiles'.

The solution seems to be adding one more layer of puppet into the mix.  This new layer will truly be a role.  A role will include multiple profiles (we are demoting all of our exisiting roles to be profiles via naming conventions...) and the role will be able to declare dependencies across those profiles.

• Role (can define ordering of profiles) (conceptually business logic)
• Profile (can define ordering of modules) (conceptually a technology stack)
• Component/Module
• Classes & Resources
• Component/Module
• Classes & Resources
• Profile 
• Component/Module
• Classes & Resources
• Component/Module
• Classes & Resources

Anyways.. its a step in the right direction...

Also you can read these posts (link) (link).
And here's a pretty picture I stole from one of those links above:



p.s. i probably still suck at puppet