On Staging with Python and Setuptools

tl;dr One environment per application revision (think package). Packages are immutable. Use symbolic links to switch between packages. Download from Github.

You’ve laboured and toiled over your code and after endless hours of polishing your app finally is the gem you intended to build. But how to bring it into production? Is there a good practices? Actually yes, there is:

Test in a clone of the production environment. The point of testing is to flush out, under controlled conditions, any problem that the system will have in production. If you test in a different environment, every difference results in a risk that what happens under test won’t happen in production.

Martin Fowler, Continuous Integration

To wrap up my understanding of staging based on Fowler’s recommendation:

1. Decrease uncertainty
2. Lever your application onto the target platform and verify its correctness
3. Provide an atomic and non-intrusive (one-step) deployment work-flow
4. Ensure a role-back option

In my opinion it is therefore necessary to uphold a set of invariants:

• Every revision operates independent of any other code present on the platform
• Every revision has its own environment
• Each revision and companion environment is an immutable package
• A verification is final

If you try to actually live up to this practice there appears to be little practical support, though. Take Setuptools for an example of a popular distribution scheme. I would love to read the success story of someone who has put Setuptools’ notion of “staging areas” and deployment into a non-intrusive work-flow without the premises I stated above.

Example

This setup is actually in use by Nerdcorner and also in a not-too-different manner by my former employer since 2009.

• Package repository: /var/webcode/{name-of-project}-{version}
• Source distributions via python3 setup.py sdist
• Two staging areas: staging and production
• Two Vhosts with document roots to /var/webcode/{staging,production}

A deployment to either stage (ie. staging) would require the following steps:

1. Build source distribution on development machine (or any other convenient platform)
2. Upload the distribution into the package repository
3. Build environment and install application into companion: deploy_py.sh projectA-0.4.tar.gz
4. Create symbolic link to the stage’s document root: rm staging && ln -s projectA-0.4 staging

That’s it. See my Github for the code: https://github.com/lusitania/keep-it-simple

Discussion

Every environment is shared by exactly one application revision. The reason for this is twofold:

• The develop Setuptools command maintains {easy-install.pth, *.egg-link} in the environment’s “staging area” site-packages that directly link the environment to the installed app. Setuptools always overwrites the application record in easy-install.pth such that only the recently installed revision will prevail. Same with projectA.egg-link which is also version-agnostic. This effectively sabotages the staging idea. Using --relocatable doesn’t help either as it only disables changes to easy-install.pth and not .egg-info. The option introduced some load failures with Pyramid (I suspect the frameworks didn’t anticipate this behaviour) which were too tedious to solve.
• With a shared environment uncertainty is introduced to already verified revisions with all subsequent setup runs: A setup may or may not install new packages that may or may not break apps previously verified. This directly contradicts the primary intention of staging (as established above).

Non-intrusive deployment. Honestly, putting your app to work is a side show. It’s a process you do once in a while, which never changes and simply has to work. You are getting paid for a running app, not for the process of making it so. That is why

• A single command should handle the bulk work (non-intrusive)
• Anything that can go wrong must be non-critical (building)
• Do the thinking only on critical actions (re-linking)

Atomicity. Critical steps must not fail. Any failure and verification must have happened before (package errors, failed unit test, …). In the example above: The system’s behaviour changes only once the vhost was re-linked. Removing and setting a symbolic link is the most atomic shell operation I can think of. This also doesn’t break Setuptools’ records in the companion environment since the application was installed in its true package location and not in one of the document roots. You are obviously not allowed to rename the folder in the package repository.

Role-back. Things can still go wrong. Don’t hamper yourself by overwriting previous states, keep your good stuff for a while.