Characteristics of a sustainable buildout

July 27, 2017 - Updated the section called "Additional Background" to include reference to the Plone Intranet project. Also swapped out the blockquoted code snippet for a github gist instead.

Buildout is a software build system used to manage development and deployment setups, especially in Python. When done well, Buildout makes everything very repeatable. Widely used in Plone projects but not exclusive to such projects, I've used buildout with Pyramid, Tryton and even for Sphinx based documentation projects. I've also seen buildouts on github for Django, Flask and Odoo based projects. 

What I expect of a proper build system

• Easy to switch between development and production
• Easy management of packages and package versions
• Easy to share with other developers
• A workflow as close to what developers expect

Here's a breakdown of how we do our buildouts to ensure that these goals are met.

The anatomy of a sustainable buildout

There are a few things that I add to my buildout to help to keep things sane. The profiles and templates folder and a requirements.txt file. I discuss their usage below.

Profiles folder

The profiles folder holds the configuration files used by the buildout. Generally the ones that we link to directly are dev.cfg and prod.cfg. These then link to the supporting configuration profiles.

Templates folder

Templates are stored in the templates folder, these templates are used to dynamically generate configuration settings. Some of them include variables which are configured in the .cfg files found in the profiles folder.

requirements.txt to define build tools

I recently added the requirements.txt file to make the workflow closer to a more typical Python development workflow, for example most Flask developers expect such a file. Since Buildout is your build tool, we define it as a dependency in the requirements.txt so that a user can easily install it.

Getting started is a matter of running the well known "pip install -r requirements.txt" command. This installs the buildout command.

buildout.cfg is not stored by git

With this approach the buildout.cfg file is not stored in the repository, so it is added to the .gitignore file. The process of kick starting things requires that a user copies the buildout template from the templates folder into the root of their buildout.

Bootstrapping

Here's a quick example with a real project, the Python Jamaica website. I've added comments to describe each step of the process. I only assume that you are familiar with virtualenv and pip.

...
...

One gotcha to look out for, after bootstrapping a local command bin/buidout is used to run the actual build (not to be confused with venv/bin/buildout).

Thanks to Maik Derstappen for pointing me towards "buildout bootstrap" which replaces the older approach of using a special bootstrap.py file.

Day to day usage

Once the buildout is installed here are some day to day tasks that a developer might find themselves doing.

Launching an instance

During development a standalone instance can be launched with the command:

bin/instance fg

This launches the reference site on port 8080.

Switching between development and production mode

Switching between development and production is as simple as changing a line in the buildout.cfg file.

Other available profiles are commented out.

[buildout]
extends =
    profiles/dev.cfg
#   profiles/prod.cfg
#   profiles/prodtest.cfg

Package management

The most common package management tasks include adding packages, removing packages, pinning the versions of packages. The buildout is configured to read the setup.py file. The big deal here is the settings associated with install_requires. Just look for that in the setup.py file and add the packages that are dependencies. (This is a standard python packaging convention learn more about install_requires).

Here's what it looks like in my buildout.

Additional Background

This is not original, while I've tweaked and hopefully improved things, I lifted the idea of a "profiles" folder from Redturtle https://github.com/RedTurtle/deployments.buildout.plone.

I've seen similar approaches implemented by others, for example Jarn https://github.com/Jarn/buildout calls their profiles folder "cfgs", Starzel https://github.com/starzel/buildout calls the profiles folder "linkto" and in the Plone Intranet project they place their .cfg files in the very "*nixily" named "buildout.d" folder. I like the name "cfgs" it is most descriptive so I may adopt this naming approach in the future.