Characteristics of a sustainable buildout

July 27, 2017 - Updated the section called "Additional Background" to include reference to the Plone Intranet project. Also swapped out the blockquoted code snippet for a github gist instead.

Buildout is a software build system used to manage development and deployment setups, especially in Python. When done well, Buildout makes everything very repeatable. Widely used in Plone projects but not exclusive to such projects, I've used buildout with Pyramid, Tryton and even for Sphinx based documentation projects. I've also seen buildouts on github for Django, Flask and Odoo based projects. 

What I expect of a proper build system

• Easy to switch between development and production
• Easy management of packages and package versions
• Easy to share with other developers
• A workflow as close to what developers expect

Here's a breakdown of how we do our buildouts to ensure that these goals are met.

The anatomy of a sustainable buildout

There are a few things that I add to my buildout to help to keep things sane. The profiles and templates folder and a requirements.txt file. I discuss their usage below.

Profiles folder

The profiles folder holds the configuration files used by the buildout. Generally the ones that we link to directly are dev.cfg and prod.cfg. These then link to the supporting configuration profiles.

Templates folder

Templates are stored in the templates folder, these templates are used to dynamically generate configuration settings. Some of them include variables which are configured in the .cfg files found in the profiles folder.

requirements.txt to define build tools

I recently added the requirements.txt file to make the workflow closer to a more typical Python development workflow, for example most Flask developers expect such a file. Since Buildout is your build tool, we define it as a dependency in the requirements.txt so that a user can easily install it.

Getting started is a matter of running the well known "pip install -r requirements.txt" command. This installs the buildout command.

buildout.cfg is not stored by git

With this approach the buildout.cfg file is not stored in the repository, so it is added to the .gitignore file. The process of kick starting things requires that a user copies the buildout template from the templates folder into the root of their buildout.

Bootstrapping

Here's a quick example with a real project, the Python Jamaica website. I've added comments to describe each step of the process. I only assume that you are familiar with virtualenv and pip.

...
...

One gotcha to look out for, after bootstrapping a local command bin/buidout is used to run the actual build (not to be confused with venv/bin/buildout).

Thanks to Maik Derstappen for pointing me towards "buildout bootstrap" which replaces the older approach of using a special bootstrap.py file.

Day to day usage

Once the buildout is installed here are some day to day tasks that a developer might find themselves doing.

Launching an instance

During development a standalone instance can be launched with the command:

bin/instance fg

This launches the reference site on port 8080.

Switching between development and production mode

Switching between development and production is as simple as changing a line in the buildout.cfg file.

Other available profiles are commented out.

[buildout]
extends =
    profiles/dev.cfg
#   profiles/prod.cfg
#   profiles/prodtest.cfg

Package management

The most common package management tasks include adding packages, removing packages, pinning the versions of packages. The buildout is configured to read the setup.py file. The big deal here is the settings associated with install_requires. Just look for that in the setup.py file and add the packages that are dependencies. (This is a standard python packaging convention learn more about install_requires).

Here's what it looks like in my buildout.

Additional Background

This is not original, while I've tweaked and hopefully improved things, I lifted the idea of a "profiles" folder from Redturtle https://github.com/RedTurtle/deployments.buildout.plone.

I've seen similar approaches implemented by others, for example Jarn https://github.com/Jarn/buildout calls their profiles folder "cfgs", Starzel https://github.com/starzel/buildout calls the profiles folder "linkto" and in the Plone Intranet project they place their .cfg files in the very "*nixily" named "buildout.d" folder. I like the name "cfgs" it is most descriptive so I may adopt this naming approach in the future.

A working Plone 2.5.5 buildout

On a recent project I needed to get Plone 2.5.5 up and running. This is a note to self so I can get it done more quickly next time. It also reminds me how far we've come from the Plone 2.5.5 and early buildout days.

As a starting point I needed a "known good Plone 2.5.5 buildout" I borrowed most of ideas from this Venezuelan repository: http://plataforma.cenditel.gob.ve/browser/proyectosInstitucionales/eGov/ppm/buildout/plone/2.5/buildout.cfg?rev=137. After reading around I mixed and matched and came up with this template: https://gist.github.com/pigeonflight/5268138 (I consider it good enough for a developer to get started)

Use collective buildout.python for building a suitable  Python 2.4

I did the following as the root user (Takes about 40 minutes to compile on an Amazon EC2 micro instance)

cd /usr/local git clone https://github.com/collective/buildout.python.git python cd python python bootstrap.py

Edit 'buildout.cfg' so that parts and eggs only reference python 2.4

extends =    src/base.cfg    src/readline.cfg    src/libjpeg.cfg    src/python24.cfg    src/links.cfg parts =    ${buildout:base-parts}    ${buildout:readline-parts}    ${buildout:libjpeg-parts}    ${buildout:python24-parts}    ${buildout:links-parts}

Then run buildout

bin/buildout  bin/install-links

As the Plone user download the custom buildout.cfg run the following commands

Get the custom buildout.
          wget https://gist.github.com/pigeonflight/5268138/raw/27f2f61e4a51d6822816a9c451d6ae35583184bf/buildout.cfg

/usr/local/python/bin/virtualenv-2.4 .bin/easy_install zc.buildout==1.4.4          # this line below does a little dance (to take advantage of buildout init) mv buildout.cfgzz && bin/buildout init && mv buildout.cfgzz buildout.cfg bin/buildout

The result should be a working Plone 2.5.5.

I plan to customize this for production (yes, they are still using Plone 2.5.5).

Plone to the cloud in 3 easy steps

update: July 31, 2015 My recommended approach is to install Plone on Cloud9 IDE.

update: July 24, 2013 This no longer works as Dotcloud has retired their free sandboxes. I've documented a way to run Plone in the Cloud with Nitrous.io.

Before you start

You will need:

• 5 to 10 minutes of spare time.

• a  reliable internet connection 
• a Cloud9 IDE account 
• a Dotcloud account 

Have your login information for both Cloud9 IDE and Dotcloud close at hand.

Disclaimer:

The underlying tools for managing Plone on Dotcloud from Cloud9 IDE are in an alpha stage at best (meaning, I've been working on this for only a few weeks)

Setup and Installation

Step 1 - Clone the URL at Cloud9 IDE

Log in to Cloud9 IDE and from the dashboard select "Create new workspace" > "Clone from URL".

In the resulting dialog box, set the "source URL" to https://github.com/pigeonflight/stack-python-plone and click "Create":

Step 2 - Preparation

Launch your newly created Cloud9 IDE instance from the dashboard and install the dependencies by issuing the following commands from a dotcloud terminal (you can launch the terminal with the shortcut key Alt+T)

easy_install pip
pip install -r requirements.txt

Also setup dotcloud (you'll be prompted for you dotcloud username and password)

dotcloud setup

Step 3 - Create a dotcloud instance

From a terminal create a new dotcloud instance and push. (Answer yes when asked about using the current directory for using

dotcloud create plonedemo
dotcloud push

This will generate a live Plone instance on dotcloud.

Accessing your Plone Site

Your Plone site is named based on your dotcloud account name and your dotcloud instance name. If you used the name 'plonedemo' and your dotcloud account name is 'scotty', then your Plone url will be: http://plonedemo-scotty.dotcloud.com.

Your credentials (admin password)

The following command will retrieve your admin password:

dotcloud run www cat code/adminPassword.txt

Make a note of your admin password.

Visit the appropriate url using your webbrowser and log in using the credentials from the output of the above command.

Have fun.

Installing Plone 4 on Solaris

Plone 4 is still under development, but I've heard enough to get me curious. This is how I got it working. I ended up installing it on OpenSolaris, should work just as well on Linux. I had issues with MacPython on OS X.
update: I was able to get it to successfully install on OS X Leopard by "downgrading" to macpython 2.6.2. Will need to try the same on Linux.

Important notes:

• Use Python 2.6 less than Python 2.6.3 (so 2.6.2 and 2.6.1 should work) (thanks to esteele for the tip)
  
• Make sure all the supporting python 2.6 packages are installed
  On Solaris this includes SUNWPython26 and SUNWPython26-setuptools (you may possibly also need SUNWPython26-extra)

Get PIL installed
Following a tip from speedbreeze. As the root user run the following command:
easy_install-2.6 -i http://dist.serverzen.com/pypi/simple PILwoTK

Checkout the source, bootstrap and build
svn co http://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.0/ plone4
cd plone4
python2.6 bootstrap.py
bin/buildout

I was then able to launch Plone 4:
bin/instance fg

Spliish Plone buildout for shared plone hosting

I'm doing an early release of a little pet project.
The Spliish hosting platform will hopefully, help to simplify the setup and management of a zope server which hosts multiple Plone instances.

I've already used it in one production project, but it needs some more "love".

My first step is to place the code in the collective so that other Plonistas can tinker with it.

Get more info in the README file.
or visit the wiki.

Goals
The project has 4 specific goals all of them to support a site administrator

An administrator should be able to do the following with ease:

1. add new domains and subdomains (works reasonably)
2. add and "compile" custom themes and associate them with specific plone instances (todo)
3. add new add-on products (todo)
4. backup site data and configurations (todo)