Removing rogue members from your Plone site

I recently had one of my Plone sites get hit by a "join form attack". Basically a spam bot which started adding new members to my site (we don't use a captcha at the moment). I ended up with far more members in the site than members of the organization. I started deleting them manually using the Zope Management Interface > acl_users > source_users, but that became tedious quickly. So I wrote a command-line script to do the job for me.

In my case the spambot wasn't super smart, all the usernames created by the bot started with capital letters. All my real users had usernames with common letters. This made it easy to filter out the bad guys.

Here's my script in a nutshell:
.
.

I used the special --object-path option to indicate the location of my Plone site, relative to the root of the Zope application server.

My final command looked something like this:

bin/instance_debug --object-path='pathto/sitein/zodb/plone' run member_cleanup.py > members_cleanup_report.txt

The resulting output went to a members_cleanup_report.txt.

Many thanks to the plone.api and plone.docs teams, being able to make use of plone.api.user made it 20 times easier to write the script.

A note about transactions

Before my script would run successfully I had to add a transaction.commit() line, it seems commandline scripts require this.

Parting thoughts

There's a lot more that can be added to the script to make it smarter. For example for certain kinds of sites you could filter based on whether the user has created any content or perhaps it might be based on log in patterns, if they have never logged in or only logged in once.

I'm weighing the pros and cons of having a captcha. At the moment members can't do much except change their portrait pictures and profiles, but I know that some spammers use the portrait for hosting "bad" images, so captchas may have to be introduced.

References

The resources I used included the following references:

http://docs.plone.org/external/plone.api/docs/api/user.html
http://docs.plone.org/develop/plone/misc/commandline.html#scripting-context
https://pypi.python.org/pypi/plone.recipe.zope2instance

Bulk adding of users to a Plone site (TTW)

This is a quick way to add new users to a Plone site, very useful in a pinch but probably not very robust (so I make no guarantees). It might possibly serve as the pattern for a "user importer", assuming someone hasn't invented one already.

Everything is done through the web (TTW) using the Zope Management Interface (ZMI), so you'll need to be an administrator.


Add the following script to your Plone instance and give the script a name, I called mine 'addusers'.

out = [] acl_users = context.acl_users for item in container.temp:     person = item.split(',') username = person[0]     password = person[1]     try:          acl_users.userFolderAddUser(username,password,['trainee'],'')     except ValueError, msg:         out.append("Skipped %s, reason: %s" % (username, msg))     else:         out.append("Success with %s" % username) return out

Note: the use of the role 'trainee' in my example, the role MUST exist for this to work. Also for the Python purists, that "hanging" return statement is perfectly fine, TTW Python scripts run inside Zope and return the output to your browser. If it absolutely bothers you, you can leave it out.

Under the ZMI > properties' add a 'lines' field called 'temp' it should include the usernames and passwords of the newusers (no spaces after the commas)

The 'temp' lines field should look like this:

Maurice,dttrrrsffd
Natalie,55542yylrw
Herman,d5rrrrr6tfdsa
Ryan,r4rffd243faz
Patrick,e443rereew
Sean,erserwrew
Michael,4343243qw

Running the 'addusers' script will now create the users with the passwords that you defined.

Update: July 1, 2013

There are a couple other tools mentioned in the comments of this post which you may also want to look at.
https://pypi.python.org/pypi/atreal.usersinout/

http://plone.org/products/collective.mass_subscriptions - can send out initial passwords in bulk

https://pypi.python.org/pypi/collective.loremipsum - generates fake content and fake users

Dexterity Development Quickstart using zopeskel.dexterity

Update:  You will probably make more progress using this: https://github.com/pigeonflight/stack-python-plone I've blogged about Plone on Dotcloud here.

These are my notes for getting started quickly with dexterity and zopeskel.dexterity.

Rule Number One: Use Unix
This is about getting started quickly. At this time, Windows will slow you down, do this on a server or something, this issue may change in the future at which point I will be able to recommend otherwise, but for now, for the sake of your sanity just use OS X or Linux for development.

Rule Number Two: Use the Unified Installer
The Unified Installer (usually available at http://plone.org/download) provides all the tools you need to get started, everything here assumes that you have successfully installed Plone via the Unified Installer on your platform.

Rule Number Three: 
Install your app and run buildout BEFORE attempting to use localcommands. This is important because additional "goodness" is added to your new package when it is properly installed to your instance.

Step 1 - Install via the UnifiedInstaller then edit the base.cfg and add zopeskel.dexterity and ZopeSkel <= 2.99.

eggs =
    PasteScript 

    ZopeSkel <= 2.99

             zopeskel.dexterity

Then re-run buildout 

    bin/buildout

 Step 2 - Create your first product

cd src
../bin/zopeskel dexterity my.app

WARNING: localcommands like "addcontent" WILL NOT WORK until you install the app and run buildout!

Step 3 - Install your app by adding it to the buildout.cfg and re-run buildout

Add the following lines to your buildout.cfg

eggs =  

        my.app 

develop =

                 src/my.app

or if you prefer to use mr.developer do something like this:

extensions = mr.developer
auto-checkout = my.app
....
[sources]
my.app = fs  my.app

The re-run buildout

bin/buildout

Step 4 - Add your first contenttype
Note that all localcommands are run from within your new app

cd src/my.app
../../bin/paster addcontent dexterity_content 

You should now be able to follow the instructions at: http://collective-docs.readthedocs.org/en/latest/content/dexterity.html

If you want to know what localcommands are available try running

../../bin/paster addcontent -l

Activate your new Add-on
To see if everything is working launch your instance and visit Site Setup > Add-ons, you should now be able to activate your new add-on.

In the screenshot below, I have already created a content type for my add-on, so after activation I can use the green content bar in my site to add a new example type.

Configuring Munin.zope - message to future me

When working with munin.zope your buildout should look like this:

parts =
...
munin

eggs =
...
munin.zope

[munin]
recipe = zc.recipe.egg
eggs = munin.zope
scripts = munin=munin
arguments = http_address='${ports:client1}',ip_address='${hosts:client1}', user='${users:credentials}'

The arguments line was not very clear to me. Here's how it works.

'http_address' is really the port number (e.g. 8080) that your zope runs on.

'ip_address' can be an ip address or domain name (e.g. localhost, or 1.2.33.44)

'user' is really the user credentials in the form 'username:password'.

So future me, hope that helps to save you a few days of knocking your head on the wall.

Python Imaging Library (PIL) with Plone 4 on OS X

The Python Imaging Library (PIL) was not working for me with the Python 2.6 packages that I installed on OS X Leopard.

I resisted for a while, but in the end the solution to my PIL problems on OS X was to compile Python 2.6.

I'm assuming that the following are already installed

• XCode should be installed

• Subversion (I use fink to get and install subversion)
  

I downloaded Python 2.6 from the python.org website

http://python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2

Then I unpacked and compiled:

tar xvfj Python-2.6.4.tar.bz2
cd Python-2.6.4
./configure && make
sudo make install

Using this version of python (which gets installed to /usr/local/bin) I was able to bootstrap the Plone 4 and compile PIL:

svn co http://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.0 plone4
cd plone4
PYTHON=/usr/local/bin/python
$PYTHON bootstrap.py
bin/buildout -c pil.cfg

Supervisord and Memmon

Three problems I've had with Plone.

1. How to start and stop Plone instances
2. How to deal with Plone instances that run away with memory
3. How to restart Plone processes that have died

Number 3 was more of a problem a few years ago with older versions of Zope, more recently Zope/Plone seems to run "forever". I used to use DJB daemontools to solve these problems and I still have servers that are "supervised" by daemontools. My problem is that, although daemontools works quite well working with it feels less than organic (read: doesn't play nicely with package managers rpm, apt or easy_install).

The solution
Well I believe that I've found the solution, and it goes by the name of "Supervisord" and the fun slogan "hang on to your processes". As a bonus, it is python based, installs nicely via easy_install and newer versions of ZopeSkel's plone_hosting template now ship with a supervisord configuration.

A recipe
In short, with a few deft strokes of the keyboard, I can have my Plone and supervise it. My recipe goes something like this:

1. Get the latest ZopeSkel

easy_install ZopeSkel

2. Install Plone hosting (I'll call my directory "usain").

paster create -t plone_hosting

3. Add lines similar to this to my crontab (by running crontab -e)

@reboot /home/plone/usain/bin/supervisord -c /home/plone/usain/etc/supervisord.conf

4. For good measure add a line like this to my supervisord.conf file:

   [eventlistener:memmon]
 command=%(here)s/../bin/memmon -a 200MB -m bob@example.com
 events=TICK_60

If any supervised processes run over 200MB of resident memory usage for more than 60 seconds, supervisor will restart them.

Me Grok? You Ruby?

So I learnt the basics of Ruby and Ruby on Rails (RoR) this week. I built a simple application following a tutorial, so I don't know if that counts.

Then, yesterday, I discovered Grok, I'm pretty good at discovering things months after they have existed, being launched in October 2006, Grok is already celebrating its first birthday.

Grok represents Zope 3 for humans. A framework that simplifies the way that a developer works with Zope 3. The highlights for me:

• Less configuration, more time to code
• It already has the KSS ajax framework, something I have been playing with in the Plone world

So I've been derailed for awhile from my RoR pursuits to look into Grok. If it does what I need I may stay away from the Ruby crowd for awhile longer.