By default results of functions that use @plone.memoize.ram.cache decorator are stored in RAMCache - zope utility for storing data. But in ram.py file from plone.memoize you can find MemcacheAdapter class. This adapter gives possibility to store this data in memcached. To turn it on you must register ICacheCooser adapter.
memcached.py:
# -*- coding: utf-8 -*-
from threading import local
from pylibmc import Client
from plone.memoize.ram import MemcacheAdapter
class MemcacheChooserUtility(object):
"""
See ICacheChooser.
"""
_v_thread_local = local()
def getClient(self):
"""
Return thread local connection to memcached.
"""
connection = getattr(self._v_thread_local, 'connection', None)
if connection is None:
connection = Client(['127.0.0.1:11211'])
self._v_thread_local.connection = connection
return connection
def __call__(self, fun_name):
"""
Create new adapter for plone.memoize.ram.
"""
return MemcacheAdapter(client=self.getClient(), globalkey=fun_name)
overrides.zcml:
<configure xmlns="http://namespaces.zope.org/zope">
<utility
provides="plone.memoize.interfaces.ICacheChooser"
factory=".memcached.MemcacheChooserUtility"
/>
</configure>
Add pylibmc to buildout. Use 0.7.4 - last version compatible with python 2.4. You can also use python-memcached, but I prefer pylibmc because this library reopen connection after memcached restart automatically.
Remember to include overrides.zcml in instance configuration.
Advantages:
Disadvantages:
Deploying Plone site to static files has the following advantages:
- Security: You can run Plone internally, and possibly on a less robust/costly server, since it will only be used by internal content managers. You could then export your published site content in its static form (.html, .css, .jpg, .swf, .pdf files, etc.) and deploy that static site easily to either a very affordable, simple shared HTML site hosting plan, or a massively scalable, load-balanced web server environment for larger organizations.
- Scalability: Since the published site is static HTML, CSS, images, etc., with no database backend to have to connect to or dynamic scripting engine needed to transform the content as a page is requested, the site is served up much faster and the deployment environment is more stable due to fewer moving parts (no database, no scripting engine, etc.) Demands on server CPU and RAM can be greatly reduced.
- Flexible Hosting Options: You can run Plone on your laptop, export your static site as simple HTML, CSS, JavaScript, and image files, and deploy to a $5/month plan shared hosting environment. Or, you can put your entire published site onto a thumb drive to carry around with you. Your entire site could be served from memory (cache the static HTML and other files), since no database is needed for the published site to run. There's no need to buy a Plone-centric hosting plan in this scenario.
- It's Achievable!: There are a few sites out there using the CMFDeployment add-on tool for Zope/Plone. This is the low-hanging fruit, but we shouldn't paint ourselves into a corner if there is a better approach. Entransit may be another option, but to-date, this tool allows for exporting to a relational DB or possibly XML, but not HTML at the moment. [1]
In this article I will show how to install, configure and use stxnext.staticdeployment.
In my previous blog entry I described how to configure frontend and backend domains. In site with static deployment there are three domains:
- backend - for editors to edit content (default Plone skin)
- preview - for editors to immediately see changes is content (custom skin)
- frontend - for visitors (custom skin)
Frontend domain is served by apache mainly from static files, but some of requests (e.g. submits of forms) are forwarded to Plone.
Edit buildout.cfg and append stxnext.staticdeployment to eggs and zcml parameters in instance section:
[instance] eggs = ... stxnext.staticdeployment zcml = ... stxnext.staticdeployment
Instance must be rebuild and restarted:
./bin/buildout ./bin/instance stop ./bin/instance start
This product must be also installed inside Plone site. Do do it, go to Site Setup -> Add/Remove Products, select checkbox near stxnext.staticdeployment and click Install button.
Every website has own configuration (different set of eggs, skin, products etc.) - this I meas as project. But website can have few instances (development, test and production instances). Because of this, configuration is split for two parts:
- instance parameters:
- parameters connected to instance - e.g.: domain
- configured in Control Panel - can be edited throw the web (Site Setup -> Static deployment -> Settings tab)
- form has description and validation - can be used by less experienced users
- website parameters:
- stored in INI file
- created by developer of website
- can be used by many sites
- default configuration (included in egg)
- default configuration can be overriden by file ${buildout:directory}/etc/staticdeployment.ini
Frontend domain must be configured in some special way. It will serve static files from disk, but if requested data do not exists in configured directory it will forward request to Plone. Similar all POST request should be also passed to Plone:
<VirtualHost *:80>
ServerName example.org
ServerAlias static.example.org
DocumentRoot /var/www/example.org/html
CustomLog /var/www/example.org/logs/access.log common
ErrorLog /var/www/example.org/logs/error.log
RewriteEngine on
RequestHeader append plone_skin "Notre Dame"
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME}index.html !-f
RewriteRule ^/(.*) "http://localhost:8080/VirtualHostBase/http/%{SERVER_NAME}:80/site/++skin++Notre Dame/VirtualHostRoot/$1" [L,P]
## make rewrite on every POST requests
RewriteCond %{REQUEST_METHOD} ^POST$
RewriteRule ^/(.*) "http://localhost:8080/VirtualHostBase/http/%{SERVER_NAME}:80/site/++skin++Notre Dame/VirtualHostRoot/$1" [L,P]
</VirtualHost>
This is only basic configuration. It's very possible that list of circumstances (when content should be served from Plone not from files) will be bigger.
When website is ready to deployment go to Site Setup -> Static deployment -> Deployment tab. Select Deploy static version of website checkbox and press Save button. Deployment will work for few seconds or minutes (it depends on size of website and server performance).
stxnext.staticdeployment v 0.3.1 is not fully functional (it will not dump whole website to static files). Some of pages can work not properly. Static files can't provide all functionalities that are in Plone (e.g. interface translation). But we use this module (with success) in few our projects, so we decided to release it under Open Source license. It can be base for bigger, better and faster solution.
| [1] | Plone Static Publishing project at CoActivate |
Slides from RuPy talk "Composite Website with Plone, Django, Xapian and SSI" by Łukasz Langa and me.
Compiled Website with Plone, Django, Xapian and SSI (PDF - 6 MB)
Default plone skin is tidy and user-friendly but is not beautiful. Because of this most of users, want to change look and feel of site build with Plone.
There are many additional skins that can be easily installed.
Most of available new skins for Plone looks better, but also skins are not so user-friendly for editors like default skin. Best situation could be if editor could use default skin and visitor could see site in other skin. This is possible - and even not so complicated to configure.
As example I will use Notre Dame - skin is used on this website.
At first egg with skin must be added to instance. Edit buildout.cfg and append skin to eggs and zcml parameters in instance section:
[instance]
eggs =
...
plonetheme.notredame
zcml =
...
plonetheme.notredame
Instance must be rebuild and restarted:
./bin/buildout ./bin/instance stop ./bin/instance start
Now new skin must be installed inside PloneSite. Do do it, go to Site Setup -> Add/Remove Products, select checkbox near Notre Dame and click Install button.
After reload you should see website in new skin.
At first site must me reconfigured in ZMI.
Go to properties tab in portal_skin (http://example.org:8080/site/portal_skins/manage_propertiesForm) and set REQUEST variable name to HTTP_PLONE_SKIN
In portal_css you must recognize all entries that was added by new skin. In most cases it will be few last entries. New style added by Notre Dame are preceded by ++resource++plonetheme.notredame - so in this case it's very easy.
In these entries you must add new condition:
python: portal.REQUEST.get('HTTP_PLONE_SKIN', *) == 'Notre Dame'
Last thing to configure is apache. Create new file with vhost configuration:
<VirtualHost *:80>
ServerName admin.example.org
ServerAdmin admin@example.org
RewriteEngine on
RequestHeader append plone_skin "Plone Default"
RewriteRule ^/(.*) http://localhost:8080/VirtualHostBase/http/%{SERVER_NAME}:80/site/VirtualHostRoot/$1 [L,P]
</VirtualHost>
<VirtualHost *:80>
ServerName example.org
ServerAdmin admin@example.org
RewriteEngine on
RequestHeader append plone_skin "Notre Dame"
RewriteRule ^/(.*) "http://localhost:8080/VirtualHostBase/http/%{SERVER_NAME}:80/site/++skin++Notre Dame/VirtualHostRoot/$1" [L,P]
</VirtualHost>
After apache reload you can use two urls:
- backend domain - http://admin.example.org (Default Plone skin)
- frontend domain - http://example.org (Notre Dame skin)
Advantages:
- editor can use the same UI on every site
- visitor can see nicer website
- it's easier to create new skin only for visitors (designer, html and css specialist do not need to bother about screens to edit content)
- caching (e.g. varnish) can be configured only for frontend domain
Site based on default Plone configuration can be extended to support video content.
There are few solutions to present videos in Plone:
- p4a.plonevideoembed
- Products.FlashVideo
- collective.flowplayer
I prefer collective.flowplayer, because it's light, user-friendly and up-to-date. This product integrates flowplayer ver. 3 (on GPL license) - it can play AVI, MP3 and MP4 media files.
Few days ago I made additional module - stxnext.transform.avi2flv. It converts videos in AVI format to FLV. This conversion is made automatically, when editor uploads file recognized as AVI.
stxnext.transform.avi2flv uses FFmpeg. It must be installed with mp3 (lame) nad flv support.
To install it on Ubuntu 9.04 you have to run:
sudo apt-get install ffmpeg libavcodec-unstripped-52
On CentOS 5.3 command is even simpler:
sudo yum install ffmpeg
Plone's 3.3 standard buildout must be extended only with few lines:
[buildout]
...
eggs =
...
stxnext.transform.avi2flv
collective.flowplayer
[zope2]
...
additional-fake-eggs =
collective.testcaselayer
[instance]
...
zcml =
...
stxnext.transform.avi2flv
collective.flowplayer
After rebuild and restart of instance this two products must be also activated. Go to Site Setup -> Add/Remove Products and install Avi2flv Transform Install.
Options for FFmpeg that controls quality of output video can be configured via ZMI. Go to http:/example.org/site/portal_transforms/avi_to_flv/manage_main
With default configuration it will execute command similar to:
ffmpeg -i "input.avi" -y -b 1024k -r 25 -acodec libmp3lame -ar 44100 'output.flv'
Description of these options and more control parameters you can find in FFmpeg manual.
Today I replaced default Plone discussion mechanism with DISQUS comments system.
It's only few easy steps to integrate DISQUS with your Plone site:
create account on DISQUS
add you website in DISQUS admin panel (remember configured website short name)
install collective.disqus: - add to your buildout:
eggs = collective.disqus ... zcml = collective.disqus
go to Site Setup -> DISQUS comment system form and configure Website short name
enable Allow discussion option in content objects
Current version of collective.disqus doesn't migrate comments. So all comments that was created before installation will be hidden (not removed).
Default Plone discussion mechanism doesn't have nice panel to administer comments. It's hard to find new comments. It's not possible to block posts with links or some other unwelcome contents.
There is few products that tries to fill up this gaps:
In clean Plone commenting is allowed only for authenticated site members. This can be of course also widen:
Installation and proper configuration of so many plugins will take some time and functionality will not be full.
But on the web there are much more specialized tools for commenting:
This comments system can be easyly integrated with sites - user just need to create account and add some special code into his website.
I use DISQUS because:
Example of DISQUS comment system integrated with Plone you can see at the bottom of this page.
Few weeks ago we had some problems with development server. After few minutes of work server totally hang - every try to connect via ssh was finished with timeout. We had to make hard reset but after few minutes it hang again. Next restart, but this time we turn on debug mode (single user mode in linux) and start services one by one. After this investigations we was surprised that deliverance after few minutes of work consume all available memory (4 GB of RAM and 6 GB of swap).
Further investigation show that CSS selector with typo was so problematic.
In deliverance v0.3 (trunk version) rules has pointers to HTML elements. This pointers consists with two parts - type of selection (e.g.: children or attributes) and selector (CSS selector or XPath) separated with colon:
<replace content="children:#content-wrapper" theme="children:#content" />
Let look at sample:
attributes(href):/html/body/a
Deliverance will use XPath to extract tag a from body of document. But if developer forgot colon (:):
attributes(href)/html/body/a
this pointer will be parsed by lxml as CSS selector. Of course this selector is not correct, but lxml (versions 2.0 - 2.2.2) cannot manage it and parse it in infinity loop and consume more and more memory. Issue is submitted on lxml bug tracker
Deliverance use not more than 512 MB of memory so environment must not allow to use more than this.
Run deliverance-proxy as separated user. Make system limits for this user. More info in manual pages on every linux:
man limits.conf
If deliverance will eat to much memory python will rise MemoryError.
Configure limits only for processes started form commend line. Create simple bash script:
#!/bin/bash ulimit -v 507200 ulimit -H -v 507200 bin/deliverance-proxy $*
Also with this script deliverance will stop with MemoryError.
Supervisor's plugin superlance has special feature that will restart process it if use to much memory. Configuration is simple with buildout:
[buildout]
extensions = gp.vcsdevelop
vcs-extend-develop = svn+http://codespeak.net/svn/z3/deliverance/trunk/#egg=deliverance
develop-dir = src
parts =
deliverance
supervisor
eggs =
deliverance
[deliverance]
recipe = repoze.recipe.egg:scripts
scripts = deliverance-proxy
eggs = ${buildout:eggs}
[supervisor]
recipe = collective.recipe.supervisor
plugins = superlance
port = 8000
user = admin
password = :)
serverurl = http://127.0.0.1:8000
programs =
10 deliverance ${buildout:bin-directory}/deliverance-proxy [${buildout:directory}/src/theme/rules.xml] true
eventlisteners =
Memmon TICK_60 ${buildout:bin-directory}/memmon [-p deliverance=500MB]
This will build deliverance-proxy controlled with supervisor and monitored for memory leaks.
I made this article because most of tutorials available on web use some strange techniques, like hacks, rolling out or manual paths configuration. Not bed is official tutorial but there is to many manual steps (like running mkzope2instance) and it doesn't show how to use deliverance. My development buildout is quite similar to solution described in six feet up blog post.
My solution is designed to be as simple as possible:
This instance will help during development of deliverance theme. Programmer or designer must edit ruleset - definition of transforms that deliverance should perform on theme and content taken form Plone.
Download and extract all files from plone_deliverance_wsgi.tar.gz archive:
cd /var/lib/zope wget http://lichota.pl/blog/uploads/plone_deliverance_wsgi.tar.gz tar -xvzf plone_deliverance_wsgi.tar.gz cd plone_deliverance_wsgi
Edit settings.cfg and configure basic settings like: domain name (be default deliverance.example.com, port number (default 8080) or admin password (admin).
If chosen domain name is not configured in any DNS, you can add it to your /etc/hosts file, and point in to 127.0.0.1.
Run buildout:
python2.4 bootstrap.py ./bin/buildout -vc buildout-devel.cfg
Configure and restart apache:
sudo cp ./etc/deliverance.example.com.conf /etc/apache2/sites-enabled sudo /etc/init.d/apache2 reload
Now you can start Paster:
./bin/ctl fg
ZMI will be available at address http://127.0.0.1:8080/manage.
When everything is ready you can you three url's:
Using first address you can omit deliverance transforms, so you will see clean plone interface. With this address you can add, edit and manage content of website.
Next two addresses use deliverance engine. Theme and transform rules are located in ./src/theme directory. All files in this directory are served by egg:Paster#static middleware. So for example main theme file is available at address http://deliverance.example.com/theme/theme.html
In archive with buildouts I included skin (released on Creative Common license) with very simple ruleset. This ruleset (./src/theme/rules.xml) takes some part of pages generated by Plone and insert it in theme. More informations about writing such ruleset you can find at deliverance documentation.
In development instance theme and ruleset is parsed on every request, so it's slower but you do not need to restart paster after every change.
Development instance has some useful Plone products - Products.DocFinderTab, stxnext.pdb. It use also egg:Paste#evalerror middleware that allows interactive debugging.
Theme layout is based on colorus template created by Reality Software and released by Flash Gallery under the Creative Commons Attribution 3.0 license.
Ruleset defines very simple transformations - copy some tags from head, removes unnecessary divs (e.g: <div class="portletWrapper">...</div>), copy content (title, top menu, portlets, page content and footer).
Rules uses CSS selectors (only in v0.3) which are much more user friendly than XPath syntax.
When website is ready we need move it to production server. It means that we need to create production instance. In this configuration Apache with mod_wsgi creates WSGI stack, so separated (backend) HTTP server is not needed.
Information how to install Apache2 with mod_wsgi you can find in mod_wsgi Wiki. Remember that mod_wsgi must use python 2.4.x. If default python interpreter at server is newer than 2.4 (2.5 or 2.6) you must compile mod_wsgi form source and set python version before executing make:
cd /tmp wget http://modwsgi.googlecode.com/files/mod_wsgi-version.tar.gz cd mod_wsgi-version ./configure --with-python=/usr/bin/python2.4 make sudo make install
Now lets create instance. Download and extract all files from plone_deliverance_wsgi.tar.gz archive:
cd /var/lib/zope wget http://lichota.pl/blog/uploads/plone_deliverance_wsgi.tar.gz tar -xvzf plone_deliverance_wsgi.tar.gz cd plone_deliverance_wsgi
Edit settings.cfg and change domain name, admin password and number of processes. In my opinion (based on some load tests) best performance of website is when number of processes is equal to number of CPU's (or CPU cores if such exists). Every process can also use few threads. Increasing threads number increase performance in small extent, so I suggest 2 threads.
Run buildout:
python2.4 bootstrap.py ./bin/buildout -vc buildout-production.cfg
You can also copy ZODB database file (./var/filestorage/Data.fs) from development instance to production server. Now you must copy created on development instance theme an place it in ./src/theme.
Because mod_wsgi will create few separated applications (in few processes and threads) database (ZODB) must be accessible simultaneously. ZEO server (build by buildout recipe) will take care of it. We just need to start it:
./bin/zeoserver start
Configure and restart apache:
sudo cp ./etc/deliverance.example.com.conf /etc/apache2/sites-enabled sudo /etc/init.d/apache2 reload
Zope will be not running on separated port. So to access ZMI there must be special address - http://zmi.deliverance.example.com
In production configuration deliverance rulset and theme is parsed only once - at application start. So when you edit something in this files you must restart apache.
One of my hobby is cycling. There are few website that collects informations about trips and than can show statistics. My favorite is BikeBrother.
Some of my rides I record with GPS device (RBT-2100) connected via bluetooth to my phone (SE P1i). Tracks are stored in GPX files. But this data aren't perfect. For example - elevation (position in meters above sea level) sometimes has very strange value (probably because low quality of GPS device or some distortion of signal).
To fix such low quality data in GPX file I made few command-line tools:
Scripts are of course written in python. More information can be found on PyPI.
Tools can be easily used in pipeline:
cat input.gpx | ./bin/gpx-fix-elevation | ./bin/gpx-cleanup | ./bin/gpx-compress > output.gpx
Correct value of elevation is calculated based on data from Shuttle Radar Topography Mission (SRTM). Very useful was eleserver by grahamjones139 - SRTM calculation code is based on this project.
gpx-cleanup and gpx-compress commands should decrease GPX file size - shorter upload time.
In plans I have next tools:
but I must find some time for small research ;)
Every good blog should have ability to leave a comment on blog entry. Plone since version 3.0 has built in discussion support. To enable it, editor must check Allow comments checkbox on edit form.
By default only logged in users can leave comments. This approach is good for administrators (less comment spam), but is quite annoying for viewers - because they must create account. But there is solution good for both sides - OpenID.
To add OpenID support to Plone-based website just go to: Site Setup / Add-on Products and install OpenID Authentication Support. Now users can log in with OpenID, but still such users can't see "Add Comment" button. To fix it I had to look into Plone code, because Google couldn't find any useful linka. Fix is very easy - authenticated user must have Reply to item permission.
Solution step by step:
On the web there are some documents that describes how to build Plone compatibile with WSGI, e.g.:
But this solutions uses mostly now very beautiful workarounds. So I made my own buildout. I will use it only for development. Deployment buildout probably will be completely different.Buildout is based on code be Mikko Ohtamaa published on Red Innovation Ltd. blog.
[buildout]
extends =
http://good-py.appspot.com/release/repoze.zope2/1.0
http://dist.plone.org/release/3.3rc3/versions.cfg
parts =
zope2
paster
instance
find-links =
http://dist.repoze.org/zope2/latest
http://dist.repoze.org/zope2/dev
http://dist.plone.org/release/3.3rc3
http://download.zope.org/ppix/
http://download.zope.org/distribution/
http://effbot.org/downloads
develop =
eggs =
Plone
stxnext.pdb
Products.DocFinderTab
elementtree
repoze.debug
versions = versions
[zope2]
recipe = plone.recipe.zope2install
url = ${versions:zope2-url}
fake-zope-eggs = true
additional-fake-eggs =
ZODB3
ZConfig
pytz
[paster]
recipe = repoze.recipe.egg
scripts = paster
eggs =
${buildout:eggs}
repoze.zope2
[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
effective-user = zope
http-address = 8080
debug-mode = on
verbose-security = on
eggs = ${buildout:eggs}
zcml =
stxnext.pdb
zope-conf-additional =
<cgi-environment>
HTTP_ACCEPT_CHARSET utf-8
</cgi-environment>
Paster to run needs development.ini file:
[DEFAULT]
debug = True
[app:zope2]
paste.app_factory = repoze.obob.publisher:make_obob
repoze.obob.get_root = repoze.zope2.z2bob:get_root
repoze.obob.initializer = repoze.zope2.z2bob:initialize
repoze.obob.helper_factory = repoze.zope2.z2bob:Zope2ObobHelper
zope.conf = %(here)s/parts/instance/etc/zope.conf
[filter:errorlog]
use = egg:repoze.errorlog#errorlog
path = /__error_log__
keep = 20
ignore =
paste.httpexceptions:HTTPUnauthorized
paste.httpexceptions:HTTPNotFound
paste.httpexceptions:HTTPFound
[pipeline:main]
pipeline =
egg:repoze.debug#pdbpm
egg:Paste#httpexceptions
egg:repoze.retry#retry
egg:repoze.tm#tm
egg:repoze.vhm#vhm_xheaders
errorlog
zope2
[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 8080
To start web server use command:
./bin/paster serve development.ini
Voilà :)