Overview

Please see ASF Content Management System for general information about the design, layout, and motivation for the CMS.

See CMS Adoption for a list of projects currently using the CMS.

The CMS wiki page contains additional useful information.

Markdown format

The mdtext files are converted to html with python markdown.

The markup syntax is described in http://daringfireball.net/projects/markdown/syntax

The following extensions are enabled for the CMS:

The extensions are part of the 'Markdown Extra' syntax documented in http://michelf.com/projects/php-markdown/extra/

Information flow

A user creates a mdtext file with the webclient in his/her own workspace, or in a working copy of the site's source. After a commit the following stages occur:

Staging

Staging is initiated by a commit to the svn repository.

  • the mdtext file is converted to html by the build and written to the staging site.
  • the converted html file is committed from the staging site to the staging repo.

Publishing

Publishing is initiated by clicking the [publish] link in the webclient, or by running http://s.apache.org/cms-cli from the command line.

  • the changes from the staging site are cheap-copied into the production URL by the POST to the publish link.
  • subsequently svnpubsub notifies the main webserver of the change and the webserver svn up's the site.

Setting up your content

Easiest way to get setup quickly is to use the boilerplate site setup. It has the right files in the right structure. Just export it and add it to svn.

svn export https://svn.apache.org/repos/infra/websites/cms/template site
svn add site
svn ci site -m "initial setup for the cms"

The location of your 'site' dir in svn is what Infra needs to get you setup in the CMS. Make sure to include this in your JIRA when requesting to use the CMS.

Once setup, you'll want to change the site/trunk/templates/skeleton.html file to use the look and feel you want for your project.

Organization of the webgui pages

For those of you who intend to go poking around in the CMS webgui, a little primer on the link organization is in order. There are essentially 2 main sets of links- those that come from the browse view and those that come from the edit view. The browse links are there to give you different perspectives on the content, whereas the edit links are there to let you alter it somehow. When viewing a directory the only difference between the browse and edit views are the sets of links presented, so it can sometimes be confusing which of those two views is operational. Just look for browse or edit in the title to keep them straight- eventually you will get used to recognizing the different link collections.

When you act on some content by POSTing form data, you will be given a brief list of links to select from. The links are ordered loosely in the order you are expected to click on them, and the UI isn't shy about expecting you to use your browser's back button to return to the CMS. One of the upsides of REST compliance is that the site is designed to facilitate state management in your web browser, not through server-side contraptions. Feel free to navigate back and forth to various pages you have visited and reuse them (with whatever embedded state they contain) at will.

Single exception regarding commits

There is a single exception here that you should be aware of: if you POST form data to commit something, and your server-side working copy has a stale version of that resource, the CMS will automatically update your working copy and retry the commit- if this happens your prior editing session will be out of sync with the content you actually committed as some external commit information may have been merged in the process.

The CMS will indicate this has happened by notifying you it had to merge external content on the post-commit page. Hence avoid going back to a prior editing session once you've committed and simply use the [Edit] link provided on the post-commit view to ensure you are working with the latest version of the resource. If you are really in the habit of going back anyway at this point, when you see the merge notification message do yourself and your project a favor and refresh the edit page before you start hacking on the content again.

FAQ

What version of Django's template library does the CMS use?

The CMS uses a fork of Dotiac::DTL version 0.8, which seems to implement this version of django templates.

How do I use the code highlighter?

Some questions have arisen regarding the syntax hilighter on code blocks. Basically the guessing algorithm isn't too great, so to give it help explicitly tell it which lexer to use from the list of short names here:

http://pygments.org/docs/lexers/

What you do is prefix your code/text block with an indented ":::shortname" or "#!shortname", where shortname is the name of the lexer to use. Often you want to use ":::text" just to tell the codehilighter to do nothing special to the indented block. The ":::shortname" variant will not supply line numbers whereas the "#!shortname" variant will.

How do I add a new file or directory to my site?

To add a resource you first edit a directory by clicking on the Edit link to a directory. Then you add the name of the new file or directory to the form input field at the top of the page; directories are signified by a trailing '/' in their name. Hit enter and a form will be provided to create the new file or directory.

How do I publish generated docs (eg. doxygen)?

First place all your generated docs into a local directory, let's call it foo/. Next create a compressed archive of foo/ by running

$ tar -czf foo.tar.gz foo

Then in the CMS navigate to the directory you'd like to add that directory to. Pull up the Edit screen and type "foo/" into the form field. Then hit enter. Fill in the file upload widget by pointing it at foo.tar.gz, click on "Quick Commit", add a log message, and hit Submit. Wait while the staging site builds the docs, and when that process is completed click Publish Site and Submit. Please note that the name of the local directory you archived must match the name of the directory to be added to the CMS.

The way to maintain this setup is to use the CMS to delete stale doc trees and replace them with fresh versions.

More sophisticated projects should be capable of scripting this into their release packaging process. The requests to the CMS would look something like this (in perl):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ua = LWP::UserAgent->new(requests_redirectable => ["GET"]);  
$ua->default_header(Accept => "application/json");
$ua->credentials("cms.apache.org:443", "ASF Committers", $user, $passwd);
$response = $ua->head("https://cms.apache.org/redirect?action=delete;uri=$urlencoded_uri_of_website_doctree");
$delete_uri = $response->header("Location")
    or die "Missing Location header";
$delete_uri .= "/" unless $delete_uri =~ m!/$!;
## be careful here, if you screwed up the uri above you could wind up deleting your entire site.
$delete_uri =~ m!/trunk/content/$! and die "Won't delete entire site!";
$response = $ua->post($delete_uri, [ submit => "Submit" ]);
die $response->status_line unless $response->is_success;
$commit_uri = $delete_uri;
$commit_uri =~ s!/delete/!/commit/!;
$response = $ua->post($commit_uri, [ submit => "Submit", message => "deleting stale doc tree" ]);
die $response->status_line unless $response->is_success;
$add_uri = $delete_uri;
$add_uri =~ s!/delete/!/add/!;
## put new doctree into foo.tar.gz
$response = $ua->post($add_uri, Content_Type => 'form-data', Content => [
   submit => "Submit",
   message => "new doctree",
   commit => 1,
   file => [ "foo.tar.gz" ],
]);

Then use the http://s.apache.org/cms-cli script to publish the site (once the staging build has completed).

On the other hand, if you find that after a while of doing this your CMS source tree seems bogged down in generated documentation trees, making dealing with the site unwieldy, there is a handy gadget here: simply create a content/extpaths.txt (or content/resources/extpaths.txt for maven builds) file which lists the paths (relative to your docroot) of the bases of the documentation trees, then after committing that file to your tree you may purge your source tree of those documentation trees and publish your site. The publishing process will keep track of those trees to ensure they remain accessible on your site. If you ever tire of publishing one of those trees simply remove the corresponding line from extpaths.txt, commit and republish. See http://www.apache.org/extpaths.txt for an example+description of the file format. Note that once added to the file, such urls are essentially frozen unless/until either the entry is removed or someone commits a change directly to its corresponding production tree in svn.

How do I run the CMS build scripts?

You can check out the scripts which can be used to build your site from https://svn.apache.org/repos/infra/websites/cms/build/. The perl script that generates the *.html site files from *.mdtext is build_site.pl. The build_site.pl script has some Python dependencies. The easiest easiest way to install these is to use the Python setuptools. First check your version of Python python --version and follow the instructions to install the appropriate set of tools from here http://pypi.python.org/pypi/setuptools.

After the python setuptools are installed other python dependencies can be installed as follows:

$ sudo easy_install Pygments
$ sudo easy_install ElementTree (Not completely sure that this is needed)
$ sudo easy_install Markdown

Depending on your Perl installation, you may need to install some CPAN modules first. A few users have had to install the following CPAN modules due to the use of the ASF::Value module:

  • install XML::Atom::Feed
  • install XML::RSS::Parser::Lite
  • install XML::Parser::Lite
  • install YAML
  • install SVN::Client

You may also need to do a force install depending on your setup.

Once everything is installed you'll need to run

$ cd /path/to/cms/build/tree
$ export MARKDOWN_SOCKET=`pwd`/markdown.socket PYTHONPATH=`pwd`
$ python markdownd.py

to get the markdown processing daemon up. Note: the local mdx_elementid.py markdown extension we use has now been updated to py-markdown 2.x, so earlier versions of the markdown module won't work any more.

How come my nested lists don't render properly?

The python implementation of markdown insists on 4-space indents per nesting level.

How do I verify compatibility if I write a client app for the CMS?

Assuming you're using json, you can verify API compatibility by checking https://cms.apache.org/compat?version=$VERSION where $VERSION is the API version of the CMS that your code was written against.

What is Quick Commit and why would I use it?

If you are making a simple edit to a single page, Quick Commit will save you the trouble, after you've submitted your edits, of pulling up the Commit screen, filling in the Log Message, and hitting Submit. It is a convenient feature that probably should be used for 80% of your work in the CMS.

Why doesn't the bookmarklet take me directly to the Edit screen for a page?

It takes you to a Browse screen for 2 reasons:

  1. to get people familiar with the features of the CMS
  2. because the bookmarklet sometimes doesn't take you to the resource you want.

An example of 2 is the following url: http://www.apache.org/licenses/LICENSE-2.0. Instead of taking you to the .mdtext representation it takes you to the .txt one, because httpd's mod_negotiation prefers the shorter representation when other factors are equal.

However if you still prefer to have the bookmarklet take you to the Edit screen, simply hack the bookmarklet to add an action=edit parameter to the query string. Like this for example: ASF CMS.

How come the publish link doesn't show diffs of newly added files?

svn diff doesn't show diffs of copied files, which is what the [Publish Site] link does with added files. There is an option available in subversion 1.7 that enables diffs of copied files.

How do I verify the markdown -> html conversion by the python implementation, before committing my changes?

All you need to do is Browse your changes after submitting your edits (in this case you want to avoid using Quick Commit).

I'm not a committer on $project, which uses the CMS. Can I still use the CMS?

If you are currently an Apache committer and want to submit a patch to the $project dev list using the CMS, you can do that right now. Simply use the bookmarklet and pull up an Edit session for the page in question and submit your changes. Then view the subsequent Diff link, which will provide a link to the Mail Diff feature. Fill in the form and your changes will be mailed to any apache.org address you choose (but only one address per mailout please).

Non-Apache committers can still checkout the site in svn and submit patches to the list. Otherwise they may access the CMS using username anonymous and an empty password and use the Mail Diff feature mentioned above to send patches to the list from the CMS.

The main idea is to {% include navigation %} in your site template, where navigation is a Django variable name. How you construct this variable and pass it along to the site template is up to you- you could break out separate paths in path.pm and add the per-path navigation to the argument list of each path. Alternately you could use view.pm to process the path and generate a navigation argument using regexp captures from path.pm.

How do I force the CMS to do a full site build?

Simply make a trivial change to a file in the templates/ or lib/ directory.

How do I use the CMS with an external build system?

Ensure your build system is supported on the cms.apache.org host and supply the proper @system_args to build_external.pl. Currently we are targetting Ant, Maven, Forrest, and hybrids of same, once the community figures out the proper @system_args for those build environments.

We've completely restructured our site and now there are a lot of leftover files on the live site. How do we clean things up?

Simply delete the content and cgi-bin subdirs of your staging tree within https://svn.apache.org/repos/infra/websites/staging and THEN trigger a full site build. Order counts- please do not delete those directories while a build is underway! Publishing your changes will clean things up on the live site.

Is there any way to turn off all these annoying buildbot commit messages appearing on our commit lists?

No- they are a core part of the CMS's internal bookkeeping and cannot be routed elsewhere. However they are not essential reading for projects whose site builds are well-understood, so feel free to route those messages to /dev/null.

Help! I made a massive commit to my CMS source tree and now I cannot use the CMS webgui, not even to publish the site!

The CMS's internal working copies are updated daily; if your working copy has not pulled in the changes you will need to Update your tree, which will take some time depending on the nature of the changes you need to pull in. (Note: you do not need to update your working copy in order to publish the site.)

The CMS's publish operation has been recoded to be based on svnmucc and is now much faster. If you think having massive trees is a problem for svn, investigate the --depth=immediates option to svn checkout. You can checkout the base dir of a very large tree quickly and simply and modify/commit any files directly contained therein. Let the CMS's build system do its job and worry about managing the full site sources so you don't have to.

The /mail URL is implemented by a symlink to /home/apmail/public-arch/$project.apache.org. As the CMS does not natively support the use of symlinks, you will need to add this symlink to your production site and add a corresponding line: "mail", to the extpaths.txt file in your source tree (mentioned earlier). Here are the manual steps:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
% svn co --depth=immediates https://svn.apache.org/repos/infra/websites/production/$project/content
% cd content
% ln -s /home/apmail/public-arch/$project.apache.org mail
% svn add mail
% svn commit -m 'add mail link' mail
% cd ..
% rm -rf content
% svn co --depth=immediates https://svn.apache.org/repos/asf/$project/site/trunk/content
% cd content
% echo mail >> extpaths.txt
% svn add extpaths.txt
% svn commit -m 'add mail to extpaths.txt' extpaths.txt

If you have time you may continue on to publish the extpaths.txt file, but it isn't strictly required so long as the document correctly appears on the staging site.

As the CMS is a REST application, no HTTP GET operations change server state- you will have to POST form data for that to happen. Hence any links presented in the webgui are safe to click on. The Static links just provide you a raw view of the underlying source files. Staged shows the current view of the file on the staging site, and Production shows the current view of the file on the live production site.

Running svn log on a file in the production tree gives too many results!

Don't do that. Run log against either the source site (consulting the cms:source-revision property to determine the associated revision) or the staging/ tree (consulting svn log -v on the production tree to determine the associated staging revision).

Running svn log on the production tree shows entries from staging.

svn log crosses copies, including the copy operation that publish is now implemented by. Run log one level up --- on production/${project} rather than production/${project}/content. (The log entries won't be very interesting; consider running log on the staging tree instead.)