Clear message

MoinMoin 2.0

MoinMoin 2.0 (short: moin2) will be a big step in moin development, we are rewriting / heavily refactoring some core components of moin.

Documentation: moin2 online docs or docs/ in the repo.

Repo: http://hg.moinmo.in/moin/2.0 (Mirror on http://bitbucket.org/thomaswaldmann/moin-2.0 )

Issue Tracker: http://bitbucket.org/thomaswaldmann/moin-2.0/issues

Minefield: http://test.moinmo.in/

See also:

• the Talk at the DLR (text in German, pictures in English).

• the Europython 2010 Lightning Talk Slides (en)

• the 27c3 Lightning Talk Slides (en)

• the Europython 2010 Lightning Talk Video (en)

• the PyCologne Talk: Ausblick auf Moin 2.0 (de)

• the PyConDE: moin-2, wie man eine wiki software eigentlich schreiben sollte. (de) Video Aufzeichnung

The big news

Sphinx-based docs

We are working on some nice docs using the Sphinx documentation system (same as e.g. python.org is using).

See the moin2 docs on readthedocs.org or docs/ in the repo.

Docs are far from finished, help is appreciated.

Storage API

moin 1.x: the storage code was mixed into Page, PageEditor and AttachFile modules (yes, quite messy).

moin 2: storage code will be in MoinMoin.storage package and misc. storage backends will be supported, all conforming to the same "storage API".

Storage Layers: stores, backends, middlewares

• there are simple "stores" that really store key/value stuff somehow:
  • fs - filesystem store
  • sqlite - sqlite3 database (comes with python)
  • sqla - supports all sorts of SQL databases supported by SQLalchemy (PostgreSQL, MySQL, sqlite, ...)
  • Kyoto Cabinet and Kyoto Tycoon store
  • memory - memory store (only for special applications, like testing/debugging)
• on the layer above these stores, there are backends that offer metadata/data storage:
  • stores - tie together 2 stores to form a backend
  • fileserver - expose parts of server filesystem (read-only)
• there are some kinds of middlewares / layers above them:
  • routing middleware - "mount" different backends into the namespace (might get replaced by namespace routing soon)
  • indexing middleware - do powerful / complex stuff with the help of Whoosh, including search and indexing
  • protecting middleware - checks permissions with access control lists (ACLs)

Namespaces

• new namespaces can be defined
• map namespaces to backends
• map external wiki namespaces

More info: MoinMoin2.0/Namespaces

Mimetype items

In moin 1.x, there were 3 main types of stored objects in the wiki: pages, attachments and user profiles

In moin 2, there will be only 1 type of stored object: an "Item".

Some facts:

• metadata can be any key: value pair
• data can be of any type (text, binary)
• a revision is made from metadata and data
• an item is made up by all revisions sharing same ITEMID in their metadata

Metadata

As seen above, moin2 does much more with (own) metadata and even enables storage of user-defined metadata.

To speed up processing, metadata of items and revisions will be indexed (using whoosh).

Index can be used for:

• selecting wiki items/revisions by some criteria
  • by current item name, by uuid, ...

  • TitleIndex like item list

    • more difficult than in moin 1.x as we have all sorts of item mimetypes
    • but we either want to show just some specific mimetypes (== "wiki pages")
    • or we only want to show toplevel items?
  • theme: when showing some item, list sub-items in some panel
  • mimetype-specific listings

  • RecentChanges == items/revisions changed since date

  • "Items edited by me/by some editor"
  • Items linking to some specific item
  • Items tagged with some tag (tags [metadata] will replace categories [data])
• selecting user items by some criteria
  • by uid, by user name, by email, by jid, by ldapdn, ...
  • users subscribed to some wiki item
• sorting such lists (by name/date/...)

Item OO user interface

moin 1.x: pages and attachments had separate user interfaces (deleting a page worked differently than deleting an attachment, uploading an attachment worked differently than uploading page content, pages had revisions, attachments not).

moin 2: unified user interface for items:

• show item (render wiki page, render image/video with some additional data - all done in wiki content area)
• item infos / history
• modify item (create, edit, upload)
• rename (change name metadata)
• delete (remove name? TODO)
• download item content (raw wiki page, raw image)

Page, PageEditor, AttachFile code modules are dead.

TODO:

• UI still needs much polishing / theming before end user can use it, see themeing.

Code cleanup

Lots of crappy code was deleted, still some left.

Tree based transformations

(See /Converters for more information)

moin 1.x: parse wiki markup line-by-line, create html on the fly (problems: crap html, no xhtml, crap include, crap TOC, ...)

moin 2.0: input -> converter -> DOM tree -> converter -> output

• wiki -> DOM

• creole -> DOM

• reST -> DOM

• pygments -> DOM

• docbook-xml -> DOM

• image / audio / video -> DOM

• DOM -> html5

moin 2.0: reverse direction (e.g. for GUI editor)

• html -> DOM

• DOM -> wiki

• DOM -> reST

• ...

Features:

• generating well-formed x(ht)ml(5) output is easy now
• we use emeraldtree, which is a fork of elementtree
• operations on the dom tree make some stuff much cleaner / easier:
  • TOC
  • include
  • section edit (TODO)

• html -> DOM -> html filters out dangerous stuff

TODO:

• more practical testing of dom code, fix bugs

• some of the stuff has been worked on in GoogleSoc2010

Processing Instructions

HelpOnProcessingInstructions have been replaced by

GUI editor update and different approach

• ckeditor 3.x
• currently just stores html items

  • no conversion problems

  • but: no collaboration between gui editor and text editor users on the same item

• conversion is WIP (see Tree based transformations)

Themeing with Jinja2 templating engine

moin 1.x: Output generation was mostly done by python code with embedded html. Some of this was within pluggable themes, some was hardcoded.

moin 2.0: use Jinja2 templating engine for theme and user interface.

some of the stuff has been worked on in GoogleSoc2010.

HTML / CSS / Images / ...

• html5 (xmlish variant), we now can really do that

• jquery as base for cool (and less self-made) javascript stuff
• maybe svgweb for supporting svg stuff on crap browsers
• reorganize / cleanup css and images
• make css / js / html / images more extensible, modular, pluggable

TODO:

• Theme/CSS still needs lots of cleanup / work / a rewrite - we need help here from some artists / web designers / ui developers!

Serialization

Storage contents can be serialized / unserialized to a custom, simple format that is basically a stream of json (revision metadata) and binary data (revision data).

Applications:

• backup / restore
• import from / export to other apps
• import from moin 1.9

New plugin system

The old plugin system is not powerful enough and too complicated.

Just rough ideas, nothing final:

• use WSGI for action-like stuff (actions have full control over http response)?
• WSGI won't be enough as a general plugin mechanism (it is too low level, only for web stuff, not usable for internal stuff like macros, converters, etc.
• plugins should be able to have own i18n, static files (css, js, html, templates)

This is all still TODO!

Framework / Libs / Python

Already done:

• we require Python 2.6 now, so no need to test with older pythons any more
• We use Flask now as the framework.
  • restful URLs
  • werkzeug for WSGI stuff (since moin 1.9, will get even more and better used in moin2)
  • jinja2 for templating
  • using flask-themes for themes
  • flatland (+ j2 macros) for forms generation / validation of entered data
• use flask-babel, babel, pytz for i18n/l10n
• using flask-script for scripts
• using flask-cache for caching
• use whoosh for indexing / selecting / searching

• use xstatic to have static files in extra pypi packages

Packaging

moin2 got rid of all the big 3rd party stuff that was bundled with moin 1.x and uses python packaging now, so you can use pip to install it (e.g. into a virtualenv) and it'll fetch all requirements from pypi.

How to help

As you see above, all this is a quite big task and we need help:

• good Python, Javascript, HTML/CSS developers
• people with a good artistic skills and taste for theme development

How to start:

• Talk with us on MoinMoinChat.

• Read and try the new code.
• Expect brokenness/incompleteness and help fixing it.
• Bring in your ideas about how to improve. Some stuff is not specified yet.

Release date

As moin2 is still in heavy development, we can't give any release date yet.

"Maybe 2012" is maybe not totally unrealistic.

But we can give some criteria that help you to choose:

choose moin2 if ...

• you are a developer and

  • you want to hack it / fix it / improve it

  • you want to write a wiki in python (don't start from scratch, we already have quite some useful stuff and there's lots to play with)

  • you plan to do a bigger software development project related to moin that won't go into production soon anyway. if you would do it for moin 1.9, you'ld likely have to rewrite it for moin2. while you develop your stuff, moin2 will get nearer to release and moin 1.9 will get more outdated.

• you are a user and
  • you like to play with cool stuff, even if it is partly broken
  • you like less, but more powerful features

choose moin 1.9 if ...

• you are a user and
  • you need something stable for "production"
  • you need functionality that is not (yet?) present in moin2

  • you need something that is highly compatible with moin 1.9

  • have lots of important data in moin 1.x format
• you are a developer and
  • you want to do no or only small changes or extensions for moin (note that you can use some stuff we also use for moin2, like jinja2, flatland, sqlalchemy, ...)

Questions

Synchronisation of 2 wikis?

How about the synchronization of 2 wikis, is that included in moin2, and if, does it include also the attachments?

A: If we implement sync, it likely will include all types of items, but it is not implemented yet.

Page tree?

For navigation it is very useful to have a navigation tree for the pages (preferred left column). Is that possible now in moin2?
-- RudolfReuter 2011-02-22 13:40:10

A: Not implemented yet, but everything is possible - it may be just not in the first 2.0 release. If you want to help with a great theme just jump on it.

Porting extensions

How would one go about porting extensions to Moin 2.0? Right now, I can see lots of dependencies on the formatter and parser APIs in Moin 1.x that I would need to replace with something else. Will there be equivalents in Moin 2.0, and if so, are these things the kind of contributions that Moin 2.0 needs? If not, which APIs would I need to target? -- PaulBoddie 2012-02-25 18:22:46

moin2 is a rewrite of moin1 and works quite differently at many places. So likely a moin2 extension will be a rewrite of a moin1 extension. Please note that there is currently no plugin system. -- ThomasWaldmann 2012-02-25 22:17:16

---

Note: Some old content can be still seen here: here