MoinMoin: Storage2009

MoinMoin Storage

As of Google's "Summer of Code" 2008, MoinMoin got a new storage abstraction layer (by ChristopherDenter) to allow usage of different backends for data/user storage and also a Mercurial storage backend (by PawelPacana). In parallel to SOC 2008, 2 filesystem backends have been written.

In Google Summer of Code 2009, work on storage will continue to make it production ready and finally leading to a MoinMoin 2.0 release that includes that storage code after SOC 2009.

See below for information on what we plan to do and what you as the end user get from it.

Kinds of TODOs:

• SOCTODO means that this is part of the SOC 2009 project of ChristopherDenter (support him, but leave the coding to him)

• TODO means: help needed, anybody is welcome to help with this (please coordinate here or on #moin-dev)

Repository

http://hg.moinmo.in/moin/2.0-storage/

SOCTODO (last week)

• remove all cfg.acl_rights_* configuration that should be done by mapper parameters now. fix the bugs you'll see afterwards.

  • Can somebody please take a look at docs/config_examples/snippets/openid_teams_wikiconfig_snippet? I don't know where the acls specified there are used and am unsure how that should be cleaned up correctly -- will inform rkerr to look at it
• language_setup action (or alternatively making unserialize more selective -- TW is working on that. Let's see what he comes up with.)

• get rid of _tests/data

• docs

  • The code itself is documented in the module, class and function/method docstrings (epydoc style). Read through all the docs and enhance them if necessary.

  • Documentation for admins: /HelpOnStorageConfiguration

    • section about migration

    • fix advanced configuration example

      • use unicode for acls

      • maybe show the mapping that the simple mapper does for "fs:..." (I mean it doesn't give much yet to show Talk or UserHomepage namespaces if they are not used by the moin code)

  • Documentation for developers: /HelpOnStorageDevelopment

  • Documentation on ACLs: /HelpOnAccessControlLists

    • hierarchic not up to date

    • there's no wikidicts cache anymore

    • remove "new in 1.6"

    • think about Trusted: part of the default acl, compare with 1.9

      • Please elaborate

    • remove references to autoadmin

• Make create_item allow non-subsequent (but monotone) revision numbers

• SQLABackend

TODO (general)

• Sane UI for 2.0-storage!
  • Proper input fields (especially for metadata..)
  • Indicate missing permissions *before* the actual action
    • Remove href on "Immutable Page"
  • More beautiful 'access denied' error pages

• deprecated handling totally messed up, it just needs to go away (See HelpOnProcessingInstructions; #deprecated PI)

• Xapian stuff must be adjusted to work with 2.0-storage.
  • Note: There has been work on Xapian in 1.9 ongoing as part of GSoC 2009. That is not yet merged.
  • Make sure that attachfile is ripped out of Xapian stuff.
  • Make Xapian indexing operate on unprotected_storage and make ABSOLUTELY sure that the indexed stuff is not accessible without prior permission checking!

• Kill LocalEditLog. It's not working properly anyway.

  • do that after xappy merge as it is used by moin's xapian code
• Get rid of Page/PageEditor.
  • editlog related stuff must go. Use rev[EDIT_LOG_CONSTANT_GOES_HERE] instead. Should be easy.
• write action that visualizes $TRASHBIN contents and offers "undelete" functionality and read-access to all revisions of deleted items
  • this is partly there, in a very simple form:
    • use .../Trash?do=index to show trash contents
    • just rename items back to their original name to "undelete" them
• action for UserProfile/ items
  • this is partly there, in a very simple form:

    • use .../UserProfiles?do=index to show user profiles

    • no rendering yet as it is item metadata, not revisions

• make FSBackend work on win32 platform ( Help! We currently have no developer working on windows. ).

• MercurialBackend:

  • needs more testing
  • needs more documentation
• flatfile backend
  • metadata support is likely lacking some stuff
  • needs more testing
  • use it as backend for caching module?
• syncpages action: it should sync generic items, not just page items

• Code that uses old other parts of the code (and not the storage api, although it should), should be refactored to use storage api and the then unused parts should be removed. (e.g. Xapian, Page, PageEditor...)

• maybe the current security policy stuff should move to the acl wrappers, just having stuff like r,w,c,d is not enough, we need more context for stuff like e.g. giving homepage owners more powers

• think about interaction of acl wrapper and acl protected group definition pages (e.g. a page AdminGroup with acl AdminGroup:...) -- elaborate!

• make sure no acl-protected pages are available through the cache (If there is such a thing where Items/Revisions end up and are cached)
• kill the global event-log?

Finished Stuff

• (Provide dummy ACL middleware)

• Make all requests pass through dummy ACL middleware

• Remove current ACL checks (ACL checks should reside in the middleware exclusively)

• Start working on the real ACL middleware

• Fix all tests that fail due to storage errors

• Keep user.may but make it use the aclbackend._may instead. (For now.)

• Documentation (ACL code has been documented)

• Switch to permanent storage

• Make RouterBackend the default

• Throw user_backend and data_backend out of wikiconfig

• Provide sufficient amount of tests for RouterBackend

• Provide item/revision wrappers for RouterBackend

• Provide EndUserBackend that just gets a string like 'memory', 'sqlite:///database.db' or 'path/to/instance' and then automatically dispatches to the correct backend. (needs to be finalized at some point. sufficient for now)

• Make methods check for the proper privileges.

• Perhaps finally get that nuke_item stuff done? (needs to be implemented by all backends)

• rip out all the underlay stuff

• implement MoinMoin.storage.upgrade_syspages(itemname)

• Add 2 metadata entries for syspages:

  • "this is a system page"

    • if the page is edited, this flag is copied to the new revision

  • "original syspage as contained in release <release>"

    • if the page is edited, this flag *won't* be copied to the new revision

• Add support for installing syspages package (item containing the serialized xml in its latest revision)

  • creates new revisions on top of existing stuff

• Make inter-backend rename copy to the correct item directly

• Add tell() to storage API

• Provide create_simple_mapping helper

• kill redundant ACLs

  • killed "revert"

  • killed "delete" (we use trashbin-like delete now, that is just a rename)

  • migration script removing "revert" and "delete" from page ACLs (built into fs19 now)

• improve AccessDenied error handling

  • internal backdoor? We have that now: request.unprotected_storage

• trashbin-like delete + empty item revs - implemented and committed -- ThomasWaldmann 2009-08-01 15:30:39

• AttachFile action shall DIE, finally.

• new, common mimetype-item-based UI, supporting pages and files in the same way.

• flexible storage configuration approach

• Router middleware is now used by default. It is configured by providing a mapping, like [('temp/', MemoryBackend(), dict(default="All:read,write,create,destroy,admin")), ('/', FSBackend('path/to/instance', dict="All:read,write,create"))]

Betatesting (TODO)

Put simply: Betatesters, we need you!

The changes to MoinMoin described above are quite huge. Since this also touches ACLs, it is even a concern of security to test early and test often. If you have some time and are willing to help us by testing early versions, please don't hesitate to contact us.

Diary

Discussion

Add your thoughts, concerns and use-cases here.

• Macros non-functional
• For later:
  • Use SQL timestamps and the datetime module

-- BastianBlank 2009-08-17 10:12:01

CategoryGsocProject