Managing Structured data in TWiki
I keep working with twiki. The ideas keep coming at fast pace.
These days I am working on making structured data manageable in twiki (and in general, wiki) framework. The Topic names provide a uniform resource locator (URL; yes, sometimes you need to spell it to highlight the significance!) which act as achors to specific sets of data you are interested in. This data could be a simple list, a set of rows of a table (for which, say, you are administratively responsible), and so on. Due to plug-in nature, the data could come from a Database, and yet be available in format we choose to keep them within the topic.
So first part is: How do you keep structured data in twiki, and yet make it editable? TWiki chooses to use simple lists with capitalized words in cases such as Access controls, groups etc. I am trying to use for the same purpose. The reasonings are:
- It is XML in disguise. Even though YAML folks claim to contrary. At least you get some power of XML, and you don't lose anything.
- Hand editable, if really required
- Good support for perl. In fact, you can parse the YAML data and directly create Perl structures.
So, I have chosen it for specifying any structured content in TWiki topic. (Typically wrapped with some markup so the plugins can identify that what follows is structured data.)
The next step is to be able to display this data nicely - in variety of ways. I now have a plugin to do exactly this. Currently it works with table data. My idea is to couple this with variety of CSS styles so user can choose how he would like to see the table.
How do we add the content to this table? Either handedit the topic. Or use forms interface. I have this done already. However, we have to think and see what are other options. For example, SVG is an emerging standard. XForms is also another standard.
And then we require integration with Databases. It is no good if my table is part of topic. Some people feel that databases are end-all of all structured data. Though I don't quite agree, we should anyway respect the fact that they are extremely well supported, and provide good integration with other tools in enterprise.
It is easy if my data has to reside only within a database. A plug-in can then act as intermediary. The topic only contains table name, database name and may be the set of columns. (Or may be the SQL query.)
But this is not acceptable. Why?
- The twiki topic is a "View". Even though it is dynamic view, capabilities such as Search would be far more effective if the view is materialized, and available to Search and other twiki functionalities.
- Integration with Email: TWiki model currently is centralized; meaning the information being processed remains central. But because it is collaboration tool, it has to, eventually, support synchronization between multiple installations. Specifically, we would like to have twiki topics delivered by email, and if required, edited and sent as email. This can only work if the topic views are same as their actual contents.
So, it becomes clear that twiki topics should have an internal cache, and mechanisms to update it. There is already a Cache plugiin. But this is not sufficient; we require good control over caching behaviour. I plan to do a plugin that allows you to do "Render and Cache" i.e. it has two arguments: the content to render (typically INCLUDE or SEARCH), and second argument is actual cache. It either shows only contents of cache, or it updates the cache with rendered twiki markup, and then shows the page.
More on this design later.
