Thoughts on Structured/Unstructured and Shared Information Management
using Wiki and other emerging technologies
Compound Documents for the web
John Udell recently highlighted the problem of Compound documents for the web:
An HTML Slidy presentation is a collection of files: a single main XHTML file, a JavaScript file, one or more CSS files, and one or more media files which can be images and, in my case, sometimes also movies. It runs identically from a local disk using the file: protocol and from the web using HTTP. But for an event, the host usually wants to receive your presentation and load it onto their machine. And the zip-transfer-unzip dance is more friction than anyone needs or wants.
Among the solutions proposed, either use RFC 2557 approach of including everything in single page (like a MIME message, but not quite so); or to use a zip/jar file to download and manage those files. Both of them are inadequate and not so web-friendly.
I think we should adopt ATOM standard for this purpose. Firstly, APP protocol (which manipulates ATOM resources) can be used to identify and manipulate individual resources/entries. Second, an atom feed is a single XML document that can contain multiple entries. Though this approach has been used to store feed entries, it is possible to come out with a simple approach where first entry has some sort of list of entries, and the rest of entries are in essence media entries, containing the inline data. All we then need is a good support (like javascript ATOM library) to extract/reference any specific content. In particular, we require a server side mechanism such that just placing this feed as a file on server will enable APP protocol capabilities to manipulate the contents and reference media entries as direct web resources.
The key reason why we should do it is: We just want a linear set of resources in a collection, and not a recursive structure (as in MIME).
powered by performancing firefox
BarCampPune ...
Finally BarCampPune is here and happening in Pune (courtesy Symphony for having given the facilities, sponsoring lunch etc.).
I presented my efforts in Structured Wiki area (Flexischema framework in twiki). If you are interested, please contact me - especially if you are good in perl.
(Some more, non-wiki blog postings in my other blog - Metavibes.
Great article in collaboration loop - by Steven Tedjamulia, Knowledge manager at Novel: Collaboration Loop - How Knowledge Workers Would Like to Work. The key points summarized from their internal survey of about 80 people:
- Standardized collaboration system that integrates tasks/emails/events etc. among all their interactions (i.e. with co-workers, customers etc.)
- Teams should be able to do content management and share information effectively
- Be able to set up collaborative workflows themselves (and not by IT dept). And that too with other information sources such as SAP!
- Be able to get to right experts.
- One search engine.
- Contextually identify the importance of document.
- User portals - where they can integrate different data themselves.
- Integration with presence information, for better simulation of face-to-face interactions (since, the whole purpose of collaborative systems is to be able to achieve good collaboration even when people are not in same room!)
In my own experience, this KM+Collaboration stack should use commonly understood paradigms. For e.g. to point out a good resource to a friend, you simply send the URL of that resource by email. But the problem is, this information remains in email. We need a way that will allow this person to integrate that information back into his own view of the same underlying collaborative world. (In any case, email integration remains a big issue in these scheme of things.)
Current recommendation for this type of stack:
- Wiki for uniform collaboration as well as document management (by having a topic for each submitted document. Use the same approach to convert a submitted mail into a topic. (Mails need to be shared this way - by forwarding to shared areas's email - so that it becomes public information in controlled manner.)
- The above wiki supported by good Search mechanisms.
- Tagging framework for subscription, ranking etc. Further the tagging framework (like delicious) needs to be uniform across intranet, extranet and internet resources. By using wiki, each resource would have unique URL, so this is indeed possible.
- User portals as extension to wiki. Have special plugins/macros to allow user to reconfigure how the page contents should look like. These plugins pull data from different sources. For e.g. twiki has plugins to display RSS feeds in any topic.
- Everyone uses wiki based collaboration as uniform approach. Team areas can be easily created in these wiki systems.
- RSS feeds off wiki to ensure easy consumption of information.
- We also require a mechanism (as part of collaborative areas) to reference other resources and organize these references in nice way.
- Expert location is very easy when everyone uses tagging infrastructure.
- Some Wiki systems allow workflows to be created by end users.
What's missing? I would of course like to see good email integration; especially for workflows. Usability is another area of importance, and wiki systems aren't exactly known for their usability (i.e. end users has to learn a lot.) Lot of ajaxy constructs are required (and usable by process owner) to create good processes within wiki frameworks.
We do keep experimenting with these approaches (-- in my day work, as being responsible for good KM implementations). Some aspects are easy, but some are cultural and still evolving...
BarCampMumbai - May 13th
It is very nice that BarCamps are providing a platform for Indian geeks.
I would be there in Mumbai for this event (at IITB, Powai), and I am interested in discussing in several emerging areas: Dynamic applications (i.e. which are created on the fly by business users), Wiki Systems (and our enterprise deployment experience), Structured Wiki (and some demos) and so on.
So do get in touch! (I work in Pune.)
-Vinod
Increasing the responsiveness email programs on desktop ...
As I type this, I am waiting for my thunderbird to refresh itself and help me see latest emails from our corporate IMAP server. This is a routine affair: As soon as I open the laptop, most of the hibernated program would have to get loaded into memory, and that often takes a long time. Sometimes as long as 10 minutes.
So what do I do? Resort to web-based email access which our corporate server provides. This seems to work because Firefox seems to consume less memory, and is "more responsive" compared to Thunderbird. Also, the issue could be with thunderbird having to initialize a large email store (I have anywhere between 3000 to 4000 emails in inbox at any time). The headers are synchronized first, and are available instantly. But the loading the body takes a lot of time.
So here is the idea: Enhance thinderbird extension so that it helps open email using browser - assuming that your webmail can take the URL with appropriate ID, and allows you to login without losing that ID...
Small niceties always help.
-Vinod
Future of template languages?
It seems like the templating mechanisms will continue to evolve. I came with some requirements and will watch how they will play out in future.
We need Javascript as well as Server side templates. Trimpath folks have a templating mechanism for javascript. Their javascript-equivalent of Ruby-on-rails has a good comparison of ruby template with their own templates (RubyRails Vs JavaScriptJunction ). One of good things this engine is the fact that it can be used both on server side (using Rhino) and client side. But my point is different: The same template should be rendered both on server side as well as client side. And that means some things are resolved by server, and some are rendered at client side. So how do we decide how to partition? For e.g. Layout is server side, but the contents in each boxes are necessarily driven by ajax. This in turn require JSON-RPC integration with underlying template mechanism. The objects used within the template need to be added to JSON-RPC bridge automatically. I should be able to simply add "remotehost=xyz.com" as one of the attribute to where the template is hosted, so the client-side template has access to server-side objects using JSON-RPC bridge seamlessly.
As the app devel cycle takes lesser and lesser time, the UI improvements will keep going on continuously. A simple "merge" operation should be sufficient to ensure that the developer-made changes are automatically integrated into next version of UI. This means templates should be designer-friendly and amenable to versioning, diff's and merges. This means that all the "code" part to be within attributes and special tags. Kid template language of Python does this right.
And again, assuming we spend time with UI, we require templates to be managed by content management system - and that means Wiki. And more important, we would like the capability to be uniformly available to end users as well. And this means we need to have "Safe" templates - those which can be edited from web itself. For example, Liquid, a ruby-based templating system, seems to do it.
Widget integration is going to be another interesting area. Dojo, prototype etc. already give us good widget environments. Widgets will probably make it easy to standardize on Layouts - such as 3-column, or 3-pane (like in outlook). How exactly should widgets get integrated with templating languages? In some sense, widgets seem to compete with standard templating approaches: A widget specificaiton is going to be very simple. Remember Dojo's ridiculously simple slideshow widget example specification? And the problem is, the widget is going to take inputs as parameters. Some of them will, well, be widgets. And this means that your template will basically be XML tree, and that's bad. (After all, templates should look like real HTML!) In any case, as long as browser can show this properly without requiring other servers, I think people will come to accept this approach. We would have to see how this will roll out. And not to forget, we are also talking of ajax-enabled widgets which need to integrate well with JSON-RPC or some such scheme.
In any case, javascript is going to be an important ingredient.
-Vinod
Structured Blogging vs. Google Base
The post from Niki Scevak
- Open letter to Google Base Product Manager highlights Structured Blogging as a better approach for aggregating structured content.
One of my focus being workflows around this concept (what I called Folkflows in my previous post), it will be interesting to see how this will progress. But the focus of this article is the very first application of Google Base (or for that matter, structured blogging): Simple match making. Like in classifieds.
Let us take a very simple use case: In my organization we have community newsgroups. People keep asking information about "recommend a doctor" or "where is a good restaurent in this location.". The information is amenable for google base: We will probably put up an interface to directly add the questions as well as results to google base. Both updates + search will be done using RSS or tab-separated file uploads, while giving some nice UI for standard shortcuts.
So what is the role of Google here?
- Being able to store the attribute-value pairs and requests.
- Be able to do matchmaking i.e. show the requests along with responses.
But we forget one more requirement: Be able to notify the people involved, and community. This is not exactly an atom feed problem: One can't subscribe to all individual feeds. For that matter, what is entry point of the feed is itself a question. Should I subscribe to a feed related "Anyone asking about anything in Pune, India"?
In some cases, it makes sense -for e.g. if I am tracking lowest prices of a particular product; but in most cases, we should realize that we are more attached to communities, and we will be ready to supply information to this closed community more easily than open-ended communities. This is a standard social fact.
This means, the feed that we subscribe to should be aggregation of all structured information at group/community level. It can't be a global feed based on item, given off by Google Base.
This model plays out very well when you add a simple Structured Data handling plugin to standard blog tools such as wordpress. And that is I would put Structured Blogging approach, integrated with standard feed servers is going to be important, and co-exist with Google Base. (We also note that group mailing list creation and management tools are also provided by google and yahoo, and they will obviously be integrated/enhanced to offer structured blogging.) We can assume that it is going to take some time to settle.
And folk-workflows will have to wait for attention till all this settle down.
Google Base will lead to "FolksFlows"
Finally long awaited Google Base has been released.
The interesting thing is: There is no separate API required to use it. Because:
- Adding an item can be done directly using RSS feed; with google specific extension tags.
- You don't delete an item; you expire it by specifying expiry date. Perhaps we need to see whether we can overwrite an item with new expiration date. We would require some kind of key for this action.
- You need RSS feed of a specific query in which you specify attribute-value pairs. Seems easy for me.
And labels act as schemas. Even though there is a lot of "control" - for e.g. can't specify a lot of words, there is enough power already. It is essentially a web-oriented database and it is going to make it easy for any application which fundamentally involves public data (such as looking for specific movies).
One question is: Would this database model result in Google making any inroads into enterprises over the time? For that matter, any other markets which are not driven by advertisements and end-consumers? For example, can small businesses hope to have applications hosted with this google database backend one day?
The answer seems to be "Yes, it will happen one day". The model opens up what I call "FolksFlows" - Workflows for general folks. What does this mean? Assume you want to set up a simple workflow. You have set of 10 sales people and you want to track sales on daily basis by these guys. Usual process is to get a guy to create a decent-sized application and worrying about things like how sales folks can update the data from field. But Google model will turn that upside down. We can now expect, with appropriate interfaces, a model where anyone can cookup a schema, use a wiki front end for UI, use google backend to store private data (this is tricky), and use Atom and other Web 2.0 approaches to provide interfaces. It can at most take about 2-3 hours by a common-sense oriented manager.
So the core of the problem is to use google database for private data. IMHO, it is actually easy part; I am sure Google will come out with a model for it as soon as Google base stabilizes. The second core of problem is: Ajaxi'fied Wiki front end that will use google base - and this is already in market by Dojo guys. And ISPs will follow suit as soon as someone targes one of the available open source wikis for this market (integrating, for e.g. Dojo toolkit for ajax-enabled widgets.)
Such apps can still use their own backend while using Google-provided standardized Schema. But Google Base will end up providing most easy way to integrate the app with other apps, and Integration is most difficult problem in these markets ...
So future is indeed exciting. Let us all hope FolksFlows will very soon become a reality!
-Vinod
Interdependence vs. Independence ...
From rediff article on Sri Sri Ravi Shankar's visit to Kashmir:
First Look: Sri Sri Ravi Shankar: Shankar responded to separatist voices by saying: 'Now the people and nations need more of interdependence rather than independence to live peacefully.'
How true! And How Spontaneous!
Wiki systems as application initiation front-end
Summary: Wiki systems can evolve to become "working area" for people; a kind of better desktop that integrates information from multiple applications. One step in this direction is to create simplified application front-ends - catered to specific task requirements most commonly required by users. (This would be a middle path between GUI driven/desktop approach and command line approach.)
A request in local unix newsgroup triggers this discussion. How can I see a sorted list of most recent 10 files in all of my home? Apart from usual 'see man pages' and even 'God helps those who read man pages' thingies, there was a largish issue that I am highlighting here. And how we could possibly tackle it by using wiki systems as our desktops.
As computing has progressed to more and more deeper levels of society, we should re-look at how quickly we can get the tasks done, especially when we are new to it. I am actually sad that after all these years, computers are still inaccessible to most (in true sense), and one has to fight to make it do what we want it to do.
The unix culture assumed that you learn once (at some decent time investments) and you will be very productive on the platform for rest of life. A simple concept of "terminal" and text-manipulation commands are sufficient. It also assumed that the users are intelligent enough to combine the tools and get the desired outputs themselves.
Windows GUI approach for applications, on the other hand, assumes no investments of time. Usually you can quickly figure out how to get something done. But then if you are better at computers, then you will certainly feel the limitedness of GUI. (This will obviously be contested by windows fans, but such is life: one can't experience everything, and make really comparative remarks anyway!) So a person requiring lot of flexibility gets stuck. There are other programmer-oriented interfaces such as VB, but then that's not exactly same in spirit as that of unix command line philosophy. A power user isn't necessarily a programmer. (We need to watch how new windows shell 'msh' will evolve: Udell has explored this here.)
So what is a middle ground, that allows intelligent users to create some most commonly required solutions and publish them in a way that other people can find and use easily? Essentially what you would do here is to reduce hundreds of combinations of use of combination of tools, to few - solution centric - uses which are known to be statistically the most required solutions.
What should be the approach?
- Solution Creation: Make sure an application can read independent 'solution config file' that integrates config file and appropriate command line interfaces, input/output destinations etc. It can change its avatar as per the associated solution config file. Intelligent users create such solution-config files. (Usually it should be XML format.) Note that a solution may use more than one tools as well.
- Solution discovery: A wiki integrates all such available solutions from multiple individuals' blogs or other publishing areas via RSS. Standardization is required to publish a solution via RSS.
- Solution use by end user: This would be within wiki environment, where the solution is amenable to provide good interfaces to consume the results and to pass around the information to other topics. A notable feature of wiki is to create a topic that represents some data output by the solution (and hence making it independent of solution.) So specifying inputs for a specific use is by merely quoting wiki topics. Wiki's also provide easy markup for things like tables and lists.
More I think about it, it seems to me that wiki is indeed a good information integration platform that multiple applications can share. It will facilititate use of XML within the integration, and yet providing simple markup interface to that data if user decides to hand-edit the data in wiki microformats.
Perhaps it makes sense for google to provide such an integration if they will also provide applications to go with (now that our emails - that constitute most of the information - are also in gmail)....
... Anyway, we never know how these things evolve! But this would be an interesting watch, since it is an open area for innovation.
-Vinod
Weblications (blog by Adam Rifkin)
Relax, Everything Is Deeply Intertwingled: Weblications:
"With web-based software, most users won't have to think about anything except the applications they use.
All the messy, changing stuff will be sitting on a server somewhere,
maintained by the kind of people who are good at that kind of thing...
Desktop software forces users to become system administrators.
Web-based software forces programmers to. There is less stress in
total, but more for the programmers..."
Unfortunately the state of web-based application development is rather sorry today. It is not as easy as simple native program development - inspite of all the MVC models, devel environments (such as Struts), and so on. There are simply so many mistakes one can make.
And, if ASP is the way to go, then what could enable rather a large scale offering of web-based applications for end users? A simple deployment environment from a big vendor such as yahoo or google. (Let us call them "deployment vendor"). Such an environment would
- Provide a clean mechanism for creating standards-based front-ends but enhanced by rich-client capabilities offered by such vendors.
- Be completely cross platform - since the deployment vendor can take care of variety of browser compatibility issues. And eventually be able to dictate a standard. (For example, we might reject a desktop platform because applications provided by google don't render well on its browser.)
- Be able to create high availability capability due to caching and other technologies - bringing the ASP app to the nearest node.
- Be able to offer a rich language processing and voice capabilities effectively creating better interaction with the applications.
- Be able to support variety of platforms such as mobile devices, PDAs etc.
- Provide offline capabilities.
And perhaps there are many other points. While most of these are usual ASP requirements, the enhancements because the vendor like google or yahoo have abundance of computing power, and can effectively create application development models around their infrastructure, is a different twist to traditional ASP.
And hopefully, these enhancements expressly incorporate principles of The web way.
Standardizing table syntax for wiki systems ...
Online table editing is very valuable tool - especially in enterprise systems. It could allow a lot of use-cases that require use of database to be done in wiki system itself. Simple form-fills for adding new rows (apart from directly getting table edits) helps simple workflows to be created quickly.
Jotspot demonstrates this nicely, but it seems to hide the actual table data - perhaps storing it in a external datastore associated with the topic. But it is nice to store the table information in topic itself. (Bulk editing, no problems in renaming ...)
But unfortunately there is no standardized syntax for table within a wiki topic (of course we don't have any standard as of now). Wikipedia is evolving to be a great centralized resource, so might propel a standard creation process.
In this context, a useful link is discussion on table syntax:
Help:Table - From Meta; discussion about Wikimedia projects. In particular, even using XHTML as table syntax would help a lot: It might allow someone to create a nice table editing tool - with plugin approach to add semantics such as Spreadsheet ...
-Vinod
Brain would like to minimize the efforts ...
Consider some of the threads in completely diverse fields:
- Amit's Thoughts: Thoughts about pricing - wherein the article, while listing choices available in pricing, brings out the point that fine-grain choices such as (per-call or per-view) are more costly when you consider the cost of thinking involved in making right decisions. For the same reason, gmail (where you don't bother to delete the emails) is much better choice than quota-enabled emails because you have to bother about making choices with each email. And when you have too many choices to make, the cost of email increases.
-
Neuro-aesthetics, Perceptual Grammar, Art, and Usability by Suman Kumar, wherein he connects usability to the fact that humans would like to minimize the work by brain (and supported by another study in this area).
And perhaps in many other areas (such as Adam Bosworth's emphasis on simiplicity in user interfaces, APIs etc.) - the same theme repeats. And yet, when we design things, we don't give sufficient importance to the simplicity angle - especially when you consider the cost of thinking involved in using and interacting with systems.
And perhaps the same principle is source of the bad designs in the first place!
Using (X)Wiki for Creating S5 presentations - Demo at Xwiki
Thanks to Ludovic Dubost for creating a demo of presentations authored using XWiki (using S5 CSS approach of Eric Meyer) - Link: XWiki . Demo . S5.
"Google suggest" Feature
What a great feature! What appeals is the simplicity of the idea and the way it was implemented. Browser as a rich-client model is going to new heights, demonstrating some really interesting capabilities.
Implemented by a google engineer Kevin Gibbs - in his 20% "research" time (as is culture in google), the capability simply uses javascript to do remote RPC with server to get a list of suggested keywords as explroed by Adam Stiles in his blog.).
Any suggestions for Javascript-based SDKs for rich-client Apps?
Beautiful quote from Nisargadatta ...
Nisargadatta : I Am That: "You observe the heart feeling, the mind thinking, the body acting; the very act of perceiving shows that you are not what you perceive."
-Vinod
PubSub: Competitor to Google? Idea of "Query sharing" frameworks ...
PubSub is a new kid on the block, and might offer some competition to google. In short, it is an amplified version of "Google Alerts" - which allowed you to track the news as it happened and send you alerts. (For example, I could track "application wiki" and get an email (or use their app) of all weblogs/news sources which will have these keywords.
So could it be a google competitor? Because ratio of searches for "existing information" to "new information" is going to reduce over the time, it could well be. PubSub has solved the tough part of matching your queries with changing information anywhere in the web. (Well, "anywhere" is probably only blogs and news sources for now, but I guess they can track changing websites as well). So storing those queries for a long-term is not a big deal. When you have enough queries that you actually find most queries are repeats, then the system is, in essence, google - without having to store all the web pages, but only the fact that they have changed. (Well, google cache is still something we love!)
Coming to sharing queries, I think that PubSub people should assign it a high priority. Stored queries introduce browses on information, and that is always good. For example "Top tutorials in XML" is a browse, whereas "Tutorials in XML" will just be a search. Obviously, the framework would also have ranking and all that, but then reputation services are going to be next big thing, right?
RESTful Flex
At last, the gap between standard HTML programming (which is primarily REST - if you design it well) and Flash-based designs which use single URL for whole application - has finally reduced with help of techniques from RESTful Flex.
But there is a fundamental thing that developers should remember: Interacting with flash application is a local activity without round trip times. So if you are forced to introduce round trip just because you want the app to be RESTful, I think it is not a good approach. Instead the URL in the browser should dynamically change with your interaction with the application.
Let me give example. Say you selected an user from the list, and want to send the URL by email to another friend. So selecting that user might trigger a URL-click, and some interaction between user and flash should ensure that URL will change in a browser, without actually going to the backend server. In other words, it is not a real URL click, but only a simulation.
-Vinod
S5: A Simple Standards-Based Slide Show System
Somebody had to do it: Using CSS and Javascript to create good presentations. Eric Meyer's S5 is precisely the same.
And it is right-fit for Wiki-based presentations: You edit a wiki page and use S5 markup to create a presentation text. And let wiki show the presentation by including right CSS and javascript files.
Some wiki systems do provide slideshow plugins, but it would be nice to see this all-thought off approach getting incorporated in them.
-Vinod
