[daisy] Daisy - fields with (large) hierarchical data structures

[Karel Vervaeke]

Dear list,

I am currently working on a project for working with large hierarchyical
data structures.  Allow me te describe the goal (it is a bit long,
sorry) followed by our approach so far.  I will shortly make a branch
and commit some initial work.
Of course, ideas and feedback of any kind are welcome and much
appreciated.

The basic goal is to create fields with selection lists on steroids.
These selection lists contain more than just a bunch of value+label
combinations: they contain hierarchical data that is too large to send
across to the browser in one piece and where each node is more than a
simple value+label combination.  The hierarchical data itself will be
stored in one or more daisy documents, and the fields will store a
nodeId, multiple nodeIds, a path of nodeIds or multiple paths of nodeIds
(depending on whether it is a multivalue and/or a hierarchical field)

The hierarchical data can be split across multiple documents (using
imports, like navigation documents).

--------------1-DSY at main:en
node 1
  node 1.1
  node 1.2
    [import 2-DSY at main:en]
  node 1.3
node 2

--------------2-DSY at main:en
node 1.2.1
node 1.2.2
node 1.2.3

Each node gets a UUID, and may have one or more properties.  For
example, consider a hierarchy that describes the components of a machine
in a hierarchical fashion.  Each part consists of smaller parts. Each
part may have properties that describe the diameter, color, weight, ...
A node can also have a label (which describes the node in a more human
term dan a generated UUID).

There may be different types of nodes which each have a different set of
properties, for example when describing a machine there may be nuts
(diameter), bolts (diamater, length), screws (length, head type, ...),
etc.  To know what properties apply to each node type, a schema document
is used.  This document describes for each node type
- which properties are required
- which node type its children are allowed to have.

The widget for browsing/editing the hierarchical structure (we have been
calling it 'histwi' internally, short for 'HIerarchical STructure
WIdeget') will have a cascading lists UI: see here
http://www.time-tripper.com/uipatterns/Cascading_Lists
and here
http://johnvey.com/features/deliciousdirector/demo.html
(this example does not really represent hierarchical data, but the
mechanism is the same).

Here is an ascii rendering the widget (the data is based on a simplistic
hierarchical representation of a house):

-------------------------------------
| search: [________]        [edit ] |
-------------------------------------
| FLOOR   | ROOM       | FUNITURE   | 
| > 0     | > living   | chair      |
| 1       | hall       | > sofa <   |
| 2       | bedroom    | table      |
-------------------------------------
|  properties:                      |
|  color: [ blue ]                  |
|  length: [ 3m ]                   |
-------------------------------------

The last selected item is indicated between > and <.  The path to this
item is indicated with >.  The lower half of the widget shows the
properties for the selected node.

The top section has a search box to help you find nodes and an edit
button that is used when you want to add nodes to the hierarchical data.

The histwi will have different operating modes depending on where it is
used:  We want the histwi to serve as an editor for hierarchy data
documents or as a widget on the document editor's field tab.  When
working for a multi-value field, the widget should have additional
checkboxes in front of each node.

Lastly, as mentioned, nodes can have a label.  For better i18n support,
nodes can also have i18n labels.  Similarly node properties can also
have labels or i18n labels:  Consider a property value like "#0000ff" - 
it is more user-friendly to add a label 'blue', and even better to add
i18n labels describing the color in different languages)

On the repository side, there will be a Hierarchy Manager component
(which is currently implemented as an extension.).  The hierarchy
manager merges information from different documents (nodes importing
other documents) into a single tree.  This tree can (later on) be
indexed to make searching faster.

Because the hierarchies are potentially large, the hierarchy manager
provides methods to retrieve sections of the hierarchy (e.g. two levels
at a time) or to update sections without having to send the entire
hierarchy or even an entire document back and forth.  While doing this,
the hierarchy manager should take access control into account (not
allowing users to read nodes that belong to documents they do not have
access to, and not allowing users to write to documents for which they
do not have write access).

The repository can not enforce that fields always contain valid and
existing nodeIds (e.g. broken references when a node is removed from a
document), but the histwi should handle all cases elegantly (showing
error nodes, similar  to the navigation manager).

The hierarchy manager will listen to document events.  When a document
changes, the goal is not to have to rebuild the entire hierarchy, but
simply to remove and/or replace the nodes which are affected.

That's about it...  If there are any questions, go ahead!

Regards,
Karel