MirrorNG : Architecture

Design schematic

Systems

MirrorNG allows defining different types of systems. An obvious type of system is a document management system such as an EMC Documentum docbase system. Another type of system is a mirror system. A mirror system stores content and metadata much like a document management system does. A mirror system is intended solely for storage and as such does not provide security, workflows, ...

A mirror system stores its metadata in a database and stores its content in a content store. We provide stores that place content on the filesystem - possibly in a compressed format - but additional store types can easily be added. A possible application is to store XML documents in an XML enabled database to enable fast searches and manipulation of the documents.

Going back in time

A mirror system not only stores the current data and content but also retains enough information to reconstruct the data and content as it was at any point in time. It is this feature that allows you to restore a document after it has been accidentally deleted. A mirror system also allows searching and browsing deleted documents and viewing the history of the metadata, thus enabling you to easily find back deleted documents or overwritten metadata.

Processes

We can define several processes on or between the different systems. An important process is a migration process which copies data from one system to another. When defined between a docbase system and a mirror system, this process is the basis for a backup solution where the mirror system is used as the backup, and for a migration solution where the mirror system is used as intermediate storage.

Other processes that are likely to be added in the future are processes that check the systems for integrity, that compare the data in two systems, or that export the data in the system to XML or to a spreadsheet.

Event driven design

The processes are event driven, enabling incremental rather than one shot processes. Each process can then be stopped at all times and be resumed later on, but more importantly it can react to changes in the systems involved. A nice consequence of this is that we can run these processes on a running system that people work with and make changes to. These changes generate events that the process picks up and acts on accordingly.

The basis of this event driven design is the use of event queues. Any process can register an event queue with a system, and the system commits to adding an event object to the queue for every event the process needs notification of. Upon registration of a queue with a system, it is filled with the events that the creation of the objects in the system would generate. This allows processes to handle already existing objects and new objects alike in a single, transparent way.

For mirror systems, adding the event objects to the registered queues is done directly in our application code. As we cannot change EMC Documentum to do the same, we need some other mechanism to adapt to changes in a docbase. Luckily, EMC Documentum allows registering events for about any change that can happen in a docbase. MirrorNG will then periodically check for new events in the docbase system and move them to the queues registered for this system.

Tasks

MirrorNG does all its work in tasks that are scheduled by a task manager. These tasks range from registering events in a docbase system to running a full migration from one docbase system to another. The task manager creates a configurable number of worker processes that are assigned tasks as needed. Currently, these tasks are executed on a single machine, but it would be easy to extend this to multiple machines to spread the load. When a task is done executing, the worker becomes idle and awaits the next task. This means that libraries remain loaded, avoiding the heavy start-up times that we have observed on some systems.