The purpose of this DesignProcess entry is to collect together informations regarding the design and structure of the high-level model of Lumiera’s Steam-Layer. Most of the information presented here is already written down somewhere, in the Development TiddlyWiki and in source code comments. This summer, we had quite some discussions regarding meta-clips, a container concept and the arrangement of tracks, which further helped to shape the model as presented here.

While the low-level model holds the data used for carrying out the actual media data processing (=rendering), the high-level model is what the user works upon when performing edit operations through the GUI (or script driven in "headless"). Its building blocks and combination rules determine largely what structures can be created within the Session. On the whole, it is a collection of media objects stuck together and arranged by placements.

Basically, the structure of the high-level model is is a very open and flexible one — every valid connection of the underlying object types is allowed — but the transformation into a low-level node network for rendering follows certain patterns and only takes into account any objects reachable while processing the session data in accordance to these patterns. Taking into account the parameters and the structure of these objects visited when building, the low-level render node network is configured in detail.

The fundamental metaphor or structural pattern is to create processing pipes, which are a linear chain of data processing modules, starting from an source port and providing an exit point. Pipes are a concept or pattern, they don’t exist as objects. Each pipe has an input side and an output side and is in itself something like a Bus treating a single media stream (but this stream may still have an internal structure, e.g. several channels related to a spatial audio system). Other processing entities like effects and transitions can be placed (attached) at the pipe, resulting them to be appended to form this chain. Optionally, there may be a wiring plug, requesting the exit point to be connected to another pipe. When omitted, the wiring will be figured out automatically. Thus, when making an connection to a pipe, output data will be sent to the source port (input side) of the pipe, wheras when making a connection from a pipe, data from it’s exit point will be routed to the destination. Incidentally, the low-level model and the render engine employ pull-based processing, but this is rather of no relevance for the high-level model.

Normally, pipes are limited to a strictly linear chain of data processors ("effects") working on a single data stream type, and consequently there is a single exit point which may be wired to an destination. As an exception to this rule, you may insert wire tap nodes (probe points), which explicitly may send data to an arbitrary input port; they are never wired automatically. It is possible to create cyclic connections by such arbitrary wiring, which will be detected by the builder and flagged as an error.

While pipes have a rather rigid and limited structure, it is allowed to make several connections to and from any pipe — even connections requiring an stream type conversion. It is not even necessary to specify 'any' output destination, because then the wiring will be figured out automatically by searching the context and finally using some general rule. Connecting multiple outputs to the input of another pipe automatically creates a mixing step (which optionally can be controlled by a fader). Several pipes may be joined together by a transition, which in the general case simultaneously treats N media streams. Of course, the most common case is to combine two streams into one output, thereby also mixing them. Most available transition plugins belong to this category, but, as said, the model isn’t limited to this simple case, and moreover it is possible to attach several overlapping transitions covering the same time interval.

Individual Media Objects are attached, located or joined together by Placements . A Placement is a handle for a single MObject (implemented as a refcounting smart-ptr) and contains a list of placement specifications, called LocatingPin. Adding an placement to the session acts as if creating an instance . (it behaves like a clone in case of multiple placements of the same object). Besides absolute and relative placement, there is also the possibility of a placement to stick directly to another MObject’s placement, e.g. for attaching an effect to a clip or to connect an automation data set to an effect. This stick-to placement creates sort of a loose clustering of objects: it will derive the position from the placement it is attached to. Note that while the length and the in/out points are a property of the MObject , it’s actual location depends on how it is placed and thus can be maintained quite dynamically. Note further that effects can have an length on their own, thus by using these attachement mechaics, the wiring and configuration within the high-level model can be quite time dependant.

Actually a clip is handled as if it was comprised of local pipe(s). In the example shown here, a two-channel clip has three effects attached, plus a wiring plug. Each of those attachments is used only if applicable to the media stream type the respective pipe will process. As the clip has two channels (e.g. video and audio), it will have two source ports pulling from the underlying media. Thus, as showed in the drawing to the right, by chaining up any attached effect applicable to the respective stream type defined by the source port, effectively each channel (sub)clip gets its own specifically adapted processing pipe.