Stream Type System

For dealing with media streams of various implementation type, we need library routines, which also yield a type classification system suitable for their intended use. Most notably, for raw sound and video data we use the GAVL library, which defines a fairly complete classification system for buffers and streams. For the relevant operations in the Steam-Layer, we access each such library by means of a Façade; it may sound surprising, but actually we just need to access a very limited set of operations, like allocating a buffer. Within the Steam-Layer, the actual implementation type is mostly opaque; all we need to know is if we can connect two streams and get an conversion plugin.

Thus, to integrate an external library into Lumiera, we need explicitly to implement such a Lib Façade for this specific case, but the intention is to be able to add this Lib Façade implementation as a plugin (more precisely as a "Feature Bundle", because it probably includes several plugins and some additional rules)

At this point the rules based configuration comes into play. Mostly, to start with, determining a suitable prototype for a given implementation type is sort of a tagging operation. But it can be supported by heuristic rules and an flexible configuration of defaults. For example, if confronted with a media with 6 sound channels, we simply can’t tell if it’s a 5.1 sound source, or if it’s a pre mixed orchestra music arrangement to be routed to the final balance mixing or if it’s a prepared set of spot pick-ups and overdubbed dialogue. But a heuristic rule defaulting to 5.1 would be a good starting point, while individual projects should be able to set up very specific additional rules (probably based on some internal tags, conventions on the source folder or the like) to get a smooth workflow.

Moreover, the set of prototypes is deliberately kept open ended. Because some projects need much more fine grained control than others. For example, it may be sufficient to subsume any video under a single prototype and just rely on automatic conversions, while other projects may want to distinguish between digitized film and video NTSC and PAL. Meaning they would be kept in separate pipes an couldn’t be mixed automatically without manual intervention.

Instead of representing types by metadata, leave the distinction implicit and instead implement the different behaviour directly in code. Have video tracks and audio tracks. Make video clip objects and audio clip objects, each utilising some specific flags, like sound is mono or stereo. Then either switch, switch-on-type or scatter out the code into a bunch of virtual functions. See the Cinelerra source code for details.

In short, following this route, Lumiera would be plagued by the same notorious problems as most existing video/sound editing software. Which is, implicitly assuming “everyone” just does “normal” things. Of course, users always were and always will be clever enough to work around this assumption, but the problem is, all those efforts will mostly stay isolated and can’t crystallise into a reusable extension. Users will do manual tricks, use some scripting or rely on project organisation and conventions, which in turn creates more and more coercion for the “average” user to just do “normal” things.

To make it clear: both approaches discussed here do work in practice, and it’s more a cultural issue, not a question guided by technical necessities to select the one or the other.