The player — shorthand for “playback and render control subsystem” is responsible for establishing and coordinating the processes and services necessary for generating continuous output or rendered media. The actual calculations take place within the render engine, while the player instructs the engine in case of any modification regarding the parameters of timed data delivery. More specifically, there is a play controller (state machine) for each instance of playback, managing an allocated output slot with a set of output sinks, plus a number of calculation streams, which in turn are cooperating with the frame dispatcher and the scheduler interface. The latter acts as low-level interface to the actual rendering operations, since it allows to schedule individual frame jobs together with their prerequisite dependencies. This setup creates a pattern of cooperation, with the goal of delivering a continuous stream of calculated data to the output sinks. Specific changes within the parametrisation of that cooperation pattern result in an overall changed mode of playback or rendering.
Rendering of final results
The classic case of “rendering the results” is treated within this framework as a special case of generic play-back: Any frames are to be delivered when done with the calculation, without any timing constraints, while, on the other hand, no glitches or quality compromises whatsoever are acceptable. Also, the play controller exposes only limited abilities: in case of rendering, all we need is the ability to pause and abort the process, plus an progress indicator.
Regular playback
The standard case of playback delivers a sequence of media data frames at a given frame rate, with simple linear temporal progression. Typically, there are multiple playback feeds linked together, each delivering a specific kind of media at a specific frame or block rate, while being coordinated by a single play controller and working to a common delivery time goal (“play head”). Which means in turn, that each individual frame has to be delivered within a pre planned time window. Lumiera employs an elaborate and precise timing control, based on triggering small atomic chunks of work in a “just in time” manner — there is no generic and built-in buffering beyond the low-level double buffering mechanisms utilised by most output facilities. Generally speaking, we prefer precise beforehand planning and discarding of untimely results over demand-driven and possibly blocking operations. Yet the buffer management and frame cache provided by the vault for storing and passing of intermediary and final results allows for a certain amount of leeway.
Non-standard playback modes
The requirements of a software player, especially in the context of an editing application, call for support of several specific operation modes of playback — which can be characterised by breaking with the rule of simple and linear progression in timed delivery. The following requirement analysis defines our understanding and implementation approach towards these mandatory features.
Requirement analysis of playback modes
For the task of editing a piece of media, the following modes of playback presentation can be determined
See also the more
technical discussion of playback in the TiddlyWiki
- regular playback
-
In this operation mode, some media or the arrangement on one of the timelines will be presented in the intended order and with the nominal speed, possibly somewhat adjusted for the display or monitoring device employed. There is a running time code display and an animated presentation of the current nominal playback position (“play head”, “cursor”, “locator”) in the GUI. Generally speaking, within Lumiera this playback position isn’t incorporated as a specific device — rather it is treated as a conceptual mapping of nominal time to wall clock time.
- jumping
-
creates a discontinuity in nominal time, while the progress of real wall clock time deadlines remains unaffected. We need to distinguish two kinds of jumps or skips:
-
a pre planned jump can be worked into the progression of frames just like in normal progression. Such a jump might be due to a skip mark in the timeline, or it may be used to implement looped playback.
-
to the contrary, a spontaneous re-adjustment of playback position is caused by unpredictable external events, like e.g. user interaction or remote control. Such an unpredictable change in the playback plan deprives the engine of any delivery headroom; to allow for catch-up to timed delivery, a configurable slippage offset can be added to the newly established real time deadlines after the skip, in order to prevent drop-outs.
Since each skip might create an output discontinuity, a “de-clicking” or “de-flickering” device can be expected to intercept on the individual output connections to improve the usage experience while exploring media.
-
- looping
-
In looped playback the nominal playback position jumps back to the starting point, after travelling over a pre defined looping time span (“looping window”). This looping time span can be re-adjusted while in playback, with immediate effect on the output. A fluid handling of loop playback is extremely important for all kinds of fine tuning work. The loop boundaries may be chosen arbitrary, not necessarily aligned to any frame grid (the fact this might cause some irregularities on the presented frame sequence is acknowledged, yet it is deemed acceptable and preferred over employing any kind of additional interpolation device not present in normal playback).
Looped playback can be conceived as a series of regularly scheduled jumps. - pausing
-
paused playback represents a special state of the player and engine, where we expect playback to be able to commence right away, with minimal latency. Contrast this to entering playback from stopped state, where services need to be initiated and connected, leading to an noticeable pre-delay. Basically, each other playback mode can be entered from paused state right away, and it is reasonable for the engine to prepare for this through background pre rendering.
- single stepping
-
through user interaction, the current nominal playback position is moved to an adjacent frame, while sending exactly this one frame to output as a still image. This is a critical operation for frame accurate editing and should be reachable from any other playback mode with minimal latency. Lumiera treats this mode of operation as an extension to paused state.
- playback direction and speed
-
while regular playback establishes a 1:1 relation between nominal time and wall clock time, a range of other proportions need to be supported, like e.g. halved and doubled presentation speed both forwards and backwards; it is conceivable to allow irregular fractions too. Support for speed adjustments in sound playback is more involved as it might seem at first glance, so it is common to allow for a degraded sound representation, or to fall back on fast-cueing.
- fast cueing
-
The purpose of cuing is to skip through a large amount of material to spot some specific parts. For this to work, the presented material needs to be still recognisable in some way. Typically this is done by presenting small continuous chunks of material interleaved with regular skips. For editing purposes, this method is often perceived as lacking, especially by experienced editors. To the contrary, the former, mechanical editing systems had the ability to run with actually increased frame rate, without skipping any material.
-
it is very common to employ speed acceleration in multiple steps to alleviate travelling to more distant locations within the media
-
to improve the editor’s working experience, we might consider actually to raise the frame rate, given the increased availability of high-framerate capable displays.
-
another approach would be to apply some kind of granular synthesis, dissolving several consecutive segments of material. The latter would prompt to include a specific buffering and processing device not present in the render path for normal playback.
-
- scrubbing
-
this term relates to working with physical media, like magnetic tape or film strips the past: instead of using the built-in transport device, the editor would spin the reels by hand in order to navigate to a desired position intuitively. When translated to the world of digital media, the actual scrubbing facility is an interactive device, typically even involving some kind of hardware control interface (“jog shuttle”). The playback engine needs to support scrubbing, which translates into chasing a playback target, which is re-adjusted regularly. Again it would be desirable to include some kind of acceleration strategy, especially to support a soft landing.