text-template.hpp File Reference

Go to the source code of this file.

Description

A minimalistic text templating engine with flexible data binding.

Text template instantiation implies the interpretation of a template specification, which contains literal text with some placeholder tags. This is combined with an actual data source; the engine needs to retrieve data values as directed by key names extracted from the placeholders and render and splice them into the placeholder locations. This process is crucial for code generation, for external tool integration and is also often used for dynamic web page generation. Several external libraries are available, offering a host of extended functionality. This library implementation for internal use by the Lumiera application however attempts to remain focused on the essential functionality, with only minimal assumptions regarding the data source used for instantiation. Rather than requiring data to be given in some map, or custom JSON data type, or some special property-tree or dynamic object type, a data binding protocol is stipulated; this way, any data type can be attached, given that five generic functions can be implemented to establish the binding. By default, a pre-defined binding is provided for a STL map and for Lumiera's »External Tree Description« format based on Record<GenNode>.

Template syntax and features

TextTemplate is able to substitute simple placeholders by name, it can handle conditional sections and supports a data iteration construct for a nested scope. The supported functionality is best explained with an example:

Rendered at ${date}.
${if critical}
WARNING: critical!
${else}(routine report)${end if critical}

**Participants**
${for person} - ${name} ${if role}(${role})${end if role}
${else} _no participants_
${end for person}

This template spec is parsed and preprocessed into an internal representation, which can then be rendered with any suitable data source.

• the placeholder ${date} is replaced by a value retrieved with the key "date"
• the conditional section will appear only if a key "critical" is defined
• when the data defines content under the key "person", and this content can be suitably interpreted as a sequence of sub-scopes, then the »for block« is instantiated for each entry, using the values retrieved through the keys "name" and "role". Typically these keys are defined for each sub-scope
• note that ${role} is enclosed into a conditional section, making it optional
• note that both for conditional sections, and for iteration, an else branch can optionally be defined in the template. How data is actually accessed and what constitutes a nested scope is obviously a matter of the actual data binding, which is picked up through a template specialisation for lib::TextTemplate::DataSource

Implementation notes

The template specification is parsed and compiled immediately when constructing the TextTemplate instance. At this point, syntactical and logical errors, e.g. mismatched conditional opening and closing tags will be detected and raised as exceptions. The compiled template is represented as a vector of action tokens, holding the constant parts as strings in heap memory and marking the positions of placeholders and block bounds. The branching and looping possibly happening later, on instantiation, is prepared by issuing appropriate branching and jump markers, referring to other points in the sequence by index number...

• TEXT stores a text segment to be included literally
• KEY marks the placeholders, storing the key to retrieve a substitution value
• COND indicates a branching point, based on a data value retrieved by key
• ITER indicates the start of an iteration over data indicated by key
• LOOP marks the end of the iterated segment, linked back to the start
• JUMP represents an unconditional jump to the index number given Whenever an else-section is specified in the template, a JUMP is emitted beforehand, while the first TEXT in the else-section is wired as refIDX from the starting token.

The actual instantiation is initiated through TextTemplate::submit(), which picks a suitable data binding (causing a compilation failure in case no binding can be established). This function yields an iterator, which will traverse the sequence of action tokens precompiled for this template and combine them with the retrieved data, yielding a std::string_view for each instantiated chunk of the template. The full result can thus be generated either by looping, or by invoking util::join() on the provided iterator.

Data Access

** The instantiation processing logic is defined in terms of a data binding, represented as TextTemplate::DataSource. This binding, assuming a generic data access protocol, has to be supplied by a concrete (partial) specialisation of the template DataSource<DAT>. This allows to render the text template with structured data, in whatever actual format the data is available. Notably, bindings are pre-defined for string data in a Map, and for Lumiera's »External Tree Description« format, based on a generic data node. Generally speaking, the following abstracted primitive operations are required to access data:

• the DataSource<DAT> object itself is a copyable value object, representing an abstracted reference to the data. We can assume that it stores a const * internally, pointing to some data entity residing elsewhere in memory.
• it must somehow be possible, to generate a nested sub-data context, represented by the same reference data type; this implies that there is some implementation mechanism in place to tap into a nested sub-scope within the data.
• bool dataSrc.contains(key) checks if a binding is available for the given key. If this function returns false no further access is attempted for this key.
• string const& retrieveContent(key) acquires a reference to string data representing the content bound to this key. This string content is assumed to remain stable in memory during the instantiation process, which exposes a std::string_view
• Iter getSequence(key) attempts to »open« a data sequence, assuming that the key somehow links to data that can somehow be interpreted as a sequence of nested sub-data-entities. The result is expected as »Lumiera Forward Iterator«.
• DataSource<DAT> openContext(Iter) is supplied with the Iter from getSequence() and assumed to return a new data binding as DataSource object, tied to the nested data entity or context corresponding to the current »yield« of the Iterator. This implies that a Iter it can be advanced by ++iter and then passed in again to get the data-src (reference handle) to access the next »sub entity«, repeating this procedure until the iterator is exhausted (bool false). Moreover, it is assumed, that recursive invocations of retrieveConent(key) on this sub-scope reference will yield the data values designated by key for this sub-entity, as well as possibly also accessing data _visible from enclosing scopes.

Map Binding

The preconfigured binding to std::map<string,string> implements this protocol — relying however on some trickery and conventions, since the map as such is one single „flat“ data repository. The intricate part relates to iteration (which can be considered more a »proof of concept« for testing). More specifically, accessing data for a loop control key should yield a CSV list of key prefixes. These are combined with the loop control key to form a prefix for individual data values: "<loop>.<entity>.<key>". When encountering a "key" while in iteration, first an access is attempted with this _decoration prefix; if this fails, a second attempt is made with the bare key alone. See TextTemplate_test::verify_iteration, which uses a special setup, where a string of key=value pairs is parsed on-the-fly to populate a map<string,string>

ETD Binding

While the Map Binding detailed in the preceding paragraph is mostly intended to handle simple key substitutions, the more elaborate binding to GenNode data (ETD), which is provided in the separate header text-template-gen-node-binding.hpp, is meant to handle structural data, as encountered in the internal communication of components within the Lumiera application — notably the »diff binding« used to populate the GUI with entities represented in the Session Model in Steam-Layer. The mapping is straight-forward, as the required concepts can be supported directly

• Key lookup is translated into Attribute Lookup — starting in the current record and possibly walking up a scope path
• the loop key accesses a nested Attribute (lib::diff::GenNode) and exposes its children for the iteration; thus each entity is again a Rec<GenNode> and can be represented recursively as a DataSource<Rec<GenNode>>
• the DataSource implementation includes an optional parent link, which is consulted whenever Attribute Lookup in the current record does not yield a result.

See also

TextTemplate_test

text-template-gen-node-binding.hpp

gnuplot-gen.hpp

SchedulerStress_test

Definition in file text-template.hpp.

#include "lib/error.hpp"
#include "lib/nocopy.hpp"
#include "lib/iter-index.hpp"
#include "lib/iter-explorer.hpp"
#include "lib/format-string.hpp"
#include "lib/format-util.hpp"
#include "lib/regex.hpp"
#include "lib/util.hpp"
#include <memory>
#include <string>
#include <vector>
#include <stack>
#include <map>

Classes
struct	TextTemplate::Action
class	TextTemplate::ActionCompiler
struct	DataSource< DAT, SEL >
class	DataSource< DAT, SEL >
	Binding to a specific data source. More...
struct	DataSource< MapS >
	Data-binding for a Map-of-strings. More...
struct	DataSource< string >
	Adapter for the Map-Data-Source to consume a string spec (for testing) More...
class	TextTemplate::InstanceCore< SRC >
	Iterator »State Core« to process the template instantiation. More...
class	TextTemplate::InstanceCore< SRC >
	Iterator »State Core« to process the template instantiation. More...
struct	TextTemplate::ParseCtx
struct	TagSyntax
class	TextTemplate
	Text template substitution engine. More...

Typedefs
using	MapS = std::map< string, string >
using	PairS = std::pair< string, string >
using	StrView = std::string_view

Functions
template<class STR , typename = meta::enable_if<meta::is_StringLike<STR>>>
	DataSource (STR const &) -> DataSource< string >
	Deduction Guide: help the compiler with picking the proper specialisation for a test-data source defined through a string spec or char literal.
return	explore (util::RegexSearchIter{input, ACCEPT_MARKUP}) .transform(classify)
else	if (mat[2].matched) tag.syntax
auto	iterBindingSeq (string const &dataDef)
auto	iterNestedKeys (string key, StrView const &iterDef)
auto	parse (string const &input)

Variables
const regex	ACCEPT_BINDING_ELM {MATCH_DELIMITER + MATCH_BINDING_TOK}
const regex	ACCEPT_DATA_ELM {MATCH_DELIMITER + MATCH_DATA_TOKEN}
const regex	ACCEPT_MARKUP
const string	MATCH_BINDING_KEY = R"~(([\w\.]+))~"
const string	MATCH_BINDING_TOK = MATCH_BINDING_KEY+"\\s*=\\s*(?:"+MATCH_BINDING_VAL+"|"+MATCH_QUOTED_VAL+")"
const string	MATCH_BINDING_VAL = R"~(([^,;"\s]+)\s*)~"
const string	MATCH_DATA_TOKEN = R"~(([^,;"\s]*)\s*)~"
	< Parser and DataSource binding for lib::TextTemplate
const string	MATCH_DELIMITER = R"~((?:^|,|;)\s*)~"
const string	MATCH_ELSE_TOK = "else"
const string	MATCH_END_TOK = "end\\s*"
const string	MATCH_ESCAPE = R"~((\\\$))~"
const string	MATCH_FIELD = "\\$\\{\\s*(?:"+MATCH_SYNTAX+")\\s*\\}"
const string	MATCH_KEY_PATH = MATCH_SINGLE_KEY+"(?:\\."+MATCH_SINGLE_KEY+")*"
const string	MATCH_LOGIC_TOK = "if|for"
const string	MATCH_QUOTED_VAL = R"~("([^"]+)"\s*)~"
const string	MATCH_SINGLE_KEY = "[A-Za-z_]+\\w*"
const string	MATCH_SYNTAX = "("+MATCH_ELSE_TOK+")|(?:("+MATCH_END_TOK+")?("+MATCH_LOGIC_TOK+")\\s*)?("+MATCH_KEY_PATH+")?"
else if("end"==mat[5]) throw error else tag	syntax = TagSyntax::KEYID
return	tag
tag	tail = rest

Namespaces
	lib
	Implementation namespace for support and library code.

---

Class Documentation

DataSource

class DataSource

Collaboration diagram for DataSource< DAT, SEL >:

TextTemplate::ParseCtx

struct TextTemplate::ParseCtx

Class Members
Clause	clause
Idx	begin
Idx	after

Collaboration diagram for TextTemplate::ParseCtx:

Variable Documentation

ACCEPT_MARKUP

const regex ACCEPT_MARKUP

Initial value:

{ MATCH_ESCAPE+"|"+MATCH_FIELD
                              , regex::ECMAScript|regex::optimize
                              }

Definition at line 247 of file text-template.hpp.