# Text Structural Representation Libraries

<!--*
freshness: { owner: 'hzeller' reviewed: '2020-10-04' }
*-->

At the heart of language-tooling libraries and applications lie various
structural representations of text, and the functions that operate on them. This
directory contains _language-agnostic_ data structures like:

*   Tokens: annotated substrings of a body of text, often what a lexer produces.
    *   Token streams: iterable representations of lexer output, including
        filtered views thereof.
*   Syntax trees: represent how parsers understand and organize code
    hierarchically.

## Key Concepts

`absl::string_view`s do not just represent text, but they represent **position**
within a larger body of text, by virtue of comparing their `begin` and `end`
bounds. This concept is leveraged heavily to avoid unnecessary string copying. A
base `string_view` that represents a body of text and serve as the basis for
interchanging between substring-views and byte-offsets relative to the start of
the base.