| # Text Structural Representation Libraries |
| |
| <!--* |
| freshness: { owner: 'hzeller' reviewed: '2020-10-04' } |
| *--> |
| |
| At the heart of language-tooling libraries and applications lie various |
| structural representations of text, and the functions that operate on them. This |
| directory contains _language-agnostic_ data structures like: |
| |
| * Tokens: annotated substrings of a body of text, often what a lexer produces. |
| * Token streams: iterable representations of lexer output, including |
| filtered views thereof. |
| * Syntax trees: represent how parsers understand and organize code |
| hierarchically. |
| |
| ## Key Concepts |
| |
| `absl::string_view`s do not just represent text, but they represent **position** |
| within a larger body of text, by virtue of comparing their `begin` and `end` |
| bounds. This concept is leveraged heavily to avoid unnecessary string copying. A |
| base `string_view` that represents a body of text and serve as the basis for |
| interchanging between substring-views and byte-offsets relative to the start of |
| the base. |
