docs/f4pga/modules/index.md

# Modules

## Interface

This document contains all the information needed to configure modules for
your _**f4pga**_ project as well as some info about the API used to write
modules.

### Configuration interface:

Modules are configured through an internal API by _**f4pga**_.
The basic requirement for a module script is to expose a class with `Module`
interface.

_**f4pga**_ reads its configuration from two different sources:
**platform's flow definition**, which is a file that usually comes bundled with f4pga
and **project's flow configuration**, which is a set of configuration options provided by the user
through a JSON file or CLI interface.

Those sources contain snippets of _module configurations_.

A _module configuration_ is a structure with the following fields:

* `takes` - a dictionary that contains keys which are names of the dependencies used by the module.
  The values are paths to those dependencies.
  They can be either singular strings or lists of strings.

* `produces` - a dictionary that contains keys which are names of the dependencies produced by the module.
  The values are requested filenames for the files generated by the module.
  They can be either singular strings or lists of strings.

* `values` - a dictionary that contains other values used to configure the module.
  The keys are value's names and the values can have any type.

### Platform-level configuration

In case of **platform's flow definition**, a `values` dictionary can be defined
globally and the values defined there will be passed to every module's config.

Those values can be overridden per-module through `module_options` dictionary.

Parameters used during module's construction can also be defined in `module_options`
as `params` (those are not a part of _module configuration_, instead they are used
during the actual construction of a module instance, before it declares any of its
input/outputs etc.. This is typically used to achieve some parametrization over module's
I/O).

Defining dictionaries for `takes` and `produces` is currently disallowed within
**platform's flow definition**.

For examples of **platform's flow definition** described here, please have a look at
`f4pga/platforms/` directory. It contains **platform flow definitions** that come bundled
with f4pga. 

### Project-level configuration

This section describes **project's flow configuration**.

Similarly to **platform's flow definition**, `values` dict can be provided.
The values provided there will overwrite the values from
**platform's flow definition** in case of a collision.

Unlike **platform's flow definition**, **project's flow configuration** may contain
`dependencies` dict. This dictionary would be used to map symbolic dependency
names to actual paths. Most dependencies can have their paths resolved implicitly
without the need to provide explicit paths, which is a mechanism that is described
in a later section of this document. However some dependencies must be provided
explicitly, eg. paths to project's Verilog source files. It should be noted that
depending on the flow definition and the dependency in question, the path does not
necessarily have to point to an already existing file. If the dependency is a
product of a module within the flow, the path assigned to it will be used
by the module to build that dependency. This is also used to in case of _on-demand_
dependencies, which won't be produced unless the user explicitly provides a path
for them.

**project's flow configuration** cannot specify `params` for modules and does not
use `module_options` dictionary. Neither it can instantiate any extra stages.

Any entry with a couple _exceptions*_ is treated as a platform name.
Enabling support for a given platform within a **project's flow configuration** file
requires having an entry for that platform.
Each of those entries may contain `dependencies`, `values` fields which will
overload the `dependecies` and `values` defined in a global scope of
**project's flow configuration**. Any other field under those platform entries
is treated as a _stage-specific-configuration_. The key is a name of a stage within
a flow for the specified platform and the values are dicts which may contain
`dependencies` and `values` fields that overload `dependencies` and `values`
respectively, locally for the stage. Additionally a `default_target` field can be
provided to specify a default target to built when the user does not specify it through
a CLI interface.

The aforementioned _*exceptions_ are:

* `dependencies` - dependencies shared by all platforms.
* `values` - values shared by all platforms
* `default_platform` - default platform to chose in case it doesn't get specified
  by the user

Those apply only to flow configuration file.

### Internal environmental variables

It's very useful to be able to refer to some data within
**platform's flow definition** and **project's flow configuration** to
either avoid redundant definitions or to store and access results of certain operations.
_**f4pga**_ allows doing that by using a special syntax for accessing internal
environmental variables.

The syntax is `${variable_name}`. Any string value within
**platform's flow definition** and **project's flow configuration** that contains
such patterns will have them replaced with the values of the variables referenced
if those values are strings. Eg.:

With the following values defined:

```json
{
  "a_value": "1234",
  "another_value": "a_value: ${a_value}"
}
```

`another_value` will resolve to:

```json
"a_value: 1234"
```

If the value is a list however, the result would be a list with all entries being
the original string with the reference to a variable replaced by following
items of the original list. Eg.:

With the following values defined
```json
{
  "list_of_values": ["a", "b", "c"],
  "some_string": "item: ${list_of_values}"
}
```

`some_string` will resolve to

```json
["item: a", "item: b", "item: c"]
```

Be careful when using this kind of resolution, as it's computational and memory
complexity grows exponentially in regards to the number of list variables being
referenced, which is a rather obvious fact, but it's still worth mentioning.

The variables that can be referenced within a definition/configuration fall into 3
categories:

* **value references** - anything declared as a `value` can be accessed by it's
  name
* **dependency references** - any dependency path can be referenced using the name
  of the dependency prefaced with a ':' prefix. Eg.: `${:eblif}` will resolve
  to the path of `eblif` dependency. Make sure that the dependency can be
  actually resolved when you are using this kind of reference. For example
  you can't use the a reference to `eblif` dependency in a module which does not
  rely on it. An exception is the producer module which can in fact reference it's
  own outputs but these references cannot be used during the _mapping_ stage
  (more on that later).
* **built-in references** - there are a couple of built-in variables which are very
  handy:
  * `shareDir` - path to f4pga's _share_ directory.
  * `binDir` - path to f4pga's _bin_ directory.
  * `prjxray_db` - Project X-Ray database path.
  * `python3` - path to Python 3 interpreter.
  * `noisyWarnings` - (this one should probably get removed)

### `Module` class

Each module is represented as a class derived from `Module` class.

The class should implement the following methods:

* `execute(self, ctx: ModuleContext)` - executes the module in _exec_ mode
* `map_io(self, ctx: ModuleContext) -> 'dict[str, ]'` - executes the module in
  _mapping_ mode
* `__init__(self, params: 'dict[str, ]')` - initializer. The `params`
  is a dict with optional parameter for the module.

Each module script should expose the class by defining it's name/type alias as
`ModuleClass`. f4pga tries to access a `ModuleClass` attribute within a package
when instantiating a module.

### Module's execution modes

A module has essentially two execution modes:

* _mapping_ mode
* _exec_ mode

#### _mapping_ mode

In _mapping_ mode the module is provided with an incomplete configuration which
includes:
  * `takes` namespace: this maps names of input dependencies to the paths of these
    dependencies
  * `values` namespace: this maps names of variables to the values of those
    variables.

The module has to provide a dictionary that will provide every output dependency
that's not _on-demand_ a default path. This is basically a promise that when
executed in _exec_ mode, the module will produce files for this paths.
Typically such paths would be derived from a path of one of it's input dependencies.
This mechanism allows the user to avoid specifying an explicit path for each
intermediate target.

It should be noted that variables referring to the output dependencies
can't be accessed at this stage for the obvious reason as their values are yet
to be evaluated.

#### _exec_ mode

In _exec_ mode the module does the actual work.

The configuration passed into this mode is full and it includes:

* `takes` namespace: this maps names of input dependencies to the paths of these
    dependencies
* `values` namespace: this maps names of variables to the values of those
   variables.
* `produces` namespace: this maps names of output dependencies to explicit paths.
  This should not be used directly really, but it's useful for
  `ModuleContext.is_output_explicit` method.
* `outputs` namespace: this maps names of output dependencies to their paths.

When the module finishes executing in _exec_ mode, all of the dependencies
described in `outputs` should be present.

### Module initialization/instantiation

In the `__init__` method of module's class, the following fields should be
set:

* `takes` - a list of symbolic dependency names for dependencies used by the module
* `produces` - a list of symbolic dependencies names for dependencies produced
  by the module.
* `values` - a list of names given to the variables used withing the module
* `prod_meta` - A dictionary which maps product names to descriptions of these
  products. Those entries are optional and can be skipped.

#### Qualifiers/decorators

By default the presence of all the dependencies and values is mandatory
(In case of `produces` that means that the module always has to produce the listed
dependencies). This can be changed by "decorating" a name in one of the following
ways:

* '`?`' _suffix_
  * In `takes` - the dependency is not necessary for the module to execute
  * In `produces` - the dependency may be produced, but it is not guaranteed.
  * In `values` the value is not required for the module to execute.
    Referring to it through `ModuleContext.values.value_name` won't raise an
    exception if the value is not present, instead `None` will be returned.
* '`!`' _suffix_
  * In `produces` - the dependency is going to be produced only if the user
    provides an explicit path for it.

Currently it's impossible to combine both '`!`' and '`?`' together. This limitation
does not have any reason behind it other than the way the qualifier system
is implemented at the moment. It might be removed in the future.

## Common modules

```{toctree}
fasm
generic_script_wrapper
io_rename
mkdirs
pack
place
place_constraints
route
synth
```