_sources/f4pga/modules/index.md.txt - symbiflow-docs - Git at Google

 # Modules

 ## Interface

 This document contains all the information needed to configure modules for
 your _**f4pga**_ project as well as some info about the API used to write
 modules.

 ### Configuration interface:

 Modules are configured through an internal API by _**f4pga**_.
 The basic requirement for a module script is to expose a class with `Module`
 interface.

 _**f4pga**_ reads its configuration from two different sources:
 **platform's flow definition**, which is a file that usually comes bundled with f4pga
 and **project's flow configuration**, which is a set of configuration options provided by the user
 through a JSON file or CLI interface.

 Those sources contain snippets of _module configurations_.

 A _module configuration_ is a structure with the following fields:

 * `takes` - a dictionary that contains keys which are names of the dependencies used by the module.
   The values are paths to those dependencies.
   They can be either singular strings or lists of strings.

 * `produces` - a dictionary that contains keys which are names of the dependencies produced by the module.
   The values are requested filenames for the files generated by the module.
   They can be either singular strings or lists of strings.

 * `values` - a dictionary that contains other values used to configure the module.
   The keys are value's names and the values can have any type.

 ### Platform-level configuration

 In case of **platform's flow definition**, a `values` dictionary can be defined
 globally and the values defined there will be passed to every module's config.

 Those values can be overridden per-module through `module_options` dictionary.

 Parameters used during module's construction can also be defined in `module_options`
 as `params` (those are not a part of _module configuration_, instead they are used
 during the actual construction of a module instance, before it declares any of its
 input/outputs etc.. This is typically used to achieve some parametrization over module's
 I/O).

 Defining dictionaries for `takes` and `produces` is currently disallowed within
 **platform's flow definition**.

 For examples of **platform's flow definition** described here, please have a look at
 `f4pga/platforms/` directory. It contains **platform flow definitions** that come bundled
 with f4pga.

 ### Project-level configuration

 This section describes **project's flow configuration**.

 Similarly to **platform's flow definition**, `values` dict can be provided.
 The values provided there will overwrite the values from
 **platform's flow definition** in case of a collision.

 Unlike **platform's flow definition**, **project's flow configuration** may contain
 `dependencies` dict. This dictionary would be used to map symbolic dependency
 names to actual paths. Most dependencies can have their paths resolved implicitly
 without the need to provide explicit paths, which is a mechanism that is described
 in a later section of this document. However some dependencies must be provided
 explicitly, eg. paths to project's Verilog source files. It should be noted that
 depending on the flow definition and the dependency in question, the path does not
 necessarily have to point to an already existing file. If the dependency is a
 product of a module within the flow, the path assigned to it will be used
 by the module to build that dependency. This is also used to in case of _on-demand_
 dependencies, which won't be produced unless the user explicitly provides a path
 for them.

 **project's flow configuration** cannot specify `params` for modules and does not
 use `module_options` dictionary. Neither it can instantiate any extra stages.

 Any entry with a couple _exceptions*_ is treated as a platform name.
 Enabling support for a given platform within a **project's flow configuration** file
 requires having an entry for that platform.
 Each of those entries may contain `dependencies`, `values` fields which will
 overload the `dependecies` and `values` defined in a global scope of
 **project's flow configuration**. Any other field under those platform entries
 is treated as a _stage-specific-configuration_. The key is a name of a stage within
 a flow for the specified platform and the values are dicts which may contain
 `dependencies` and `values` fields that overload `dependencies` and `values`
 respectively, locally for the stage. Additionally a `default_target` field can be
 provided to specify a default target to built when the user does not specify it through
 a CLI interface.

 The aforementioned _*exceptions_ are:

 * `dependencies` - dependencies shared by all platforms.
 * `values` - values shared by all platforms
 * `default_platform` - default platform to chose in case it doesn't get specified
   by the user

 Those apply only to flow configuration file.

 ### Internal environmental variables

 It's very useful to be able to refer to some data within
 **platform's flow definition** and **project's flow configuration** to
 either avoid redundant definitions or to store and access results of certain operations.
 _**f4pga**_ allows doing that by using a special syntax for accessing internal
 environmental variables.

 The syntax is `${variable_name}`. Any string value within
 **platform's flow definition** and **project's flow configuration** that contains
 such patterns will have them replaced with the values of the variables referenced
 if those values are strings. Eg.:

 With the following values defined:

 ```json
 {
   "a_value": "1234",
   "another_value": "a_value: ${a_value}"
 }
 ```

 `another_value` will resolve to:

 ```json
 "a_value: 1234"
 ```

 If the value is a list however, the result would be a list with all entries being
 the original string with the reference to a variable replaced by following
 items of the original list. Eg.:

 With the following values defined
 ```json
 {
   "list_of_values": ["a", "b", "c"],
   "some_string": "item: ${list_of_values}"
 }
 ```

 `some_string` will resolve to

 ```json
 ["item: a", "item: b", "item: c"]
 ```

 Be careful when using this kind of resolution, as it's computational and memory
 complexity grows exponentially in regards to the number of list variables being
 referenced, which is a rather obvious fact, but it's still worth mentioning.

 The variables that can be referenced within a definition/configuration fall into 3
 categories:

 * **value references** - anything declared as a `value` can be accessed by it's
   name
 * **dependency references** - any dependency path can be referenced using the name
   of the dependency prefaced with a ':' prefix. Eg.: `${:eblif}` will resolve
   to the path of `eblif` dependency. Make sure that the dependency can be
   actually resolved when you are using this kind of reference. For example
   you can't use the a reference to `eblif` dependency in a module which does not
   rely on it. An exception is the producer module which can in fact reference it's
   own outputs but these references cannot be used during the _mapping_ stage
   (more on that later).
 * **built-in references** - there are a couple of built-in variables which are very
   handy:
   * `shareDir` - path to f4pga's _share_ directory.
   * `binDir` - path to f4pga's _bin_ directory.
   * `prjxray_db` - Project X-Ray database path.
   * `python3` - path to Python 3 interpreter.
   * `noisyWarnings` - (this one should probably get removed)

 ### `Module` class

 Each module is represented as a class derived from `Module` class.

 The class should implement the following methods:

 * `execute(self, ctx: ModuleContext)` - executes the module in _exec_ mode
 * `map_io(self, ctx: ModuleContext) -> 'dict[str, ]'` - executes the module in
   _mapping_ mode
 * `__init__(self, params: 'dict[str, ]')` - initializer. The `params`
   is a dict with optional parameter for the module.

 Each module script should expose the class by defining it's name/type alias as
 `ModuleClass`. f4pga tries to access a `ModuleClass` attribute within a package
 when instantiating a module.

 ### Module's execution modes

 A module has essentially two execution modes:

 * _mapping_ mode
 * _exec_ mode

 #### _mapping_ mode

 In _mapping_ mode the module is provided with an incomplete configuration which
 includes:
   * `takes` namespace: this maps names of input dependencies to the paths of these
     dependencies
   * `values` namespace: this maps names of variables to the values of those
     variables.

 The module has to provide a dictionary that will provide every output dependency
 that's not _on-demand_ a default path. This is basically a promise that when
 executed in _exec_ mode, the module will produce files for this paths.
 Typically such paths would be derived from a path of one of it's input dependencies.
 This mechanism allows the user to avoid specifying an explicit path for each
 intermediate target.

 It should be noted that variables referring to the output dependencies
 can't be accessed at this stage for the obvious reason as their values are yet
 to be evaluated.

 #### _exec_ mode

 In _exec_ mode the module does the actual work.

 The configuration passed into this mode is full and it includes:

 * `takes` namespace: this maps names of input dependencies to the paths of these
     dependencies
 * `values` namespace: this maps names of variables to the values of those
    variables.
 * `produces` namespace: this maps names of output dependencies to explicit paths.
   This should not be used directly really, but it's useful for
   `ModuleContext.is_output_explicit` method.
 * `outputs` namespace: this maps names of output dependencies to their paths.

 When the module finishes executing in _exec_ mode, all of the dependencies
 described in `outputs` should be present.

 ### Module initialization/instantiation

 In the `__init__` method of module's class, the following fields should be
 set:

 * `takes` - a list of symbolic dependency names for dependencies used by the module
 * `produces` - a list of symbolic dependencies names for dependencies produced
   by the module.
 * `values` - a list of names given to the variables used withing the module
 * `prod_meta` - A dictionary which maps product names to descriptions of these
   products. Those entries are optional and can be skipped.

 #### Qualifiers/decorators

 By default the presence of all the dependencies and values is mandatory
 (In case of `produces` that means that the module always has to produce the listed
 dependencies). This can be changed by "decorating" a name in one of the following
 ways:

 * '`?`' _suffix_
   * In `takes` - the dependency is not necessary for the module to execute
   * In `produces` - the dependency may be produced, but it is not guaranteed.
   * In `values` the value is not required for the module to execute.
     Referring to it through `ModuleContext.values.value_name` won't raise an
     exception if the value is not present, instead `None` will be returned.
 * '`!`' _suffix_
   * In `produces` - the dependency is going to be produced only if the user
     provides an explicit path for it.

 Currently it's impossible to combine both '`!`' and '`?`' together. This limitation
 does not have any reason behind it other than the way the qualifier system
 is implemented at the moment. It might be removed in the future.

 ## Common modules

 ```{toctree}
 fasm
 generic_script_wrapper
 io_rename
 mkdirs
 pack
 place
 place_constraints
 route
 synth
 ```
	# Modules

	## Interface

	This document contains all the information needed to configure modules for
	your _f4pga_ project as well as some info about the API used to write
	modules.

	### Configuration interface:

	Modules are configured through an internal API by _f4pga_.
	The basic requirement for a module script is to expose a class with `Module`
	interface.

	_f4pga_ reads its configuration from two different sources:
	platform's flow definition, which is a file that usually comes bundled with f4pga
	and project's flow configuration, which is a set of configuration options provided by the user
	through a JSON file or CLI interface.

	Those sources contain snippets of _module configurations_.

	A _module configuration_ is a structure with the following fields:

	* `takes` - a dictionary that contains keys which are names of the dependencies used by the module.
	The values are paths to those dependencies.
	They can be either singular strings or lists of strings.

	* `produces` - a dictionary that contains keys which are names of the dependencies produced by the module.
	The values are requested filenames for the files generated by the module.
	They can be either singular strings or lists of strings.

	* `values` - a dictionary that contains other values used to configure the module.
	The keys are value's names and the values can have any type.

	### Platform-level configuration

	In case of platform's flow definition, a `values` dictionary can be defined
	globally and the values defined there will be passed to every module's config.

	Those values can be overridden per-module through `module_options` dictionary.

	Parameters used during module's construction can also be defined in `module_options`
	as `params` (those are not a part of _module configuration_, instead they are used
	during the actual construction of a module instance, before it declares any of its
	input/outputs etc.. This is typically used to achieve some parametrization over module's
	I/O).

	Defining dictionaries for `takes` and `produces` is currently disallowed within
	platform's flow definition.

	For examples of platform's flow definition described here, please have a look at
	`f4pga/platforms/` directory. It contains platform flow definitions that come bundled
	with f4pga.

	### Project-level configuration

	This section describes project's flow configuration.

	Similarly to platform's flow definition, `values` dict can be provided.
	The values provided there will overwrite the values from
	platform's flow definition in case of a collision.

	Unlike platform's flow definition, project's flow configuration may contain
	`dependencies` dict. This dictionary would be used to map symbolic dependency
	names to actual paths. Most dependencies can have their paths resolved implicitly
	without the need to provide explicit paths, which is a mechanism that is described
	in a later section of this document. However some dependencies must be provided
	explicitly, eg. paths to project's Verilog source files. It should be noted that
	depending on the flow definition and the dependency in question, the path does not
	necessarily have to point to an already existing file. If the dependency is a
	product of a module within the flow, the path assigned to it will be used
	by the module to build that dependency. This is also used to in case of _on-demand_
	dependencies, which won't be produced unless the user explicitly provides a path
	for them.

	project's flow configuration cannot specify `params` for modules and does not
	use `module_options` dictionary. Neither it can instantiate any extra stages.

	Any entry with a couple _exceptions*_ is treated as a platform name.
	Enabling support for a given platform within a project's flow configuration file
	requires having an entry for that platform.
	Each of those entries may contain `dependencies`, `values` fields which will
	overload the `dependecies` and `values` defined in a global scope of
	project's flow configuration. Any other field under those platform entries
	is treated as a _stage-specific-configuration_. The key is a name of a stage within
	a flow for the specified platform and the values are dicts which may contain
	`dependencies` and `values` fields that overload `dependencies` and `values`
	respectively, locally for the stage. Additionally a `default_target` field can be
	provided to specify a default target to built when the user does not specify it through
	a CLI interface.

	The aforementioned _*exceptions_ are:

	* `dependencies` - dependencies shared by all platforms.
	* `values` - values shared by all platforms
	* `default_platform` - default platform to chose in case it doesn't get specified
	by the user

	Those apply only to flow configuration file.

	### Internal environmental variables

	It's very useful to be able to refer to some data within
	platform's flow definition and project's flow configuration to
	either avoid redundant definitions or to store and access results of certain operations.
	_f4pga_ allows doing that by using a special syntax for accessing internal
	environmental variables.

	The syntax is `${variable_name}`. Any string value within
	platform's flow definition and project's flow configuration that contains
	such patterns will have them replaced with the values of the variables referenced
	if those values are strings. Eg.:

	With the following values defined:

	```json
	{
	"a_value": "1234",
	"another_value": "a_value: ${a_value}"
	}
	```

	`another_value` will resolve to:

	```json
	"a_value: 1234"
	```

	If the value is a list however, the result would be a list with all entries being
	the original string with the reference to a variable replaced by following
	items of the original list. Eg.:

	With the following values defined
	```json
	{
	"list_of_values": ["a", "b", "c"],
	"some_string": "item: ${list_of_values}"
	}
	```

	`some_string` will resolve to

	```json
	["item: a", "item: b", "item: c"]
	```

	Be careful when using this kind of resolution, as it's computational and memory
	complexity grows exponentially in regards to the number of list variables being
	referenced, which is a rather obvious fact, but it's still worth mentioning.

	The variables that can be referenced within a definition/configuration fall into 3
	categories:

	* value references - anything declared as a `value` can be accessed by it's
	name
	* dependency references - any dependency path can be referenced using the name
	of the dependency prefaced with a ':' prefix. Eg.: `${:eblif}` will resolve
	to the path of `eblif` dependency. Make sure that the dependency can be
	actually resolved when you are using this kind of reference. For example
	you can't use the a reference to `eblif` dependency in a module which does not
	rely on it. An exception is the producer module which can in fact reference it's
	own outputs but these references cannot be used during the _mapping_ stage
	(more on that later).
	* built-in references - there are a couple of built-in variables which are very
	handy:
	* `shareDir` - path to f4pga's _share_ directory.
	* `binDir` - path to f4pga's _bin_ directory.
	* `prjxray_db` - Project X-Ray database path.
	* `python3` - path to Python 3 interpreter.
	* `noisyWarnings` - (this one should probably get removed)

	### `Module` class

	Each module is represented as a class derived from `Module` class.

	The class should implement the following methods:

	* `execute(self, ctx: ModuleContext)` - executes the module in _exec_ mode
	* `map_io(self, ctx: ModuleContext) -> 'dict[str, ]'` - executes the module in
	_mapping_ mode
	* `__init__(self, params: 'dict[str, ]')` - initializer. The `params`
	is a dict with optional parameter for the module.

	Each module script should expose the class by defining it's name/type alias as
	`ModuleClass`. f4pga tries to access a `ModuleClass` attribute within a package
	when instantiating a module.

	### Module's execution modes

	A module has essentially two execution modes:

	* _mapping_ mode
	* _exec_ mode

	#### _mapping_ mode

	In _mapping_ mode the module is provided with an incomplete configuration which
	includes:
	* `takes` namespace: this maps names of input dependencies to the paths of these
	dependencies
	* `values` namespace: this maps names of variables to the values of those
	variables.

	The module has to provide a dictionary that will provide every output dependency
	that's not _on-demand_ a default path. This is basically a promise that when
	executed in _exec_ mode, the module will produce files for this paths.
	Typically such paths would be derived from a path of one of it's input dependencies.
	This mechanism allows the user to avoid specifying an explicit path for each
	intermediate target.

	It should be noted that variables referring to the output dependencies
	can't be accessed at this stage for the obvious reason as their values are yet
	to be evaluated.

	#### _exec_ mode

	In _exec_ mode the module does the actual work.

	The configuration passed into this mode is full and it includes:

	* `takes` namespace: this maps names of input dependencies to the paths of these
	dependencies
	* `values` namespace: this maps names of variables to the values of those
	variables.
	* `produces` namespace: this maps names of output dependencies to explicit paths.
	This should not be used directly really, but it's useful for
	`ModuleContext.is_output_explicit` method.
	* `outputs` namespace: this maps names of output dependencies to their paths.

	When the module finishes executing in _exec_ mode, all of the dependencies
	described in `outputs` should be present.

	### Module initialization/instantiation

	In the `__init__` method of module's class, the following fields should be
	set:

	* `takes` - a list of symbolic dependency names for dependencies used by the module
	* `produces` - a list of symbolic dependencies names for dependencies produced
	by the module.
	* `values` - a list of names given to the variables used withing the module
	* `prod_meta` - A dictionary which maps product names to descriptions of these
	products. Those entries are optional and can be skipped.

	#### Qualifiers/decorators

	By default the presence of all the dependencies and values is mandatory
	(In case of `produces` that means that the module always has to produce the listed
	dependencies). This can be changed by "decorating" a name in one of the following
	ways:

	* '`?`' _suffix_
	* In `takes` - the dependency is not necessary for the module to execute
	* In `produces` - the dependency may be produced, but it is not guaranteed.
	* In `values` the value is not required for the module to execute.
	Referring to it through `ModuleContext.values.value_name` won't raise an
	exception if the value is not present, instead `None` will be returned.
	* '`!`' _suffix_
	* In `produces` - the dependency is going to be produced only if the user
	provides an explicit path for it.

	Currently it's impossible to combine both '`!`' and '`?`' together. This limitation
	does not have any reason behind it other than the way the qualifier system
	is implemented at the moment. It might be removed in the future.

	## Common modules

	```{toctree}
	fasm
	generic_script_wrapper
	io_rename
	mkdirs
	pack
	place
	place_constraints
	route
	synth
	```