| # Verible |
| |
| The Verible project's main mission is to parse SystemVerilog (IEEE 1800-2017) |
| for a wide variety of applications. |
| |
| It was born out of a need to parse *un-preprocessed* source files, which is |
| suitable for single-file applications like style-linting and formatting. In |
| doing so, it can be adapted to parse *preprocessed* source files, which is what |
| real compilers and toolchains require. |
| |
| The spirit of the project is that no-one should ever have to develop a |
| SystemVerilog parser for their own application, because developing a |
| standard-compliant parser is an enormous task due to the syntactic complexity of |
| the language. |
| |
| A lesser (but notable) objective is that the language-agnostic components of |
| Verible be usable for rapidly developing language support tools for other |
| languages. |
| |
| ## Build |
| |
| Verible's code base is written in C++. |
| |
| To build, you need the [bazel] build system and a C++11 compatible compiler |
| (e.g. >= g++-7; Using clang currently fails to compile the m4 dependency). |
| |
| ```bash |
| # Build all tools and libraries |
| bazel build -c opt //... |
| ``` |
| |
| You can access the generated artifacts under `bazel-bin/`. For instance the |
| syntax checker will be at |
| `bazel-bin/verilog/tools/syntax/verible-verilog-syntax` (corresponding to the |
| target name `//verilog/tools/syntax:verible-verilog-syntax`). |
| |
| ### Installation |
| |
| For simple installation, we provide regular [binary releases]. |
| |
| If you prefer to install the binaries on your workstation yourself, use the |
| `:install` target which you invoke with `bazel run`. |
| |
| The following builds and installs the `verible-verilog-format`, |
| `verible-verilog-lint` and `verible-verilog-syntax` binaries in the provided |
| location: |
| |
| ```bash |
| # In your home directory |
| bazel run :install -c opt -- ~/bin |
| |
| # For a system directory that requires root-access, call with -s option. |
| # (Do _not_ run bazel with sudo.) |
| bazel run :install -c opt -- -s /usr/local/bin |
| ``` |
| |
| ### Test |
| |
| To run the tests in [bazel]: |
| |
| ```bash |
| # Run all tests |
| bazel test //... |
| ``` |
| |
| ## Mailing Lists |
| |
| Join the Verible community! |
| |
| * Developers: verible-dev@googlegroups.com |
| ([join](https://groups.google.com/forum/#!forum/verible-dev/join)) |
| * Users: verible-users@googlegroups.com |
| ([join](https://groups.google.com/forum/#!forum/verible-users/join)) |
| |
| ## SystemVerilog Support |
| |
| ### Parser |
| |
| The lexer is implemented using GNU Flex, and the parser is implemented using GNU |
| Bison (yacc). To parse un-preprocessed input, preprocessing constructs had to be |
| handled explicitly in the parser, and are permitted in limited places. The |
| grammatic rules in the yacc input are approximate and permissive; it may accept |
| some syntactically invalid constructs. The priority is to *accept* all |
| syntactically *valid* SystemVerilog, as defined in the [SV-LRM]. Status: As of |
| 2019, it accepts the vast majority of SystemVerilog (IEEE 1800-2017), but there |
| is work ahead to reach 100%. |
| |
| The lexer and parser are *decoupled*, which means that the lexer can be used |
| standalone to tokenize text, and the parser is adapted to accept tokens from |
| sources other than the direct use of the lexer. This separation enables the |
| insertion of different passes between the lexer and parser, such as integrated |
| preprocessing, and context-based lexical disambiguation (with arbitrary |
| lookahead) where required by the language. |
| |
| The parser can be tested as a standalone program, |
| `//verilog/tools/syntax:verible-verilog-syntax`. |
| |
| ``` |
| verible-verilog-syntax: usage: verible-verilog-syntax [options] <file> [<file>...] |
| |
| Flags from verilog/tools/syntax/verilog_syntax.cc: |
| -printrawtokens (Prints all lexed tokens, including filtered ones.); |
| default: false; |
| -printtokens (Prints all lexed and filtered tokens); default: false; |
| -printtree (Whether or not to print the tree); default: false; |
| -verifytree (Verifies that all tokens are parsed into tree, prints unmatched |
| tokens); default: false; |
| |
| Try --helpfull to get a list of all flags. |
| ``` |
| |
| #### Selecting Parsing Mode {#parsing-mode} |
| |
| If your source file is not a standalone Verilog description as defined by the |
| LRM, (e.g. a body snippet of a module, class, task, or function without the |
| header or end) the parser can use an alternate parsing mode by using a comment |
| directive before the first (non-comment) token. |
| |
| ```verilog |
| // This file is `included inside a module definition body. |
| // verilog_syntax: parse-as-module-body |
| wire x; |
| initial begin |
| x = 0; |
| end |
| ``` |
| |
| The `verilog_syntax` directive tells the parser that the contents of this file |
| are to be treated as if they are inside a module declaration. |
| |
| Available parser selection modes include: |
| |
| * `parse-as-statements`: Text contains only statements (e.g. inside function, |
| always, ...) |
| * `parse-as-expression`: Text occurs inside an expression. |
| * `parse-as-module-body`: Text occurs inside a module definition. |
| * `parse-as-class-body`: Text occurs inside a class definition. |
| * `parse-as-package-body`: Text occurs inside a package definition. |
| |
| #### Token Stream |
| |
| The lexer partitions a text buffer into a sequence of tokens with annotations |
| (token stream). `verible-verilog-syntax --printtokens` shows the tokens that |
| feeds into the parser, and `--printrawtokens` to shows all tokens including |
| whitespaces, comments, and attributes. |
| |
| For example, the following code: |
| |
| ``` |
| // This is module foo. |
| module foo(input a, b, output z); |
| endmodule : foo |
| ``` |
| |
| produces the following tokens (shown using `--printrawtokens`): |
| |
| ``` |
| All lexed tokens: |
| All lexed tokens: |
| (#"// end of line comment" @0-22: "// This is module foo.") |
| (#"<<\\n>>" @22-23: " |
| ") |
| (#"module" @23-29: "module") |
| (#"<<space>>" @29-30: " ") |
| (#SymbolIdentifier @30-33: "foo") |
| (#'(' @33-34: "(") |
| (#"input" @34-39: "input") |
| (#"<<space>>" @39-40: " ") |
| (#SymbolIdentifier @40-41: "a") |
| (#',' @41-42: ",") |
| (#"<<space>>" @42-43: " ") |
| (#SymbolIdentifier @43-44: "b") |
| (#',' @44-45: ",") |
| (#"<<space>>" @45-46: " ") |
| (#"output" @46-52: "output") |
| (#"<<space>>" @52-53: " ") |
| (#SymbolIdentifier @53-54: "z") |
| (#')' @54-55: ")") |
| (#';' @55-56: ";") |
| (#"<<\\n>>" @56-57: " |
| ") |
| (#"endmodule" @57-66: "endmodule") |
| (#"<<space>>" @66-67: " ") |
| (#':' @67-68: ":") |
| (#"<<space>>" @68-69: " ") |
| (#SymbolIdentifier @69-72: "foo") |
| (#"<<\\n>>" @72-73: " |
| ") |
| (#"<<\\n>>" @73-74: " |
| ") |
| (#$end @74-74: "") |
| ``` |
| |
| The token names (after `#`) correspond to description strings in the yacc |
| grammar file; keywords are shown the same as the text they match. Byte offsets |
| are shown as the range that follows '@'. The raw, unfiltered token stream is |
| lossless with respect to the original input text. |
| |
| #### Concrete Syntax Tree |
| |
| The parser produces a concrete syntax tree (CST), which can be diagnosed with |
| `verible-verilog-syntax --printtree`. |
| |
| For example, the following code (same as above): |
| |
| ``` |
| // This is module foo. |
| module foo(input a, b, output z); |
| endmodule : foo |
| ``` |
| |
| produces this CST (rendered by `verible-verilog-syntax --printtree`): |
| |
| ``` |
| Parse Tree: |
| Node @0 (tag: kDescriptionList) { |
| Node @0 (tag: kModuleDeclaration) { |
| Node @0 (tag: kModuleHeader) { |
| Leaf @0 (#"module" @23-29: "module") |
| Leaf @2 (#SymbolIdentifier @30-33: "foo") |
| Node @5 (tag: kParenGroup) { |
| Leaf @0 (#'(' @33-34: "(") |
| Node @1 (tag: kPortDeclarationList) { |
| Node @0 (tag: kPortDeclaration) { |
| Leaf @0 (#"input" @34-39: "input") |
| Node @2 (tag: kDataType) { |
| } |
| Node @3 (tag: kUnqualifiedId) { |
| Leaf @0 (#SymbolIdentifier @40-41: "a") |
| } |
| Node @4 (tag: kUnpackedDimensions) { |
| } |
| } |
| Leaf @1 (#',' @41-42: ",") |
| Node @2 (tag: kPort) { |
| Node @0 (tag: kPortReference) { |
| Node @0 (tag: kUnqualifiedId) { |
| Leaf @0 (#SymbolIdentifier @43-44: "b") |
| } |
| } |
| } |
| Leaf @3 (#',' @44-45: ",") |
| Node @4 (tag: kPortDeclaration) { |
| Leaf @0 (#"output" @46-52: "output") |
| Node @2 (tag: kDataType) { |
| } |
| Node @3 (tag: kUnqualifiedId) { |
| Leaf @0 (#SymbolIdentifier @53-54: "z") |
| } |
| Node @4 (tag: kUnpackedDimensions) { |
| } |
| } |
| } |
| Leaf @2 (#')' @54-55: ")") |
| } |
| Leaf @7 (#';' @55-56: ";") |
| } |
| Node @1 (tag: kModuleItemList) { |
| } |
| Leaf @2 (#"endmodule" @57-66: "endmodule") |
| Node @3 (tag: kLabel) { |
| Leaf @0 (#':' @67-68: ":") |
| Leaf @1 (#SymbolIdentifier @69-72: "foo") |
| } |
| } |
| } |
| ``` |
| |
| The `N` in `Node @N` or `Leaf @N` refers to the child rank of that node/leaf |
| with respect to its immediate parent node, starting at 0. `nullptr` nodes are |
| skipped and will look like gaps in the rank sequence. |
| |
| Nodes of the CST may link to other nodes or leaves (which contain tokens). The |
| nodes are tagged with language-specific enumerations. Each leaf encapsulates a |
| token and is shown with its corresponding byte-offsets in the original text (as |
| `@left-right`). Null nodes are not shown. |
| |
| The exact structure of the SystemVerilog CST is fragile, and should not be |
| considered stable; at any time, node enumerations can be created or removed, and |
| subtree structures can be re-shaped. In the above example, `kModuleHeader` is an |
| implementation detail of a module definition's composition, and doesn't map |
| directly to a named grammar construct in the [SV-LRM]. The `verilog/CST` library |
| provides functions that abstract away internal structure. |
| |
| #### Abstract Syntax Tree |
| |
| An abstract syntax tree (AST) does not exist yet, but is planned. |
| |
| ### Style Linter |
| |
| The style linter is an analysis tool that identifies constructs or patterns |
| deemed undesirable according to a style guide. The main goal is to relieve |
| humans the burden of reviewing code for style compliance. Many lint rules use |
| syntax tree pattern matching to find style violations. The tool provides |
| generating a [built-in lint documentation][lint-rule-list] with details to each |
| rule; you can generate it by invoking the linter with `--generate_markdown` or |
| use the build script: |
| |
| ```bash |
| # Generating documentation |
| bazel build :lint_doc |
| |
| # It will be generated into |
| bazel-bin/documentation_verible_lint_rules.md |
| ``` |
| |
| The linter tool is available as `//verilog/tools/lint:verible-verilog-lint`. |
| |
| ``` |
| verible-verilog-lint: usage: verible-verilog-lint [options] <file> [<file>...] |
| |
| Flags from verilog/analysis/verilog_linter.cc: |
| --rules (Comma-separated of lint rules to enable. No prefix or a '+' prefix |
| enables it, '-' disable it. Configuration values for each rules placed |
| after '=' character.); default: ; |
| --rules_config (Path to lint rules configuration file.); |
| default: ".rules.verible_lint"; |
| --ruleset ([default|all|none], the base set of rules used by linter); |
| default: default; |
| |
| Flags from verilog/tools/lint/verilog_lint.cc: |
| -generate_markdown (If true, print the description of every rule formatted |
| for the markdown and exit immediately. Intended for the output to be |
| written to a snippet of markdown.); default: false; |
| -help_rules ([all|<rule-name>], print the description of one rule/all rules |
| and exit immediately.); default: ""; |
| -lint_fatal (If true, exit nonzero if linter finds violations.); |
| default: false; |
| -parse_fatal (If true, exit nonzero if there are any syntax errors.); |
| default: false; |
| |
| Try --helpfull to get a list of all flags. |
| ``` |
| |
| #### Rule configuration |
| |
| The `--rules` flag allows to enable/disable rules as well as pass configuration |
| to rules that accept them. It accepts a comma-separated list |
| [rule names][lint-rule-list]. If prefixed with a `-` (minus), the rule is |
| disabled. No prefix or a '+' (plus) prefix enables the rule. An optional |
| configuration can be passed after an `=` assignment. |
| |
| The following example enables the |
| [`enum-name-style`][lint-rule-list_enum-name-style] rule, enables and configures |
| the [`line-length`][lint-rule-list_line-length] rule (80 characters length) and |
| disables the [`no-tabs`][lint-rule-list_no-tabs] rule. |
| |
| ``` |
| verible-verilog-lint --rules=enum-name-style,+line-length=length:80,-no-tabs ... |
| ``` |
| |
| Additionally, the `--rules_config` flag can be used to read configuration stored |
| in a file. The syntax is the same as above, except the rules can be also |
| separated with the newline character. |
| |
| #### Waiving Lint Violations {#lint-waiver} |
| |
| In the rare circumstance where a line needs to be waived from a particular lint |
| rule, you can use the following waiver comment: |
| |
| ``` |
| // This example waives the line after the waiver. |
| // verilog_lint: waive rule-name |
| The next non-comment line like this one is waived. |
| |
| // This example waives the same line as the waiver. |
| This line is waived. // verilog_lint: waive rule-name |
| |
| // This example shows accumulation of waivers over multiple lines. |
| // verilog_lint: waive rule-name-1 |
| // verilog_lint: waive rule-name-2 |
| // Other comments, possibly waivers for other tools. |
| This line will be waived for both rule-name-1 and rule-name-2. |
| |
| // This example shows how to waive an entire range of lines. |
| // verilog_lint: waive-start rule-X |
| ... |
| All lines in between will be waived for rule-X |
| ... |
| // verilog_lint: waive-stop rule-X |
| ``` |
| |
| Another option is to use a configuration file with the `--waiver_files` flag. |
| The flag accepts a single file or a list of files (comma-separated). Specifying |
| multiple files is equivalent to concatenating the files in order of appearance. |
| By default, the rules are applied to all files, but with `--location` you can to |
| choose to only apply them to filenames matching the location regexp. |
| |
| The format of this file is as follows: |
| |
| ``` |
| waive --rule=rule-name-1 --line=10 |
| waive --rule=rule-name-2 --line=5:10 |
| waive --rule=rule-name-3 --regex="^\s*abc$" |
| waive --rule=rule-name-4 --line=42 --location=".*some_file.*" |
| ``` |
| |
| The `--line` flag can be used to specify a single line to apply the waiver to or |
| a line range (separated with the `:` character). Additionally the `--regex` flag |
| can be used to dynamically match lines on which a given rule has to be waived. |
| This is especially useful for projects where some of the files are |
| auto-generated. |
| |
| The name of the rule to waive is at the end of each diagnostic message in `[]`. |
| |
| Syntax errors cannot be waived. A common source of syntax errors is if the file |
| is not a standalone Verilog program as defined by the LRM, e.g. a body snippet |
| of a module, class, task, or function. In such cases, the syntax parser can be |
| directed to treat the code as a snippet by selecting a |
| [parsing mode](#parsing-mode). |
| |
| ### Formatter |
| |
| The formatter is a transformative tool that manages whitespace in accordance |
| with a particular style. The main goal is to relieve humans of having to |
| manually manage whitespace, wrapping, and indentation, and to provide a tool |
| that can be integrated into any editor to enable editor-independent consistency. |
| |
| The formatter tool is available as |
| `//verilog/tools/formatter:verible-verilog-format`. |
| |
| ``` |
| verible-verilog-format: usage: verible-verilog-format [options] <file> |
| To pipe from stdin, use '-' as <file>. |
| |
| Flags from verilog/tools/formatter/verilog_format.cc: |
| --failsafe_success (If true, always exit with 0 status, even if there were |
| input errors or internal errors. In all error conditions, the original |
| text is always preserved. This is useful in deploying services where |
| fail-safe behaviors should be considered a success.); default: true; |
| --inplace (If true, overwrite the input file on successful conditions.); |
| default: false; |
| --lines (Specific lines to format, 1-based, comma-separated, inclusive N-M |
| ranges, N is short for N-N. By default, left unspecified, all lines are |
| enabled for formatting. (repeatable, cumulative)); default: ; |
| --max_search_states (Limits the number of search states explored during line |
| wrap optimization.); default: 100000; |
| --show_equally_optimal_wrappings (If true, print when multiple optimal |
| solutions are found (stderr), but continue to operate normally.); |
| default: false; |
| --show_inter_token_info (If true, along with show_token_partition_tree, |
| include inter-token information such as spacing and break penalties.); |
| default: false; |
| --show_largest_token_partitions (If > 0, print token partitioning and then |
| exit without formatting output.); default: 0; |
| --show_token_partition_tree (If true, print diagnostics after token |
| partitioning and then exit without formatting output.); default: false; |
| --stdin_name (When using '-' to read from stdin, this gives an alternate |
| name for diagnostic purposes. Otherwise this is ignored.); |
| default: "<stdin>"; |
| --verify_convergence (If true, and not incrementally formatting with |
| --lines, verify that re-formatting the formatted output yields no further |
| changes, i.e. formatting is convergent.); default: true; |
| |
| Try --helpfull to get a list of all flags. |
| ``` |
| |
| #### Disabling Formatting {#disable-formatting} |
| |
| When you want to exempt a range of text from formatting, write: |
| |
| ``` |
| // verilog_format: off |
| ... untouched code ... |
| // verilog_format: on |
| ``` |
| |
| or |
| |
| ``` |
| /* verilog_format: off */ |
| ... untouched code ... |
| /* verilog_format: on */ |
| ``` |
| |
| As a good practice, include a reason why you choose to disable a section. |
| |
| ``` |
| // verilog_format: off // my alignment is prettier than the tool's |
| |
| // verilog_format: off // issue #N: working around. |
| ``` |
| |
| These directives take precedence over `--lines` specifications. |
| |
| #### Interactive Formatting |
| |
| The formatter is ever a work-in-progress and may not always behave the way a |
| user would like it to behave. If you wish to review changes introduced by the |
| formatter and apply them interactively, we've provided a helper script (that is |
| actually formatter-agnostic) for the job. |
| |
| ``` |
| verible-transform-interactive.sh -- verible-verilog-format -- files... |
| ``` |
| |
| This will run the formatter on the affected files without modifying them |
| initially, and pass the `diff`-generated patch into a tool |
| (`verible-patch-tool`) that interactively applies patch hunks. |
| |
| Answer the \[y/n\] prompts. |
| |
| The original files will be modified in-place with the elected changes. |
| |
| For convenience, one could create an alias like: |
| |
| ``` |
| alias verilog-format-interactive='verible-transform-interactive.sh -- verible-verilog-format --' |
| ``` |
| |
| #### Incremental Formatting |
| |
| If you wish to format only changed lines (a.k.a. incremental or partial |
| formatting), the following tools are provided to assist. |
| |
| ##### Git users |
| |
| `git-verible-verilog-format.sh` (installed along with `verible-verilog-format`) |
| can be run from within any subdirectory of a Git project. It automatically |
| detects new and changed lines, and generates the `--lines` flag for each |
| modified file in your workspace. |
| |
| From `--help`: |
| |
| ``` |
| git-verible-verilog-format.sh: |
| Performs incremental file formatting (verible-verilog-format) based on current diffs. |
| New files explicitly git-add-ed by the user are wholly formatted. |
| |
| Actions: |
| 1) Runs 'git add -u' to stage currently modified files to the index. |
| To format new files (wholly), 'git add' those before calling this script. |
| 2) Runs 'git diff -u --cached' to generate a unified diff. |
| 3) Diff is scanned to determine added or modified lines in each file. |
| 4) Invokes 'verible-verilog-format --inplace' on all touched or new Verilog files, |
| but does not 'git add' so that the changes may be examined and tested. |
| Formatting can be easily undone with: |
| 'git diff | git apply --reverse -'. |
| |
| usage: git-verible-verilog-format.sh [script options] [-- [verilog_format options]] |
| (no positional arguments) |
| Run from anywhere inside a git project tree. |
| |
| script options: (options with arguments can be: --flag=VALUE or --flag VALUE) |
| --help | -h : print help and exit |
| --verbose | -v : execute verbosely |
| --dry-run : stop before running formatter, and print formatting commands |
| --formatter TOOL : path to verilog_format binary |
| [using: /usr/local/bin/verible-verilog-format] |
| -- : stops option processing, and forwards remaining args as flags to the |
| underlying --formatter tool. |
| ``` |
| |
| #### Incremental Interactive Formatting |
| |
| In your locally modified client (p4, git) run: |
| |
| ```shell |
| verible-verilog-format-changed-lines-interactive.sh |
| ``` |
| |
| and follow the prompts. |
| |
| > NOTE: git and hg users can pass in a different base revision to diff against: |
| > |
| > ```shell |
| > # Diff against the previous revision in hg |
| > verible-verilog-format-changed-lines-interactive.sh --rev .^ |
| > # Diff against upstream mainline in git |
| > verible-verilog-format-changed-lines-interactive.sh --rev origin/main |
| > ``` |
| |
| #### Aligned Formatting |
| |
| There are several sections of code that are eligible for aligned formatting. In |
| each of these contexts, the user has some control over the alignment behavior. |
| Without alignment, the default behavior is to flush-left, which respects |
| indentation and minimum inter-token spacing constraints. |
| |
| Yet, alignment is not always desired. So how does the formatter know whether or |
| not the user intended for alignment? It can only examine the original code and |
| make some judgment. |
| |
| Consider the following example (module formal parameters): |
| |
| ``` |
| module m #( |
| int W, |
| type T |
| ); |
| ... |
| endmodule |
| ``` |
| |
| With alignment, this formats to: |
| |
| ``` |
| module m #( |
| int W, |
| type T |
| ); |
| ... |
| endmodule |
| ``` |
| |
| When the spacing difference between between aligned and flushed-left formatting |
| is "sufficiently small", the formatter will align because the small difference |
| has low risk of compromising readability. |
| |
| In contrast, aligning the following example _could_ be considered less readable. |
| |
| original: |
| |
| ``` |
| module m #( |
| int W, |
| some_long_name T |
| ); |
| ... |
| endmodule |
| ``` |
| |
| aligned: |
| |
| ``` |
| module m #( |
| int W, |
| some_long_name T |
| ); |
| ... |
| endmodule |
| ``` |
| |
| To detect the above condition, the formatter examines the number of _excess_ |
| spaces (spacing errors) within an alignment group (original vs. flushed-left). |
| If that value is lower than a threshold, the formatter infers that the author |
| intended flush-left formatting. |
| |
| To induce alignment, the author needs to _inject_ four excess spaces _between_ |
| any two tokens in any row in the aligned section, not before first tokens (which |
| fall under indentation, not alignment). In this example, spaces are deliberately |
| injected before `W` (but after `W` would work too): |
| |
| ``` |
| module m #( |
| int W, |
| some_long_name T |
| ); |
| ... |
| endmodule |
| ``` |
| |
| formatted with alignment: |
| |
| ``` |
| module m #( |
| int W, |
| some_long_name T |
| ); |
| ... |
| endmodule |
| ``` |
| |
| This also implies that previously aligned code will most likely remain aligned. |
| |
| Finally, if none of the above conditions hold, the formatter will leave the |
| original code as-is, preserving all pre-existing spaces. |
| |
| ### Lexical Diff |
| |
| `verible-verilog-diff` compares two input files for equivalence, where |
| equivalence is determined by the `--mode` of operation. |
| |
| ``` |
| verible-verilog-diff: usage: verible-verilog-diff [options] file1 file2 |
| |
| Flags from verilog/tools/diff/verilog_diff.cc: |
| --mode (Defines difference functions. |
| format: ignore whitespaces, compare token texts |
| This is useful for verifying formatter (e.g. verilog_format) output. |
| obfuscate: preserve whitespaces, compare token texts' lengths only. |
| This is useful for verifying verilog_obfuscate output. |
| ); default: format; |
| ``` |
| |
| Exit codes: |
| |
| * 0: files are equivalent |
| * 1: files differ, or contain lexical errors |
| * 2: error reading file |
| |
| ### Code Obfuscator |
| |
| `verible-verilog-obfuscate` transforms Verilog code by replacing identifiers |
| with obfuscated names of equal length, and preserving all other text, including |
| spaces. Output is written to stdout. The resulting file size is the same as the |
| original. |
| |
| This is useful for preparing potentially sensitive test cases with tool vendors. |
| |
| ``` |
| verible-verilog-obfuscate: usage: verible-verilog-obfuscate [options] < original_file > output |
| |
| verible-verilog-obfuscate mangles Verilog code by changing identifiers. |
| All whitespaces and identifier lengths are preserved. |
| Output is written to stdout. |
| |
| Flags from verilog/tools/obfuscator/verilog_obfuscate.cc: |
| --decode (If true, when used with --load_map, apply the translation |
| dictionary in reverse to de-obfuscate the source code, and do not |
| obfuscate any unseen identifiers. There is no need to --save_map with this |
| option, because no new substitutions are established.); default: false; |
| --load_map (If provided, pre-load an existing translation dictionary |
| (written by --save_map). This is useful for applying pre-existing |
| transforms.); default: ""; |
| --save_map (If provided, save the translation to a dictionary for reuse in a |
| future obfuscation with --load_map.); default: ""; |
| ``` |
| |
| ### Preprocessor |
| |
| `verible-verilog-preprocessor` is a collection of preprocessor-like tools, (but |
| does not include a fully-featured Verilog preprocessor yet.) |
| |
| #### Strip Comments |
| |
| Removing comments can be useful for preparing to obfuscate code for sharing with |
| EDA tool vendors. |
| |
| ```shell |
| $ verible-verilog-preprocessor strip-comments FILE |
| ``` |
| |
| This strips comments from a Verilog/SystemVerilog source file, _including_ |
| comments nested inside macro definition bodies and macro call arguments. |
| |
| Comments are actually replaced by an equal number of spaces (and newlines) to |
| preserve the byte offsets and line ranges of the original text. This makes it |
| easier to trace back diagnostics on obfuscated code back to the original code. |
| |
| ## Indexing Verible C++ Code using Kythe |
| |
| The steps mentioned here can be generalized for indexing Bazel-based projects. |
| |
| More information about indexing Bazel-based projects using Kythe |
| [here](https://github.com/kythe/kythe/tree/master/kythe/cxx/extractor#bazel-c-extractor). |
| |
| ### Initializing Kythe |
| |
| Download the latest Kythe release from https://github.com/kythe/kythe/releases |
| and then unpack it for a snapshot of Kythe’s toolset. |
| |
| ```bash |
| tar xzf kythe-v*.tar.gz |
| rm -rf /opt/kythe |
| mv kythe-v*/ /opt/kythe |
| ``` |
| |
| More information can be found |
| [here](https://github.com/kythe/kythe#getting-started). |
| |
| Clone Kythe from [here](https://github.com/kythe/kythe). Then from within the |
| Kythe clone, build the web frontend and copy its files into /opt/kythe/ |
| |
| ```bash |
| bazel build //kythe/web/ui |
| mkdir -p /opt/kythe/web/ui |
| cp -r bazel-bin/kythe/web/ui/resources/public/* /opt/kythe/web/ui |
| cp -r kythe/web/ui/resources/public/* /opt/kythe/web/ui |
| chmod -R 755 /opt/kythe/web/ui |
| ``` |
| |
| More information can be found |
| [here](https://github.com/google/haskell-indexer#building-from-source). |
| |
| ### Extracting Compilations for Verible |
| |
| #### VNames.json File |
| |
| `vnames.json` file is used for renaming certain filepaths during extraction, but |
| renaming isn’t needed here but you can add the renaming you find suitable. |
| |
| #### Run the extractor |
| |
| ```bash |
| # run on all targets |
| bazel test --experimental_action_listener=:extract_cxx //... |
| |
| # run on specific target (e.g. some cc_binary or cc_library) |
| bazel test --experimental_action_listener=:extract_cxx //verilog/analysis:default_rules |
| ``` |
| |
| Extracted kzip files will be in |
| bazel-out/local-fastbuild/extra_actions/extractor folder. One kzip file per |
| target. |
| |
| ```bash |
| find -L bazel-out -name '*cxx.kzip' |
| ``` |
| |
| More information can be found |
| [here](https://github.com/kythe/kythe/tree/master/kythe/cxx/extractor). |
| |
| ### Indexing Extracted Kzip Files and Generating GraphStore |
| |
| For extracting the indexing facts from extracted `kzip` files, run the following |
| command to apply the indexing for each `kzip` file and generate the result into |
| `kythe graphstore`. |
| |
| ```bash |
| for i in $(find -L bazel-out -name '*cxx.kzip'); do |
| # Indexing C++ compilations |
| /opt/kythe/indexers/cxx_indexer --ignore_unimplemented "$i" > entries |
| |
| # Write entry stream into a GraphStore |
| /opt/kythe/tools/write_entries --graphstore leveldb:.kythe_graphstore < entries |
| done |
| ``` |
| |
| ### Generating Serving Tables |
| |
| ```bash |
| # Generate corresponding serving tables |
| /opt/kythe/tools/write_tables --graphstore .kythe_graphstore --out .kythe_serving |
| ``` |
| |
| ### Visualizing Cross-References |
| |
| Run Kythe's `UI` to visualize cross-reference and code navigation. |
| |
| ```bash |
| # --listen localhost:8080 allows access from only this machine; change to |
| # --listen 0.0.0.0:8080 to allow access from any machine |
| /opt/kythe/tools/http_server \ |
| --public_resources /opt/kythe/web/ui \ |
| --listen localhost:8080 \ |
| --serving_table .kythe_serving |
| ``` |
| |
| ### Future Intent |
| |
| The Verible team is interested in exploring how it can help other tool |
| developers in providing a SystemVerilog front end, for example, emitting an |
| abstract syntax tree (AST). If you are interested in collaborating, contact us. |
| |
| [bazel]: https://bazel.build/ |
| [SV-LRM]: https://ieeexplore.ieee.org/document/8299595 |
| [lint-rule-list]: https://google.github.io/verible/lint.html |
| [lint-rule-list_enum-name-style]: https://google.github.io/verible/lint.html#enum-name-style |
| [lint-rule-list_line-length]: https://google.github.io/verible/lint.html#line-length |
| [lint-rule-list_no-tabs]: https://google.github.io/verible/lint.html#no-tabs |
| [binary releases]: https://github.com/google/verible/releases |