Update the preprocessor command to 'preprocess'. Fix the unit test to actually fail on failures (there were a bunch of `exit 0` in there.) Update documentation slightly and include in README. Signed-off-by: Henner Zeller <hzeller@google.com>
diff --git a/verilog/tools/preprocessor/README.md b/verilog/tools/preprocessor/README.md index 4235dd0..321b967 100644 --- a/verilog/tools/preprocessor/README.md +++ b/verilog/tools/preprocessor/README.md
@@ -1,24 +1,88 @@ # SystemVerilog (Pseudo) Preprocessor <!--* -freshness: { owner: 'hzeller' reviewed: '2020-10-04' } +freshness: { owner: 'hzeller' reviewed: '2022-11-16' } *--> `verible-verilog-preprocessor` is a collection of preprocessor-like tools, (but does not include a fully-featured Verilog preprocessor yet.) -## Strip Comments {#strip-comments} +```shell +$ verible-verilog-preprocessor help +available commands: + generate-variants + preprocess + strip-comments +``` + +## Preprocess + +Preprocess the given file(s) and expand macros and includes. + +#### Synopsis +``` +verible-verilog-preprocessor preprocess [define-include-flags] file [file...] +``` + +#### Inputs + Accepts one or more Verilog or SystemVerilog source files to preprocess. + Each one of them will be prepropcessed independently which means that + declaration scopes will end by the end of each file, and won't be seen from + other files (so multiple files will _not_ be treated as compilation unit). + The `+define+` and `+include+` directives on the commandline are honored by + the preprocessor. + +#### Output + The preprocessed files content (same contents with directives interpreted) + will be written to stdout, concatenated. + +## Strip Comments Removing comments can be useful for preparing to obfuscate code for sharing with EDA tool vendors. -```shell -$ verible-verilog-preprocessor strip-comments FILE -``` - -This strips comments from a Verilog/SystemVerilog source file, _including_ -comments nested inside macro definition bodies and macro call arguments. +The `strip-comments` command strips comments from a Verilog/SystemVerilog +source file, _including_ comments nested inside macro definition bodies and +macro call arguments. Comments are actually replaced by an equal number of spaces (and newlines) to preserve the byte offsets and line ranges of the original text. This makes it easier to trace back diagnostics on obfuscated code back to the original code. + +#### Synopsis +``` +verible-verilog-preprocessor strip-comments file [replacement-char] +``` + +#### Inputs + `file` is a Verilog or SystemVerilog source file (Use `-` to read from stdin). + The optional `replacement-char` is a character to replace comments with. + If not given, or given as a single space character, the comment contents and + delimiters are replaced with spaces. + If an empty string, the comment contents and delimiters are deleted. Newlines + are not deleted. + If a single character, the comment contents are replaced with the character. + +#### Output + Writes to stdout with contents of original file with `//` and `/**/` + comments removed. + + +## Generate Variants + +Generates a number of variants of the input file by selectively switching +on/off available macros. + +#### Synopsis +``` +verible-verilog-preprocessor generate-variants file [-limit_variants number] +``` + + +#### Inputs + `file` is a Verilog or SystemVerilog source file. + `-limit_variants` flag limits variants to 'number' (20 by default). + +#### Output + Output to stdout. Generates every possible variant of `ifdef + blocks considering the conditional directives.
diff --git a/verilog/tools/preprocessor/verilog_preprocessor.cc b/verilog/tools/preprocessor/verilog_preprocessor.cc index aebc4d0..a340d88 100644 --- a/verilog/tools/preprocessor/verilog_preprocessor.cc +++ b/verilog/tools/preprocessor/verilog_preprocessor.cc
@@ -149,10 +149,8 @@ return absl::InvalidArgumentError("ERROR: Missing file argument."); } for (const absl::string_view source_file : files) { - message_stream << source_file << ":\n"; RETURN_IF_ERROR(PreprocessSingleFile(source_file, preprocessing_info, outs, message_stream)); - outs << '\n'; } return absl::OkStatus(); } @@ -217,6 +215,21 @@ } static const std::pair<absl::string_view, SubcommandEntry> kCommands[] = { + {"preprocess", + {&MultipleCU, + R"(preprocess [define-include-flags] file [file...] +Inputs: + Accepts one or more Verilog or SystemVerilog source files to preprocess. + Each one of them will be prepropcessed independently which means that + declaration scopes will end by the end of each file, and won't be seen from + other files (so multiple files will _not_ be treated as compilation unit). + The +define+ and +include+ directives on the commandline are honored by + the preprocessor. +Output: (stdout) + The preprocessed files content (same contents with directives interpreted) + will be written to stdout, concatenated. +)"}}, + {"strip-comments", {&StripComments, R"(strip-comments file [replacement-char] @@ -232,17 +245,7 @@ Output: (stdout) Contents of original file with // and /**/ comments removed. )"}}, - {"multiple-compilation-unit", - {&MultipleCU, - R"(multiple-compilation-unit file [more_files] -Inputs: - 'file' is a Verilog or SystemVerilog source file. - There can be multiple SystemVerilog source files. - Each one of them will be prepropcessed separatly which means that declarations - scoopes will end by the end of each file, and won't be seen from other files. -Output: (stdout) - The preprocessed files content (same contents with directives interpreted). -)"}}, + {"generate-variants", {&GenerateVariants, R"(generate-variants file [-limit_variants number] @@ -250,7 +253,8 @@ 'file' is a Verilog or SystemVerilog source file. '-limit_variants' flag limits variants to 'number' (20 by default). Output: (stdout) - Generates every possible variant considering the conditional directives. + Generates every possible variant of `ifdef blocks considering the + conditional directives. )"}}, // TODO(karimtera): We can add another argument to `generate-variants`, // Which allows us to set some defines, as if we are only interested
diff --git a/verilog/tools/preprocessor/verilog_preprocessor_test.sh b/verilog/tools/preprocessor/verilog_preprocessor_test.sh index 360aaae..33aa429 100755 --- a/verilog/tools/preprocessor/verilog_preprocessor_test.sh +++ b/verilog/tools/preprocessor/verilog_preprocessor_test.sh
@@ -32,7 +32,7 @@ preprocessor="$(rlocation ${TEST_WORKSPACE}/$1)" ################################################################################ -echo "=== Test no command." +echo "=== Line:${LINENO} Test no command." "$preprocessor" > "$MY_OUTPUT_FILE" 2>&1 @@ -43,7 +43,7 @@ } ################################################################################ -echo "=== Test invalid command." +echo "=== Line:${LINENO} Test invalid command." "$preprocessor" bad-subcommand > "$MY_OUTPUT_FILE" 2>&1 @@ -54,7 +54,7 @@ } ################################################################################ -echo "=== Test the 'help' command." +echo "=== Line:${LINENO} Test the 'help' command." "$preprocessor" help > "$MY_OUTPUT_FILE" 2>&1 @@ -71,7 +71,7 @@ } ################################################################################ -echo "=== Test 'help' on a specific command." +echo "=== Line:${LINENO} Test 'help' on a specific command." "$preprocessor" help help > "$MY_OUTPUT_FILE" 2>&1 @@ -82,7 +82,7 @@ } ################################################################################ -echo "=== Test strip-comments: missing file argument." +echo "=== Line:${LINENO} Test strip-comments: missing file argument." "$preprocessor" strip-comments > /dev/null @@ -93,7 +93,7 @@ } ################################################################################ -echo "=== Test strip-comments: white out comments" +echo "=== Line:${LINENO} Test strip-comments: white out comments" cat > "$MY_INPUT_FILE" <<EOF // fake Verilog file. @@ -146,7 +146,7 @@ } ################################################################################ -echo "=== Test strip-comments: delete comments" +echo "=== Line:${LINENO} Test strip-comments: delete comments" cat > "$MY_INPUT_FILE" <<EOF // fake Verilog file. @@ -183,7 +183,7 @@ } ################################################################################ -echo "=== Test strip-comments: replace comments" +echo "=== Line:${LINENO} Test strip-comments: replace comments" cat > "$MY_INPUT_FILE" <<EOF // fake Verilog file. @@ -223,7 +223,7 @@ ################################################################################ -echo "=== Test strip-comments: on a lexically invalid source file" +echo "=== Line:${LINENO} Test strip-comments: on a lexically invalid source file" cat > "$MY_INPUT_FILE" <<EOF module 1m; /* comment */ endmodule @@ -238,7 +238,7 @@ status="$?" [[ $status == 0 ]] || { "Expected exit code 0, but got $status" - exit 0 + exit 1 } diff --strip-trailing-cr -u "$MY_EXPECT_FILE" "$MY_OUTPUT_FILE" || { @@ -257,7 +257,7 @@ exit 1 } ################################################################################ -echo "=== Test multiple-compilation-unit: on a source file with conditionals" +echo "=== Line:${LINENO} Test preprocess: on a source file with conditionals" cat > "$MY_INPUT_FILE" <<EOF module m(); @@ -283,15 +283,14 @@ // post-blank line endmodule - EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" +"$preprocessor" preprocess "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" status="$?" [[ $status == 0 ]] || { "Expected exit code 0, but got $status" - exit 0 + exit 1 } diff --strip-trailing-cr -u "$MY_EXPECT_FILE" "$MY_OUTPUT_FILE" || { @@ -299,7 +298,7 @@ } ################################################################################ -echo "=== Test generate-variants: on a source file with nested conditionals" +echo "=== Line:${LINENO} Test generate-variants: on a source file with nested conditionals" cat > "$MY_INPUT_FILE" <<EOF \`ifdef A @@ -337,7 +336,7 @@ status="$?" [[ $status == 0 ]] || { "Expected exit code 0, but got $status" - exit 0 + exit 1 } diff --strip-trailing-cr -u "$MY_EXPECT_FILE" "$MY_OUTPUT_FILE" || { @@ -345,7 +344,7 @@ } ################################################################################ -echo "=== Test generate-variants: with limited variants" +echo "=== Line:${LINENO} Test generate-variants: with limited variants" cat > "$MY_INPUT_FILE" <<EOF \`ifdef A @@ -379,7 +378,7 @@ status="$?" [[ $status == 0 ]] || { "Expected exit code 0, but got $status" - exit 0 + exit 1 } diff --strip-trailing-cr -u "$MY_EXPECT_FILE" "$MY_OUTPUT_FILE" || { @@ -387,7 +386,7 @@ } ################################################################################ -echo "=== Test generate-variants: with invalid conditionals" +echo "=== Line:${LINENO} Test generate-variants: with invalid conditionals" cat > "$MY_INPUT_FILE" <<EOF \`ifdef A @@ -401,11 +400,11 @@ status="$?" [[ $status == 1 ]] || { "Expected exit code 1, but got $status" - exit 0 + exit 1 } ################################################################################ -echo "=== Test generate-variants: with multiple files" +echo "=== Line:${LINENO} Test generate-variants: with multiple files" cat > "$MY_INPUT_FILE" <<EOF \`ifdef A @@ -420,22 +419,22 @@ status="$?" [[ $status == 1 ]] || { "Expected exit code 1, but got $status" - exit 0 + exit 1 } ################################################################################ -echo "=== Test multiple-compilation-unit: with missing files" +echo "=== Line:${LINENO} Test preprocess: with missing files" -"$preprocessor" multiple-compilation-unit > "$MY_OUTPUT_FILE" 2>&1 +"$preprocessor" preprocess > "$MY_OUTPUT_FILE" 2>&1 status="$?" [[ $status == 1 ]] || { "Expected exit code 1, but got $status" - exit 0 + exit 1 } ################################################################################ -echo "=== Test generate-variants: with multiple files" +echo "=== Line:${LINENO} Test generate-variants: with multiple files" cat > "$MY_INPUT_FILE" <<EOF \`ifdef A @@ -450,11 +449,11 @@ status="$?" [[ $status == 1 ]] || { "Expected exit code 1, but got $status" - exit 0 + exit 1 } ################################################################################ -echo "=== Test multiple-compilation-unit: with multiple files" +echo "=== Line:${LINENO} Test preprocess: with multiple files" cat > "$MY_INPUT_FILE" <<EOF \`ifdef A @@ -470,18 +469,16 @@ A_FALSE - A_FALSE - EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" +"$preprocessor" preprocess "$MY_INPUT_FILE" "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" status="$?" [[ $status == 0 ]] || { "Expected exit code 0, but got $status" - exit 0 + exit 1 } diff --strip-trailing-cr -u "$MY_EXPECT_FILE" "$MY_OUTPUT_FILE" || { @@ -489,7 +486,7 @@ } ################################################################################ -echo "=== Test multiple-compilation-unit: passing defines" +echo "=== Line:${LINENO} Test preprocess: passing defines" cat > "$MY_INPUT_FILE" <<EOF \`ifdef A @@ -514,16 +511,15 @@ B_FALSE - EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" +define+A > "$MY_OUTPUT_FILE" +"$preprocessor" preprocess "$MY_INPUT_FILE" +define+A > "$MY_OUTPUT_FILE" status="$?" [[ $status == 0 ]] || { "Expected exit code 0, but got $status" - exit 0 + exit 1 } diff --strip-trailing-cr -u "$MY_EXPECT_FILE" "$MY_OUTPUT_FILE" || { @@ -538,29 +534,28 @@ B_TRUE - EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" +define+A=a_value+B=b_value > "$MY_OUTPUT_FILE" +"$preprocessor" preprocess "$MY_INPUT_FILE" +define+A=a_value+B=b_value > "$MY_OUTPUT_FILE" status="$?" [[ $status == 0 ]] || { "Expected exit code 0, but got $status" - exit 0 + exit 1 } diff --strip-trailing-cr -u "$MY_EXPECT_FILE" "$MY_OUTPUT_FILE" || { exit 1 } -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" +define+A +define+B > "$MY_OUTPUT_FILE" +"$preprocessor" preprocess "$MY_INPUT_FILE" +define+A +define+B > "$MY_OUTPUT_FILE" status="$?" [[ $status == 0 ]] || { "Expected exit code 0, but got $status" - exit 0 + exit 1 } diff --strip-trailing-cr -u "$MY_EXPECT_FILE" "$MY_OUTPUT_FILE" || { @@ -568,7 +563,7 @@ } ################################################################################ -echo "=== Test multiple-compilation-unit: including a file with an absolute path" +echo "=== Line:${LINENO} Test preprocess: including a file with an absolute path" cat > "$MY_INPUT_FILE" <<EOF input_content_0 @@ -582,15 +577,13 @@ cat > "$MY_EXPECT_FILE" <<EOF -${MY_INPUT_FILE}: input_content_0 included1_content input_content_1 - EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 +"$preprocessor" preprocess "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 status="$?" @@ -604,7 +597,7 @@ } ################################################################################ -echo "=== Test multiple-compilation-unit: including a file with a relative path with the correct incdir" +echo "=== Line:${LINENO} Test preprocess: including a file with a relative path with the correct incdir" cat > "$MY_INPUT_FILE" <<EOF \`include "${MY_RELATIVE_INCLUDED_FILE_1}" @@ -617,14 +610,12 @@ cat > "$MY_EXPECT_FILE" <<EOF -${MY_INPUT_FILE}: included1_content input_content - EOF -"$preprocessor" multiple-compilation-unit +incdir+${MY_INCLUDED_FILE_PATH_1} "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 +"$preprocessor" preprocess +incdir+${MY_INCLUDED_FILE_PATH_1} "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 status="$?" @@ -638,7 +629,7 @@ } ################################################################################ -echo "=== Test multiple-compilation-unit: including a file with a relative path without incdir" +echo "=== Line:${LINENO} Test preprocess: including a file with a relative path without incdir" cat > "$MY_INPUT_FILE" <<EOF \`include "${MY_RELATIVE_INCLUDED_FILE_1}" @@ -649,7 +640,7 @@ included1_content EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 +"$preprocessor" preprocess "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 status="$?" @@ -659,7 +650,7 @@ } ################################################################################ -echo "=== Test multiple-compilation-unit: including nested files with absolute paths" +echo "=== Line:${LINENO} Test preprocess: including nested files with absolute paths" cat > "$MY_INPUT_FILE" <<EOF \`include "${MY_ABSOLUTE_INCLUDED_FILE_1}" @@ -676,16 +667,14 @@ EOF cat > "$MY_EXPECT_FILE" <<EOF -${MY_INPUT_FILE}: included2_content included1_content input_content - EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 +"$preprocessor" preprocess "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 status="$?" @@ -699,7 +688,7 @@ } ################################################################################ -echo "=== Test multiple-compilation-unit: including nested files with relative paths with the correct incdirs" +echo "=== Line:${LINENO} Test preprocess: including nested files with relative paths with the correct incdirs" cat > "$MY_INPUT_FILE" <<EOF \`include "${MY_RELATIVE_INCLUDED_FILE_1}" @@ -716,16 +705,14 @@ EOF cat > "$MY_EXPECT_FILE" <<EOF -${MY_INPUT_FILE}: included2_content included1_content input_content - EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" +incdir+${MY_INCLUDED_FILE_PATH_1}+${MY_INCLUDED_FILE_PATH_2} > "$MY_OUTPUT_FILE" 2>&1 +"$preprocessor" preprocess "$MY_INPUT_FILE" +incdir+${MY_INCLUDED_FILE_PATH_1}+${MY_INCLUDED_FILE_PATH_2} > "$MY_OUTPUT_FILE" 2>&1 status="$?" @@ -739,7 +726,7 @@ } ################################################################################ -echo "=== Test multiple-compilation-unit: including nested files with relative paths without incdir" +echo "=== Line:${LINENO} Test preprocess: including nested files with relative paths without incdir" cat > "$MY_INPUT_FILE" <<EOF \`include "${MY_RELATIVE_INCLUDED_FILE_1}" @@ -755,7 +742,7 @@ included2_content EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 +"$preprocessor" preprocess "$MY_INPUT_FILE" > "$MY_OUTPUT_FILE" 2>&1 status="$?" @@ -765,7 +752,7 @@ } ################################################################################ -echo "=== Test multiple-compilation-unit: including nested files, passing +define+" +echo "=== Line:${LINENO} Test preprocess: including nested files, passing +define+" cat > "$MY_INPUT_FILE" <<EOF \`include "${MY_ABSOLUTE_INCLUDED_FILE_1}" @@ -791,7 +778,6 @@ EOF cat > "$MY_EXPECT_FILE" <<EOF -${MY_INPUT_FILE}: included2_content A_TRUE @@ -806,10 +792,9 @@ A_TRUE - EOF -"$preprocessor" multiple-compilation-unit "$MY_INPUT_FILE" +define+A > "$MY_OUTPUT_FILE" 2>&1 +"$preprocessor" preprocess "$MY_INPUT_FILE" +define+A > "$MY_OUTPUT_FILE" 2>&1 status="$?"