Merge pull request #74 from AndresNavarro82/machxo2-pullreq0
Bitstream decompression and better handling of crc & dummies (so they also work with MachXO2)
diff --git a/docs/architecture/bitstream_format.rst b/docs/architecture/bitstream_format.rst
index 273dfd5..3b7234d 100644
--- a/docs/architecture/bitstream_format.rst
+++ b/docs/architecture/bitstream_format.rst
@@ -22,7 +22,7 @@
The CRC16 is accumulated over all commands until a CRC16 check is reached. It is not reset at the end of commands
without a CRC16 check - except the ``LSC_RESET_CRC`` command, and after the actual bitstream payload
-(``LSC_PROG_INCR_RTI``).
+(``LSC_PROG_INCR_RTI`` or ``LSC_PROG_INCR_CMP``).
The CRC16 is calculated using the polynomial 0x8005 with no bit reversal. This algorithm is sometimes known as
"CRC16-BUYPASS".
@@ -45,6 +45,11 @@
| ``VERIFY_ID`` | E2 | - 24 bit info: all 0 | This checks the actual device ID against the given|
| | | - 32 bit device JTAG ID | value and fails if they do not match. |
+-------------------------------+-----+--------------------------+---------------------------------------------------+
+| ``LSC_WRITE_COMP_DIC`` | 02 | - 24 bit info: all 0 | This stores the 8 most common bytes in the frames |
+| | | - 8 bit Pattern7 | |
+| | | - ... (6 more patterns) | |
+| | | - 8 bit Pattern0 | |
++-------------------------------+-----+--------------------------+---------------------------------------------------+
| ``LSC_PROG_CNTRL0`` | 22 | - 24 bit info: all 0 | This sets the value of device control register 0 |
| | | - 32 bit CtlReg0 value | Normally 0x40000000 |
+-------------------------------+-----+--------------------------+---------------------------------------------------+
@@ -60,7 +65,8 @@
Configuration Data
-------------------
-The FPGA configuration data itself is programmed by using command ``LSC_PROG_INCR_RTI`` (0x82). Following this command,
+The FPGA configuration data itself is programmed by using command ``LSC_PROG_INCR_RTI`` (0x82) if no compression is
+used and command ``LSC_PROG_INCR_CMP`` (0xB8) when using compression. Following either of these commands,
there are some setup bits:
- 1 bit: CRC16 comparison flag, normally set
@@ -72,21 +78,59 @@
This is then followed by a number of frames, each in the following format:
- - The configuration frame itself, such that bit 0 of the first byte sent is the MSB of the frame,
- bit 7 of the first byte the MSB-7 and bit 0 of the last byte (if there are no dummy bits) being the LSB of the frame.
- - Any dummy bits needed to pad the frame to a whole number of bytes
- - A CRC-16 checksum:
-
- - For the first frame, this also covers any other commands sent
- before the programming command but after a CRC reset, and the programming command itself.
+ - The configuration frame itself (compressed in the case of the ``LSC_PROG_INCR_RTI`` command),
+ such that bit 0 bit 0 of the first byte is the MSB of the frame, bit 7 of the first byte the
+ MSB-7 and bit 0 of the last byte (if there are no dummy bits) being the LSB of the frame.
+ - Any dummy bits needed to pad the frame to a whole number of bytes.
+ - If the second flag is cleared (see above) a CRC-16 checksum:
+ - For the first frame, this also covers any other commands sent before the programming command
+ but after a CRC reset, and the programming command itself.
- For subsequent frames, this excludes dummy bytes between frames
- Dummy 0xFF bytes, usually only 1
The highest numbered frame in the chip is sent first.
-Separate commands are used if EBR needs to be configured in the bitstream. ``EBR_ADDRESS`` (0xF6) is used to select the
-EBR to program and the starting address in the EBR; and ``LSC_EBR_WRITE`` (0xB2) is used to program the EBR itself using
-72-bit frames. The specifics of these still need to be documented.
+If the second flag is set there's no CRC sent in between frames but there's still one CRC-16 checksum
+after all the frames (this also covers any other commands sent before the programming command but after a CRC reset,
+and the programming command itself.).
+
+Separate commands are used if EBR needs to be configured in the bitstream. EBR data can't use compression.
+``EBR_ADDRESS`` (0xF6) is used to select the EBR to program and the starting address in the EBR;
+and ``LSC_EBR_WRITE`` (0xB2) is used to program the EBR itself using 72-bit frames. The specifics of these
+still need to be documented.
+
+Compression Algorithm
+------------------------------
+
+ - Before compression, the frame is left padded with zeroes (0) to make the data frame 64-bit bounded.
+ - After compressing the frame data, the resulting bits are right padded with zeroes (0) to make the data
+frame byte bounded.
+
+After padding, every byte in the bitstream is compressed by a simple prefix-free code with just 4 cases:
+
++--------------+--------------+--------+---------------------+
+| Code | Argument | Length | Encoded byte |
++==============+==============+========+=====================+
+| 0 | | 1 | zero |
++--------------+--------------+--------+---------------------+
+| 100xxx | bit position | 6 | byte with 1 bit set |
++--------------+--------------+--------+---------------------+
+| 101xxx | byte index | 6 | stored byte |
++--------------+--------------+--------+---------------------+
+| 11xxxxxxxx | lit. byte | 10 | all others |
++--------------+--------------+--------+---------------------+
+
+- The first case is for the byte zero (``00000000``). That's just represented by a single zero bit (``0``).
+- The second case is for bytes with just one bit set. After a ``100`` the set bit position is encoded
+ in the following 3 bits. For example the byte ``00100000`` is encoded as ``100101`` because only the
+ bit 5 is set (with 0 being the lsb and 7 the msb).
+- The third case is for selecting one of the bytes stored by the ``LSC_WRITE_COMP_DIC`` instruction. Those
+ bytes are selected as the 8 most common bytes (ignoring the zero bytes and the bytes with just one bit set,
+ because those are encoded with the two previous cases). After a ``101`` the number of the selected pattern
+ is encoded with 3 bits. For example to select pattern3 the code would be ``101011``.
+- The fourth case is for all remaining bytes. In that case after a ``11`` the complete byte is copied. For example
+ byte ``11001010`` would be encoded as ``1111001010``.
+
Device-Specific Information
------------------------------
@@ -105,4 +149,4 @@
| LFE5U-85 | 0x41113043 | 13294 | 1136 | 0 |
+-----------+-------------+--------+-----------------------+----------------------+
| LFE5UM-85 | 0x01113043 | 13294 | 1136 | 0 |
-+-----------+-------------+--------+-----------------------+----------------------+
\ No newline at end of file
++-----------+-------------+--------+-----------------------+----------------------+
diff --git a/libtrellis/include/Bitstream.hpp b/libtrellis/include/Bitstream.hpp
index 72c475b..3d04cb6 100644
--- a/libtrellis/include/Bitstream.hpp
+++ b/libtrellis/include/Bitstream.hpp
@@ -51,6 +51,7 @@
static Bitstream generate_jump(uint32_t address);
// Deserialise a bitstream to a Chip
+ Chip deserialise_chip();
Chip deserialise_chip(boost::optional<uint32_t> idcode = boost::optional<uint32_t>());
// Write a Lattice .bit file (metadata + bitstream)
diff --git a/libtrellis/src/Bitstream.cpp b/libtrellis/src/Bitstream.cpp
index afaefc2..1ea0bdb 100644
--- a/libtrellis/src/Bitstream.cpp
+++ b/libtrellis/src/Bitstream.cpp
@@ -9,6 +9,7 @@
#include <boost/optional.hpp>
#include <iomanip>
#include <fstream>
+#include <array>
namespace Trellis {
@@ -69,6 +70,18 @@
return val;
}
+ // The command opcode is a byte so this works like get_byte
+ // but doesn't update the CRC if it's a dummy, because the docs
+ // says that dummy commands don't update the crc
+ inline BitstreamCommand get_command_opcode() {
+ assert(iter < data.end());
+ uint8_t val = *(iter++);
+ BitstreamCommand cmd = BitstreamCommand(val);
+ if (cmd != BitstreamCommand::DUMMY)
+ update_crc16(val);
+ return cmd;
+ }
+
// Write a single byte and update CRC
inline void write_byte(uint8_t b) {
data.push_back(b);
@@ -84,6 +97,83 @@
}
}
+ // Decompress and copy multiple bytes into an OutputIterator and update CRC
+ template<typename T>
+ void get_compressed_bytes(T out, size_t count, array<uint8_t, 8> compression_dict) {
+ // Here we store data already read by read_byte(), it may be more than 1 byte at times!!
+ uint16_t read_data = 0;
+ size_t remaining_bits = 0;
+ bool next_bit;
+
+ uint8_t udata;
+
+ //
+ // Every byte can be encoded by on of 4 cases
+ // It's a prefix-free code so we can identify each one just by looking at the first bits:
+ // 0 -> Byte zero (0000 0000)
+ // 100 xxx -> Stored byte in compression_dict, xxx is the index (0-7)
+ // 101 xxx -> Byte with a single bit set, xxx is the index of the set bit (0 is lsb, 7 is msb)
+ // 11 xxxxxxxx -> Literal byte, xxxxxxxx is the encoded byte
+ //
+ for (size_t i = 0; i < count; i++) {
+ // Make sure we have at least one bit in the buffer
+ if (!remaining_bits) {
+ read_data = (uint32_t) get_byte();
+ remaining_bits = 8;
+ }
+ next_bit = bool(read_data >> (remaining_bits-1) & 1);
+ remaining_bits--;
+
+ // Check the 4 cases leaving the uncompressed byte in udata
+ if (next_bit) {
+ // Starts with 1, so check next bit/bits
+ // For each of the 3 remaining cases we will need at least 5 more bits,
+ // so if we have less than that it's ok to read another byte
+ if (remaining_bits < 5) {
+ read_data = (read_data << 8) | ((uint32_t) get_byte());
+ remaining_bits += 8;
+ }
+ next_bit = bool(read_data >> (remaining_bits-1) & 1);
+ remaining_bits--;
+
+ if (next_bit) {
+ // 11 xxxx xxxx: Literal byte, just read the next 8 bits & use that
+ // we consumed 10 bits total
+ if (remaining_bits < 8) {
+ read_data = (read_data << 8) | ((uint32_t) get_byte());
+ remaining_bits += 8;
+ }
+ udata = uint8_t((read_data >> (remaining_bits - 8)) & 0xff);
+ remaining_bits -= 8;
+ } else {
+ // Starts with 10, it could be a stored literal or a single-bit-set byte
+ // 10 ? xxx: In both cases we need the index xxx, so extract it now
+ // We already have all the bits we need buffered
+ next_bit = bool(read_data >> (remaining_bits-1) & 1);
+ remaining_bits--;
+ size_t idx = (size_t) ((read_data >> (remaining_bits-3)) & 0x7);
+ remaining_bits -= 3;
+ if (next_bit) {
+ // 101 xxx: Stored byte. Just use xxx as index in the dictionary,
+ // we consumed 6 bits
+ udata = compression_dict[idx];
+ } else {
+ // 100 xxx: Single-bit-set byte, xxx is the index of the set bit
+ // we consumed 6 bits
+ udata = uint8_t(1 << idx);
+ }
+ }
+ } else {
+ // 0: the uncompressed byte is zero
+ // we consumed just one bit
+ udata = 0;
+ }
+ *out = udata;
+ ++out;
+ }
+ // if remaining bits > 0 they are just padding bits added to the end so we can ignore them
+ }
+
// Write multiple bytes from an InputIterator and update CRC
template<typename T>
void write_bytes(T in, size_t count) {
@@ -101,18 +191,6 @@
for (size_t i = 0; i < count; i++) write_byte(0x00);
}
- // Skip over a possible-dummy command section of N bytes, updating CRC only if command is not 0xFF
- uint8_t skip_possible_dummy(int size) {
- uint8_t cmd = *(iter++);
- if (cmd == 0xFF) {
- iter += (size - 1);
- } else {
- update_crc16(cmd);
- skip_bytes(size - 1);
- }
- return cmd;
- }
-
// Insert dummy bytes into the bitstream, without updating CRC
void insert_dummy(size_t count) {
for (size_t i = 0; i < count; i++)
@@ -236,11 +314,17 @@
static const vector<uint8_t> preamble = {0xFF, 0xFF, 0xBD, 0xB3};
+Chip Bitstream::deserialise_chip() {
+ return deserialise_chip(boost::none);
+}
+
Chip Bitstream::deserialise_chip(boost::optional<uint32_t> idcode) {
cerr << "bitstream size: " << data.size() * 8 << " bits" << endl;
BitstreamReadWriter rd(data);
boost::optional<Chip> chip;
bool found_preamble = rd.find_preamble(preamble);
+ boost::optional<array<uint8_t, 8>> compression_dict;
+
if (!found_preamble)
throw BitstreamParseError("preamble not found in bitstream");
@@ -248,8 +332,8 @@
int addr_in_ebr = 0;
while (!rd.is_end()) {
- uint8_t cmd = rd.get_byte();
- switch ((BitstreamCommand) cmd) {
+ BitstreamCommand cmd = rd.get_command_opcode();
+ switch (cmd) {
case BitstreamCommand::LSC_RESET_CRC:
BITSTREAM_DEBUG("reset crc");
rd.skip_bytes(3);
@@ -293,40 +377,88 @@
rd.check_crc16();
}
break;
+ case BitstreamCommand::LSC_WRITE_COMP_DIC: {
+ bool check_crc = (rd.get_byte() & 0x80) != 0;
+ rd.skip_bytes(2);
+ compression_dict = boost::make_optional(array<uint8_t, 8>());
+ // patterns are stored in the bitstream in reverse order: pattern7 to pattern0
+ for (int i = 7; i >= 0; i--) {
+ uint8_t pattern = rd.get_byte();
+ compression_dict.get()[i] = pattern;
+ }
+ BITSTREAM_DEBUG("write compression dictionary: " <<
+ "0x" << hex << setw(2) << setfill('0') << int(compression_dict.get()[0]) << " " <<
+ "0x" << hex << setw(2) << setfill('0') << int(compression_dict.get()[1]) << " " <<
+ "0x" << hex << setw(2) << setfill('0') << int(compression_dict.get()[2]) << " " <<
+ "0x" << hex << setw(2) << setfill('0') << int(compression_dict.get()[3]) << " " <<
+ "0x" << hex << setw(2) << setfill('0') << int(compression_dict.get()[4]) << " " <<
+ "0x" << hex << setw(2) << setfill('0') << int(compression_dict.get()[5]) << " " <<
+ "0x" << hex << setw(2) << setfill('0') << int(compression_dict.get()[6]) << " " <<
+ "0x" << hex << setw(2) << setfill('0') << int(compression_dict.get()[7]));;
+ if (check_crc)
+ rd.check_crc16();
+ }
+ break;
case BitstreamCommand::LSC_INIT_ADDRESS:
rd.skip_bytes(3);
BITSTREAM_DEBUG("init address");
break;
+ case BitstreamCommand::LSC_PROG_INCR_CMP:
+ // This is the main bitstream payload (compressed)
+ BITSTREAM_DEBUG("Compressed bitstream found");
+ if (!compression_dict)
+ throw BitstreamParseError("start of compressed bitstream data before compression dictionary was stored", rd.get_offset());
+ // fall through
case BitstreamCommand::LSC_PROG_INCR_RTI: {
// This is the main bitstream payload
if (!chip)
throw BitstreamParseError("start of bitstream data before chip was identified", rd.get_offset());
+ bool reversed_frames;
+ if (chip->info.family == "MachXO2")
+ reversed_frames = false;
+ else if (chip->info.family == "ECP5")
+ reversed_frames = true;
+ else
+ throw BitstreamParseError("Unknown chip family: " + chip->info.family);
+
uint8_t params[3];
rd.get_bytes(params, 3);
BITSTREAM_DEBUG("settings: " << hex << setw(2) << int(params[0]) << " " << int(params[1]) << " "
<< int(params[2]));
- size_t dummy_bytes = (params[0] & 0x0FU);
+ // I've only seen 0x81 for the ecp5 and 0x8e for the xo2 so far...
+ bool check_crc = params[0] & 0x80U;
+ // inverted value: a 0 means check after every frame
+ bool crc_after_each_frame = check_crc && !(params[0] & 0x40U);
+ // I don't know what these two are for I've seen both 1s (XO2) and both 0s (ECP5)
+ // The names are from the ECP5 docs
+ // bool include_dummy_bits = params[0] & 0x20U;
+ // bool include_dummy_bytes = params[0] & 0x10U;
+ size_t dummy_bytes = params[0] & 0x0FU;
size_t frame_count = (params[1] << 8U) | params[2];
- BITSTREAM_NOTE(
- "reading " << std::dec << frame_count << " config frames (with " << std::dec << dummy_bytes
- << " dummy bytes)");
+ BITSTREAM_NOTE("reading " << std::dec << frame_count << " config frames (with " << std::dec << dummy_bytes << " dummy bytes)");
size_t bytes_per_frame = (chip->info.bits_per_frame + chip->info.pad_bits_after_frame +
chip->info.pad_bits_before_frame) / 8U;
+ // If compressed 0 bits are added to the stream before compression to make it 64 bit bounded, so
+ // we should consider that space here but they shouldn't be copied to the output
+ if (cmd == BitstreamCommand::LSC_PROG_INCR_CMP)
+ bytes_per_frame += (7 - ((bytes_per_frame - 1) % 8));
unique_ptr<uint8_t[]> frame_bytes = make_unique<uint8_t[]>(bytes_per_frame);
for (size_t i = 0; i < frame_count; i++) {
- rd.get_bytes(frame_bytes.get(), bytes_per_frame);
+ size_t idx = reversed_frames? (chip->info.num_frames - 1) - i : i;
+ if (cmd == BitstreamCommand::LSC_PROG_INCR_CMP)
+ rd.get_compressed_bytes(frame_bytes.get(), bytes_per_frame, compression_dict.get());
+ else
+ rd.get_bytes(frame_bytes.get(), bytes_per_frame);
+
for (int j = 0; j < chip->info.bits_per_frame; j++) {
size_t ofs = j + chip->info.pad_bits_after_frame;
- chip->cram.bit((chip->info.num_frames - 1) - i, j) = (char) (
- (frame_bytes[(bytes_per_frame - 1) - (ofs / 8)] >> (ofs % 8)) & 0x01);
+ chip->cram.bit(idx, j) = (char)
+ ((frame_bytes[(bytes_per_frame - 1) - (ofs / 8)] >> (ofs % 8)) & 0x01);
}
- rd.check_crc16();
+ if (crc_after_each_frame || (check_crc && (i == frame_count-1)))
+ rd.check_crc16();
rd.skip_bytes(dummy_bytes);
}
- // Post-bitstream space for SECURITY and SED
- // TODO: process SECURITY and SED
- rd.skip_possible_dummy(8);
- rd.skip_possible_dummy(4);
}
break;
case BitstreamCommand::LSC_EBR_ADDRESS: {
diff --git a/libtrellis/src/PyTrellis.cpp b/libtrellis/src/PyTrellis.cpp
index 2d58620..52e0fbf 100644
--- a/libtrellis/src/PyTrellis.cpp
+++ b/libtrellis/src/PyTrellis.cpp
@@ -63,7 +63,7 @@
.def("write_bit", &Bitstream::write_bit_py)
.def_readwrite("metadata", &Bitstream::metadata)
.def_readwrite("data", &Bitstream::data)
- .def("deserialise_chip", &Bitstream::deserialise_chip);
+ .def("deserialise_chip", static_cast<Chip (Bitstream::*)()>(&Bitstream::deserialise_chip));
class_<DeviceLocator>("DeviceLocator")
.def_readwrite("family", &DeviceLocator::family)
