SoC Integration Testing: Higher-Level Software Debug Visibility

Debug is a key task in any development task. Whether debugging application-level software or a hardware design, a key to productive debug is getting a higher-level view of what is happening in the design. Blindly stepping around in source code or staring at low-level waveforms is rarely a productive approach to debugging. Debug-log messages provide a high-level view of what's happening in an software application, allowing us to better target what source we actually inspect. Testbench logging, coupled with a transaction-level view of interface activity, provides us that higher-level view when verifying IP-level designs. Much of this is lacking when it comes to verifying SoC integration.

Challenges at SoC Level

We face a few unique challenges when doing SoC-integration testing. Software (okay, really firmware) is an integral part of our test environment, but that software is running really really slowly since it is running at RTL-simulation speeds. That makes using debug messages impractical, since simulating execution of the code to produce messages makes our test software run excruciatingly slowly. In addition, the types of issues we are likely to find -- especially early on -- are not at the application-level anyway. 

Processor simulation models often provide some form of execution trace, such as ARM's Tarmac file, which provides us a window into what's happening in the software. The downsides, here, are that we end up having to manually correlate low-level execution with higher-level application execution and what's happening in the waveform. There are also some very nice commercial integrated hardware/software debug tools that dramatically simplify the task of debugging software at the source level and correlating that with what's happening in the hardware design -- well worth checking out if you have access.

RISC-V VIP

At IP level, it's common to use Verification IP to relate the signal-level view of implementation with the more-abstract level we use when developing tests and debugging. It's highly desirable, of course, to be able to use Verification IP across multiple IPs and projects. This requires the existence of a common protocol that VIP can be developed to comprehend. 

If we want VIP that exposes a higher-level view of a processor's execution, we'll need just such a common protocol to interpret. The good news is that there is such a protocol for the RISC-V architecture: the RISC-V Formal Interface (RVFI). As its name suggests, the RISC-V Formal Interface was developed to enable a variety of RISC-V cores to be formally verified using the same library of formal properties. Using the RVFI as our common 'protocol' to understand the execution of a RISC-V processor enables us to develop a Verification IP that supports any processor that implements the RVFI.

RISC-V Debug BFM

The RISC-V Debug BFM is part of the PyBfms project and, like the other Bus-Functional Models within the project, implements low-level behavior in Verilog and higher-level behavior in Python. Like other PyBfms models, the RISC-V Debug BFM works nicely with cocotb testbench environments.

Instruction-Level Trace

Like other BFMs, the Verilog side of the RISC-V Debug BFM contains various mechanics for converting the input signals to a higher-level instruction trace. Consequently, the signals that expose the higher-level view of software execution are collected in a sub-module of the BFM instance.

The image above shows the elements within the debug BFM. The ctxt scope contains the higher-abstraction view of software execution, while the regs scope inside it contains the register state.

The first level of debug visibility that we receive is at the instruction level. The RISC-V Debug BFM exposes a simple disassembly of the executed instructions on the disasm signal within the ctxt scope. Note that you need to set the trace format to ASCII or String (depending on your waveform viewer) to see the disassembly. 

C-Level Execution Trace

Seeing instruction execution and register values is useful, but still leaves us looking at software execution at a very low level. This is very limiting, and especially so if we're attempting to understand the execution of software that we didn't write -- booting of an RTOS, for example. 

Fortunately, our BFM is connected to Python and there's a readily-available library (pyelftools) for accessing symbols and other information from the software image being executed by the processor core.

The code snippet above shows our testbench obtaining the path to the ELF file from cocotb, and passing this to the RISC-V Debug BFM. Now, what can we do with a stream of instruction-execution events and an ELF file? How about reconstructing the call stack?

The screenshot above shows the call stack of the Zephyr OS booting and running a short user program. If we need to debug a design failure, we can always correlate it to where the software was when the failure occurred. 

The screenshot above covers approximately 2ms of simulation time. At this scale, the signal-level details at the top of the waveform view are incomprehensible. The instruction-level view in the middle are difficult to interpret, though perhaps you could infer something from the register values. However, the C-level execution view at the bottom is still largely legible. Even when function execution is too brief to enable the function name to be legible, sweeping the cursor makes the execution flow easy to follow.

Current Status and Looking Forward

The RISC-V Debug BFM is still early in its development cycle, with additional opportunities for new features (stay tuned!) and a need for increased stability and documentation. That said, feel free to have a look and consider whether having access to the features described above would improve your SoC bring-up experience.

Looking forward in this series of blog posts, we'll be looking next at some of the additional things we can do with the information and events collected by the RISC-V Debug BFM. Among other things, these will allow us to more tightly connect the execution of our Python-based testbench with the execution of our test software.

Finally, the process of creating the RISC-V BFM has me thinking about the possibilities when assembling an SoC from IPs with integrated higher-level debug. What if not only the processor core but also the DMA engine, internal accelerators, and external communication IPs were all able to show a high-level view of what they were doing? It would certainly give the SoC integrator a better view of what was happening, and even facilitate discussions with the IP developer. How would IP with integrated high-level debug improve your SoC bring-up experience?

Disclaimer

The views and opinions expressed above are solely those of the author and do not represent those of my employer or any other party.

FWRISC: Designing an FPGA-friendly Core in 30 Days

Designing a processor is often considered to be a large and complex undertaking, so how did I decide to design and implement in a month? For a few reasons, really. For one, my background is in hardware design, despite having worked in the EDA (Electronic Design Automation) software industry for many years. The last time I did a full hardware design was quite a few years ago using an i386EX embedded processor and other packaged ICs. Recently, though, I've been looking for opportunities to brush up on my digital-design skills. The primary reason, however, was that I saw the call for contestants in the 2018 RISC-V soft-core processor contest. I've found contests to be a fun way to learn because the organizers' criteria often cause me to learn something I otherwise wouldn't have thought to investigate. This contest was certainly no different!

The 2018 RISC-V contest certainly had some unique criteria. The contest required that verification be done using the Verilator "simulator", an open-source Verilog to C++ translator that is very fast and powerful, but also has some interesting quirks. Also required was support for Zephyr, a real-time operating system (RTOS) that I certainly wasn't aware of before the contest. Most interesting to me, though, was the contest category for smallest RISC-V FPGA implementation.

Small, you say?

When thinking about processor design, I often think about maximizing performance. However, there are many applications -- especially in the IoT space -- where having a small amount of processing power that requires little resources is very important. Often these applications are dominated today by older processor architectures, such as the venerable 8051. Despite it's somewhat-small size in an FPGA implementation, the 8051 processor isn't terribly friendly to C compilers, and is very slow. What if a modern architecture, such as the RISC-V ISA, could take the place of these older architectures while matching, or even improving, on their small size?

Despite seeing the value of having small RISC-V implementations, my first reaction when seeing the contest announcement was puzzlement. Weren't there already several small RISC-V implementations? Well, as it turns out, yes and no. There were several existing small implementations. However, the ones I found were not truly compliant with the RV32I architecture specification. The tradeoffs taken were often taken to reduce the implementation size by removing features that required resources, but were not needed for the author's intended application. These tradeoffs often meant that a special compiler toolchain was needed, or that users needed to be cautious when attempting to reuse existing software written for the RISC-V ISA.

Results?

Well, bottom line, I was able to design, verify, and implement a 32-bit RV32I RISC-V core in 30 days, and you can find the code on GitHub. A netlist of the design is shown at the beginning of this post. Early results are quite promising with respect to the balance between performance and size, and there are several known areas for improvement. Through the process, I've learned a lot -- rediscovering RTL design, gaining a much deeper appreciation of the RISC-V ISA, and learning about new tools like Verilator and infrastructure like Zephyr. Over the next few weeks, I'll be writing more about specific details of the design and verification process and what I learned. So, stay tuned for future posts!

Disclaimer

The views and opinions expressed above are solely those of the author and do not represent those of my employer or any other party.