Frameworks have been enormously helpful in making testing and verification productive and reusable. In the RTL verification space, AVM, VMM, OVM, and UVM have all contributed to standardizing a pattern for RTL verification that encompasses encapsulation and reuse, automation for stimulus generation, and mechanisms for results checking. In the software space, frameworks such as JUnit and CPPUnit have, likewise, provide structure around standardizing how test results are checked and how test suites are composed and executed in an automated fashion.
As more focus is given to verification issues at SoC and system level, it makes sense to ask: are the requirements for a verification framework in a system level environment met by existing (primarily RTL-centric) verification frameworks, or is there something fundamentally different at system level? As it turns out, there are some rather unique aspects of system-level verification that make existing verification frameworks unsuitable for application in this space.
Perhaps the most visible difference in an system-level design is the presence of embedded processors that are simultaneously part of the design and part of the verification environment. The verification framework must enable automation for test generation, as well as facilitate managing the processors as design and verification resources in configurations ranging from 'bare metal', to OS and test, to an OS running application software and a test.
Another difference with a system-level environment is that verification must look both forward and backwards in the verification process. It is a requirement that the same test scenario be able to run in an SoC-level simulation/emulation context that includes a SystemVerilog/UVM testbench, as well as running in a prototype context (FPGA or first silicon), as well as potentially running on the end product. Now, that certainly doesn't mean that the same verification will be done at each step: the purpose and goals of verification in each context are quite different. However, being able to re-run the same scenario in two verification contexts provides some continuity between the contexts and avoids having to start from scratch when changing contexts. For example, consider an SoC-level environment in simulation. Much of the test stimulus is still hardware-centric, but early embedded-software scenarios are being executed. When moving to emulation, it is enormously valuable to be able to run the same scenario that was proven to run in simulation and not have to start developing a test from scratch. Continuity is even more helpful when moving from emulation (which still largely maintains the appearance of simulation) to an FPGA prototype environment that is radically different.
As mentioned in a previous post, system-level environments tend to look more like cooperating islands of test functionality rather than the monolithic testbench used for block and subsystem RTL verification. A system-level verification framework must enable reuse of verification infrastructure across these islands, as well as facilitating the cooperation of these islands in carrying out the broader verification task.
Just because the requirements are different for a system-level verification framework doesn't mean that design of a system-level verification framework must start from first principals. Hardware- and software-centric test frameworks have been in development for over a decade (some would argue much, much longer), and there is much to be gained from the features that, over time, were found to be useful for verifying hardware and software.
I've recently started work on a lightweight C++-based verification framework that targets the requirements of system-level verification. The framework borrows features and patterns from existing verification frameworks, and adds features and patterns to address some of the unique requirements of driving embedded software tests and coordinating the multiple 'islands' of a system-level verification environment. The framework is designed to be encapsulated in environments as diverse as UVM and special-purpose verification hardware accelerators. The next few blog posts will explore some of the specifics of this developing framework.
~Everything is a system -- some are just smaller than others
