PyUCIS: Manipulating Coverage Data

 

In a prior post, we looked at how to inspect coverage as a text report and export coverage data using the PyVSC API, and view coverage graphically using the PyUCIS-Viewer. Recent enhancements have enabled the PyUCIS library to provide even more ways to manipulate coverage data. Over the next couple of posts, we’ll look at those enhancements. 

New ‘ucis’ Command

PyUCIS is a library for working with the Accellera UCIS data model. It started as a library for other applications and libraries, such as PyVSC and the PyUCIS Viewer, to use to read and write data using the UCIS data model. Recent enhancements have added standalone functionality which can meaningfully be accessed from the command line. 

You can find documentation for the ucis command and sub-commands in the PyUCIS documentation. Fundamentally, there are four key operations:

• Convert coverage data from one format to another
• Merge coverage data from multiple databases into a single database
• Produce coverage reports in various formats
• Obtain information about available coverage data and report formats

These commands are just a starting point. They will be enhanced over time, and more commands may be added as well. If you have suggestions for new commands and/or new capabilities for existing commands, feel free to add an enhancement request on the PyUCIS GitHub page.

Plug-in Framework

PyUCIS has added a plug-in framework with support for database formats and report formats. The goal is to make commands operating on coverage data extensible extensible from the beginning, as well as to enable the set of supported coverage-data formats and report formats to be easily extended without changing PyUCIS.  I’ll devote a future post to the plug-in framework. For now, the ucis command supports listing the available coverage-data and report plug-ins. For example:

% ucis list-db-formats

libucis - Reads coverage data via an implementation of the UCIS C API

xml     - Supports reading and writing UCIS XML interchange

yaml    - Reads coverage data from a YAML file

New Input Format

One often-requested PyUCIS feature is the ability to merge coverage data from several input coverage databases into a single resulting coverage database. One of the first challenges I faced in implementing this functionality was how to write tests. The UCIS API is written with applications in mind. I’ve found it to be a pretty-verbose API when it comes to writing tests. Consequently, tests written directly in terms of the API aren’t particularly easy to follow from a code perspective.

I decided to define a YAML format to make it simpler to capture coverage data in an easy-to -read way. Initially, this was just for testing. However, it may also be a useful interchange format that is less verbose and complex (also, quite possibly, more simplistic) that the XML interchange format defined by the UCIS standard.

A simple coverage specification is shown above. This coverage data describes a covergroup type (my_cvg) with a single instance (i1). A single coverpoint (cp1) has two bins (b1, b2) of which one has a single hit and one has no hits. While this coverage specification was created to make setting of test coverage data simpler for a human, I believe it may also be useful as a simple coverage-interchange format. If you find it useful, please let the community know via the Discussion forum on the PyUCIS GitHub page.

You can find more details on the YAML Coverage Data Format reference documentation page. 

Merging Coverage Data

One consistently-requested feature for PyUCIS is the ability to merge multiple databases into a single unified coverage database. PyUCIS now supports basic merge functionality. Currently, PyUCIS performs a union merge where all unique coverage features found in all the input databases are propagated to the output database. I anticipate that more merge algorithms will need to be added over time, but hopefully this is a good start.

Let’s take a look at a very simple case. Let’s say we have two coverage-data sets shown below:

The structure of these two coverage databases is the same (same covergroup type, instance, and coverpoint). Each coverage database has 50% coverage. Let’s merge these two databases and report the coverage.

% ucis merge -if yaml -o merge.xml coverage_1.ycdb coverage_2.ycdb

We specify the two input databases, as well as their format (yaml). We specify the output database as merge.xml.

The resulting coverage report on the merged database will report 100% coverage, as expected:

% ucis report merge.xml

TYPE i1 : 100.000000%

    CVP cp1 : 100.000000%

Reporting Coverage Data

Reporting is a key activity when working with coverage data. We’ve looked at the ability to browse coverage data graphically using the PyUCIS-Viewer, but getting a textual report is every bit as important. In addition to presenting information concisely, textual reports can be processed programmatically to extract key pieces of data. 

We can list the currently-available report plugins using the ucis command:

% ucis list-rpt-formats

json - Produces a machine-readable JSON coverage report

txt  - Produces a human-readable textual coverage report

The default report is textual. Let’s create a textual report on the YAML coverage-data above:

% ucis report -if yaml coverage.ycdb 

Note that we need to specify the format of the input data (yaml). The result is a simple human-readable report of the coverage data in the database.

What if we wanted to post-process the data using a script? We certainly could extract what we need by parsing the output above, but working with data in a machine-readable format is often much simpler. Let’s report our data in JSON format:

% ucis report -if yaml -of json coverage.ycdb 

Obviously, the data is less compact and more verbose. But, reading this into a Python script for further post-processing is incredibly simple! If you’re interested in the JSON report format, have a look at the schema documentation <https://pyucis.readthedocs.io/en/latest/reference/coverage_report_json.html>.

So, for now, PyUCIS supports two textual report formats, and would benefit from more report formats. For example, a plain HTML report and a fancy interactive web-based report. If someone in the community has the skills and is interested, the project would definitely be interested!

Next Steps

PyUCIS continues to evolve, adding a more more hopefully-useful features at a time. Stay tuned for a future post on the plug-in interface, and the addition of more coverage-database and report formats. 

Copyright 2022 Matthew Ballance

The views and opinions expressed above are solely those of the author and do not represent those of my employer or any other party.