Simplifying Custom Template-Generated Content

 

As a verification engineer, it's quite common to work with data and code that follow a regular pattern. Having an efficient way to create this repetitive code is a significant productivity boost. While there certainly are places in the code where 'your critical generation or checking algorithm' goes, much of the structure of an agent, a test environment, etc remain the same. The same goes for other parts of the flow, such as project meta-data, test lists, etc. There are two things that keep us from just making copies of a set of 'golden' files to create the basis for a new UVM agent, project, etc: some or all of the files need to have some data substituted or changed. For example, we want to substitute the name of the new UVM agent we're creating into most of the new SystemVerilog source code.

Custom code generators have been developed for some of these tasks. These often focus on providing a domain-specific way to capture input data, such as the structure of a UVM testbench or the layout of registers in a design. But there are many more opportunities to generate template-driven code that cannot justify the investment to create a focused solution.

A few years ago, I created the Verification Template Engine (VTE) to serve my needs for generating template-driven content. I developed VTE with three user-experience requirements in mind:

• Creating a new template should be very easy, but have access to powerful generation features
• Managing the available templates should be simple for a user. 
• The core tools should be generic, and make few or no assumptions about what is being generated

VTE focuses on organizing and discovering template content, but leverages the Jinja2 template engine to do the heavy lifting of template expansion. In some sense, you can think of VTE as providing a user interface to the Jinaj2 library.

I've been using VTE since developing it, but am just getting back to create proper documentation, which you can find here: https://fvutils.github.io/vte/. As part of that work, I created a quickstart guide which is both in the documentation, and forms the remainder of this post. 

Installing VTE

The easiest way to install VTE is from PyPi.

% python3 -m pip install --user vte

Test that you can run VTE by running the command (vte) and/or invoking the module:

% vte --help

% python3 -m vte --help

Creating a Template

VTE discovers templates by searching directories on the VTE_TEMPLATE_PATH environment variable. VTE uses a marker file named .vte to identify the root of a template. All files and directories in and below a template directory are considered to be part of the template. The template identifier is composed from the directory names between the directory listed in VTE_TEMPLATE_PATH and the directory containing the .vte marker file.

Let’s look at an example to illustrate the rules.

templates

  uvm

    agent

      .vte

    component

      .vte

  doc

    blog_post

      .vte

    readme

      .vte

Let’s assume we add the templates directory to VTE_TEMPLATE_PATH. VTE will find four templates:

uvm.agent

uvm.component

doc.blog_post

doc.readme

All files in and below the directory containing the .vte marker will be rendered when the template is used.

Creating the Template Structure

Let’s create a very simple template structure. Create the following directory structure:

templates

  doc

    readme

Change directory to templates/doc/readme and run the quickstart command:

% vte quickstart

Verification Template Engine Quickstart

Template directory: templates/doc/readme

Template Description []? Create a simple README

This command will prompt for a description to use for the template. Enter a description and press ENTER. This will create the .vte marker file.

View the .vte file. You’ll see that the initial version is quite simple. For now, this is all we need.

template:

  description: Create a simple README

  parameters: []

#   - name: param_name

#     description: param_desc

#     default: param_default

Creating the Template File

Now, let’s create the template file that will be processed when we render the template. Our readme template only has one file: README.md.

Create a file named README.md containing the following content in the templates/doc/readme directory:

# README for {{name}}

TODO: put in some content of interest

VTE supports defining and using multiple parameters, but defines one built-in parameter that must be supplied for all templates: name. Our template file references name using Jinja2 syntax for variable references.

We have now created a simple template for creating README.md files.

Rendering a Template

In order to render templates, VTE must first be able to discover them. Add the templates directory to the VTE_TEMPLATE_PATH environment variable.

% export VTE_TEMPLATE_PATH=<path>/templates # Bourne shell

% setenv VTE_TEMPLATE_PATH <path>/templates # csh/tsh

Let’s test this out by running the vte list command:

% vte list

doc.readme - Create a simple README

If you see the doc.readme line above, VTE has successfully discovered the template.

Now, let’s actually generate something. Let’s create a new directory parallel to the templates directory in which to try this out

% mkdir scratch

% cd scratch

Finally, let’s run the generate command:

% vte generate doc.readme my_project

Note: processing template README.md

VTE prints a line for each template file is processes. The output above confirms that is processed the template README.md file.

Let’s have a look at the result. View the README.md file in the scratch directory.

# README for my_project

TODO: put in some content of interest

Node that the {{name}} reference was replaced by the name (my_project) that we specified.

You have now created your first VTE template!

Conclusion

As the tutorial above illustrates, creating a new template for use with VTE is no more effort than making a few name substitutions. If you use the template more than once, you will already have received a positive return on the effort invested. While templates can be simple, you have the full power of the Jinja2 template engine when you need to do something more complex. I encourage you to check out the VTE documentation and look for opportunities where using template-driven content generation can make your life easier and make you more productive.

Copyright 2022 Matthew Ballance

The views and opinions expressed above are solely those of the author and do not represent those of my employer or any other party.

Generate Custom Content Quickly with a Template Engine

I was at DVCon earlier this week, and attended the UVM Update Tutorial delivered by Cliff Cummings. Cliff described his typical process of creating UVM elements, which is to have a set of 'golden' template files that can be copied and hand-customized to the task at hand. It's certainly something I've done frequently on UVM projects and in many other contexts as well.

If you work in verification and you're anything like me, you create lots of files pretty much every day. Scripts, UVM code, C++ code, Makefiles, meta-data files, the list goes on and on. While all of these files will contain custom content, there is often a lot of boilerplate content as well. There are certainly specialized tools focused on quickly creating content for specific domains. For example, there are several tools and libraries to help quickly create UVM elements based on some high-level inputs from the user. But, what about files that are custom to your environment? Can we do better than the standard process of copy, rename, and then manually doing a search/replace in each file?

VTE
I've recently been playing with just such a template engine for my own use. For lack of a better name, I've been calling it the Verification Template Engine (VTE), since I'm largely using it for creating files in the context of a verification environment. VTE is implemented in Python and leverages the Jinja2 template engine, which is typically used for creating web content from a template.

Given how much VTE is just reusing existing functionality, what is the value-add? Two things, really:

• VTE provides a user interface that makes it easy to see what templates are available in the environment
• VTE specifies a template structure that makes it really easy to create new templates from those 'golden' template files that you would normally copy, rename, and modify.

Using Templates
VTE currently ships a couple of small templates with the tool. Your custom templates will be found via the VTE_TEMPLATE_PATH environment variable.

VTE currently supports two commands: list and generate. The list command shows the set of templates that were found on the template path.

% vte list
project.ivpm.googletest-hdl - IVPM project with Googletest-HDL dependence
verif.ip.googletest-hdl     - IP verification environment using Googletest-HDL
verif.uvm.test              - Creates a UVM test class

The generate command, as you might expect, generates content from a specified template. 

usage: vte generate [-h] [-force] template name [KEY=VALUE [KEY=VALUE ...]]

positional arguments:
  template    ID of template
  name        Name to use in the template
  KEY=VALUE   Specify other template variables

optional arguments:
  -h, --help  show this help message and exit
  -force      force overwrite of existing files

Here's an example using VTE to create the skeleton project structure that I use when developing an IP:

% vte generate project.ivpm.googletest-hdl my_proj
Note: processing template etc/ivpm.info
Note: processing template etc/packages.mf
Note: processing template etc/env.sh
Note: processing template scripts/ivpm.py
Note: processing template scripts/ivpm.mk
% find -type f
./etc/ivpm.info
./etc/packages.mf
./etc/my_proj_env.sh
./scripts/ivpm.py
./scripts/ivpm.mk

Creating Templates

Creating new template is simple as well. VTE. uses a marker file, named .vte to identify the root directory of a template. Any files in that directory or its subdirectories is considered to be a template file. The ID for a template is composed from the path elements between the VTE_TEMPLATE_PATH entry and the directory containing the .vte marker file.

For a very simple template, all you need is an empty .vte marker. However, adding a little information to the .vte file makes your template more useful. For example, you can add a short description for your template, which will be shown with the vte list command is run:

[template]
desc = Creates a UVM test class

The template files, themselves, use Jinja2 markup to refer to template parameters and specify directives.

/****************************************************************************
 * {{name}}.svh
 * 
 ****************************************************************************/
{%set filename = "{{name}}.svh" %}
/**
 * Class: {{name}}
 * TODO: Document UVM Test {{name}}
 */
class {{name}} extends {{base_class}};
 `uvm_component_utils({{name}})
 
 function new(string name, uvm_component parent=null);
  super.new(name, parent);
 endfunction
 
endclass




The example above is a template for a UVM Test class. The {% set filename ... %} directive specifies the name of the output file, which depends on the value of the name parameter. The rest of the template contains a reference to the name parameter wherever we want that to be substituted in. As you can probably guess, these locations where {{name}} is referenced are the very locations that you'd go through with a text editor and manually substitute in some content.


Sound Interesting?
If this all sounds interesting, check out VTE on GitHub: https://github.com/fvutils/vte. You can download a release tarball from the Releases area, add the bin directory to your PATH, and try it out right away. You can also find documentation in the README file.
If you find an issue with VTE, or would like to see it support a new feature, file a ticket on the GitHub issue tracker. Even better, send me a pull request!

My hope is that a tool like VTE can make creating boilerplate code much simpler and faster, allowing you to focus your efforts on adding truly-unique content. That is, after all, always the goal of engineering and automation, right?