This is the second in a series of three blog posts on how to use ClearCase to help generate accurate and comprehensive Software Bills of Materials (SBOM) for your applications.
Virtually any software system is dependent on 3rd-party components (i.e. not authored by the team), even if just the utility classes/functions provided with the language used (e.g. the C++ Standard Library). It is frequently said, “Put everything in a VOB!”
Applying this practice to 3rd-party components not only makes it easy to answer the what/where/who/when/how questions to assemble an SBOM, it provides significant additional benefits. Building with components that are managed in VOBs isolates builds from the environment of the machines performing the build. It no longer matters if the correct version and patch level of the components are installed on the build machines (or installed at all). Baselines of product builds/releases will also include the versions of components referenced. If ClearCase audited builds are utilized, the 3rd-party components will be included in the audit records (configuration records) yielding authoritative traceability. This practice significantly eases the burden of creating accurate Software Bills of Materials when you release your application.
Structuring 3rd-party Component Imports
There are different ways of structuring the imports of 3rd-party components. Which approach is appropriate for a given component depends on the form of the component and the needs of the project(s) dependent on the component.
Forms of Components
There are three common forms of components:
- Software Development Kit (SDK) – This form includes the files needed to build your application with that component as a dependency and to distribute the needed runtime parts of the component with a dependent application. The SDK typically does not include source code for the component.
- Integrated source code – This form is a distribution of the source code where the build of the component is integrated with the build of your application. A complete build of your application will also build any integrated source code components.
- SDK from source code – This form is a distribution of the source code for the component. Instead of integrating the build of the component with the build of your application, the component is imported into a VOB but built independently, and the resulting SDK is imported into a VOB. The imported SDK is then referenced by the application build. Unless the component is updated frequently, the “SDK from source code” form is usually preferable over the “Integrated source code” form.
Selecting Component Version
There are two ways of structuring how the version of a component is chosen for a build:
- SCM configuration-based version selection – With this approach, new releases of the component will be overlaid on the predecessor release, changed files checked in, and everything labeled (or baselined) with a descriptive label (e.g. V2.7.5). The config spec for your application builds will specify the version of the component to use (e.g. “element /vobs/that_component/… V2.7.5”).
- Build environment-based version selection – With this approach, new releases of the component are placed in an entirely new directory with new elements created for each file. The new directory would be named with the release of the component (e.g. “/vobs/that_component/v2.7.5”). The build environment/scripts will specify the directory name to use for the appropriate version of the component.
An application may build with mixture of components using different approaches, as appropriate.
Component Root Directories
Component Root Directories may be multi-level with each level representing important distinctions. The top-level directory will typically be the name of the component (e.g. “/vobs/that_component”). If the component is operating system (OS) and architecture-specific (e.g. binaries for Red Hat Linux 8.x on Intel x86 64-bit), subdirectories may represent the target architecture. This can be structured based on your preferences. Examples include:
- …/LINUX/linux_x86_64_rhat8
- …./LINUX/rhat8/x86_64
- …/rhat8_x86_64
Give thought to how many combinations and permutations you will need to support and that will likely influence your choice of structure.
The entire architecture-specific SDK would then be imported in its normal structure in that Component Root Directory. For an SDK for the C language, you might expect to see an ‘include’ directory, a ‘lib’ directory, and perhaps a ‘man’ directory in the Component Root Directory.
Importing the Component
Most components should be imported in their entirety and in their normal structure. The last-modified-timestamp should be preserved to aid with traceability. Supply a descriptive comment for checked in files. Apply a meaningful label to new versions (e.g. “V2.0.1”).
In general, the easiest way to import a component is to use ‘clearfsimport’ ( https://www.ibm.com/docs/en/clearcase/11.0.0?topic=systems-clearfsimport ). Run ‘clearfsimport’ in a dynamic view as the VOB owner or root, if possible. To optimize ‘clearfsimport’ performance, consider running it on the server hosting the VOB using a local view.
Importing Symbolic Links
ClearCase supports symbolic links in VOBs. However, symbolic links (symlinks) to absolute path names are unlikely to work as intended. Thus, any absolute symlinks in the “import from” location should be recreated as relative symlinks before the import. Alternatively, imported absolute symlinks should recreated as relative symlinks after the import.
Capturing Version/Provenance/License Information
As part of each import, you should capture pertinent component version detail and provenance information. This can be stored in a file(s) in the component root directory. The information can be a combination of the output of commands and manually entered data. Many SBOM generation tools prefer to extract pertinent information from structured data (e.g. JSON or XML) files. Consider creating a structured data file that includes key information needed to assemble an SBOM. It is also wise to include the license associated with the component in another file. Use the same file names for this captured information in each component.
Although the way this information is captured will vary, the principles apply regardless of operating system (e.g. Unix / Windows / Linux) and type of component (e.g. open source such as Apache or commercial such as Microsoft Visual Studio).
As an example, below are commands you might run when importing the OS environment from a Red Hat Linux 8.x system and example output (some truncated) that would be captured.
% uname -a
Linux my_hostname 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Fri Oct 16 13:38:49 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
% rpm –query redhat-release
redhat-release-8.2-1.0.el8.x86_64
% hostnamectl
Static hostname: my_hostname
Icon name: computer-vm
Chassis: vm
Machine ID: 0c9bfe4519b44ff7af399c4156c6ded8
Boot ID: 1973bb7d1ddd4da5af4fea2eb782949f
Virtualization: vmware
Operating System: Red Hat Enterprise Linux 8.2 (Ootpa)
CPE OS Name: cpe:/o:redhat:enterprise_linux:8.2:GA
Kernel: Linux 4.18.0-193.28.1.el8_2.x86_64 Architecture: x86-64
% rpm –query –all –info –list
Name : libpng12-0-32bit
Relocations : (not relocatable)
Version : 1.2.31
Vendor : SUSE LINUX Products GmbH, Nuernberg, Germany
Release : 5.10
Build Date : Fri 20 Feb 2009 09:53:02 PM EST
Install Date : Tue 07 Feb 2017 01:21:10 PM EST
Build Host : baur
Group : System/Libraries
Source RPM : libpng12-0-1.2.31-5.10.src.rpm
Size : 165848
License : zlib/libpng License
Signature : RSA/8, Fri 20 Feb 2009 09:53:06 PM EST, Key ID e3a5c360307e3d54
Packager : https://bugs.opensuse.org
URL : https://www.libpng.org/pub/png/libpng.html
Summary : Library for the Portable Network Graphics Format (PNG)
Description :
libpng is the official reference library for the Portable NetworkGraphics format (PNG).
Authors:
——–
Andreas Dilger adilger@enel.ucalgary.ca
Distribution: SUSE Linux Enterprise 11
/usr/lib/libpng12.so.0
/usr/lib/libpng12.so.0.31.0
Organizing Components in VOBs
A simplistic approach to organizing components is to create one VOB for each. Organization and management can be simplified by placing multiple components in VOBs based on various criteria that may include provenance, licensing, degree of sharing, etc. For instance, it might make more sense to keep all components with the same open source license in the same VOB. That might result in a number of VOBs including:
- /vobs/opensrcMIT (MIT/X11 license)
- /vobs/opensrcGNULGPL (GNU Lesser General Public License)
- /vobs/opensrcGNUGPL (GNU General Public License)
Questions to consider when deciding how to organize components include:
- Will this component be shared across multiple projects or teams?
- How often will the component be updated?
- Is it likely we will provide our own fixes/enhancements to the components?
- Do the licensing terms impact the organization?
Note that ClearCase provides fine-grained access control (down to the individual file level) and effective use of the authorization mechanisms can also simplify organization and allow hosting a wide variety of components in VOBs.
Giving thought to your unique requirements and reflecting that in the organization of the components will minimize or eliminate the need to reorganize them in the future.
More Detail about Source Code Components
Source code components have a series of releases, often in parallel, that benefit from careful organization.
Branch / Stream Organization
A sequence of releases of a component should generally be imported on a single branch in the correct sequence. For example, importing components using Semantic Versioning (see https://semver.org/ ), a natural structure would be to have a branch for each MINOR version with each PATCH version advancing on that branch.
Import-only Branches/Streams
Branches used to import source code components should be used only for that purpose. No other changes (other than converting absolute symlinks to relative symlinks) should be allowed on these “import-only” branches. If you need to tweak the component build files (e.g. a ‘Makefile’) to work in your environment (perhaps to utilize ClearCase build auditing) or if your team needs to fix defects or enhance the component, those changes should be made on other branches for that purpose. This will make it easy to import later releases of the components to an import-only branch and use ClearCase’s powerful automatic merge capability to update your branch(es).
Importing Refactored Source Code
Refactoring is a common practice. When a component has been refactored in a way that involved renaming or moving files/directories (e.g. renaming a Java class or package), it can be beneficial to detect this and account for it during the import process.
‘clearfsimport’ will create any files and/or directories that are not already in the VOB. If you use the ‘-rmname’ option, it will also remove any VOB file/directory names (but not the underlying elements) that are no longer in the component being imported. This may be adequate for your needs but detecting the refactoring and moving the file/directory in the VOB as part of the import process will preserve the history of changes and follow the refactoring. This is especially valuable if your team makes defect fixes or enhancements to the component. Merging the refactored import will automatically reflect the refactoring on your team’s branch(es) and preserve the entire history of changes faithfully.
‘clearfsimport’ has a ‘-preview’ flag that will just print the actions it would take. You can examine this output and see if any ‘rmname’ operations (reported as “removed”) have matching ‘mkelem’ operations (reported as “new element” or “new directory”). If so, you can ‘cleartool mv’ the old name(s) to the new name(s) before running the actual import with ‘clearfsimport’.
Conclusion
Building your applications using 3rd-party components maintained in VOBs requires a bit of up-front work. That work usually pays off well by ensuring correct and consistent builds regardless of what is installed on the build or developer machines. It also allows complete authoritative traceability to all 3rd-party dependencies if you use ClearCase’s audited build mechanisms. This all makes it much easier to generate accurate, comprehensive SBOMs.
These benefits can also be extended to build tools which will be the topic of a future blog post in this series.