Auditability and Traceability in IBM Content Assistant
Traceability
IBM Content Assistant allows FileNet Content Manager users to ask questions in an assistant UI and get generative AI responses informed by the content of their business documents. When a user asks a question, IBM Content Assistant performs a search to find the document excerpts that are most relevant to the users question, and then uses those excerpts, along with the users question, to ask Watsonx.ai to generate an answer.
If the user wants to know what information that answer was based on, they can easily see this by clicking on the down arrow below the answer. This brings up a list of the documents that the answer was based on, and show the specific excerpt(s) from each document that were used to generate the answer:
This provides traceability: the ability to trace the answer back to the source. This is one of the key features of IBM Content Assistant.
Auditability
But what if I am an administrator, and I want to audit the questions that one of my users has been asking, and the responses from IBM Content Assistant? This capability is provided by IBM Content Assistant as well. A record of every question and it’s response is maintained in the Content Engine object store. To understand how this works, we will first need to drill down a little bit into how IBM Content Assistant performs its magic within the FileNet Content Engine server.
Background: The Genai Query objects
The IBM Content Assistant AddOn introduces a set of new classes that can be used by Content Engine API callers to initiate a generative AI query. These classes include:
- GenaiBaseQuery: Given a prompt, returns a response from Watsonx.ai (not based on any documents)
- GenaiDocumentQuery: Given a prompt and a reference to a document, returns a response from Watsonx.ai based on the contents of that document
- GenaiMultiDocumentQuery: Given a prompt and a list of references to documents, returns a response from Watsonx.ai based on the contents of those documents
- GenaiVectorQuery: Given a prompt, searches an object store for the document excerpts most closely matching the prompt, and returns a response from Watsonx.ai based on the contents of those document excerpts
- GenaiAdhocSummary: Given references to one or more documents, returns a summary from Watsonx.ai based on the contents of those documents
More details on these classes can be found here: https://www.ibm.com/docs/en/content-assistant?topic=applications-genai-query-classes
When an instance of one of these classes is created, Content Engine will make a call to the IBM Content Assistant SaaS service to find the document excerpts most closely related to the prompt, and make the inference call, with the excerpts, as well as the original prompt, to Watsonx.ai. The prompt, document excerpts, and response are then saved in the properties of the Genai Query object:
- GenaiLLMPrompt: The prompt entered by the end user
- GenaiLLMResponse: The response generated by Watsonx.ai
- GenaiVectorChunks: JSON containing the document chunks that were submitted to the LLM as context for the Watsonx.ai query, including metadata on the documents that they came from
- DateCreated: The date and time of the request
- Creator: The username of the user who created the request
These query objects are saved permanently in the object store. They are only accessible by the user who created the query object, or by an object store administrator. A non-administrator can never see anyone’s queries other than their own.
Querying the Query objects
The query objects described in the previous section provide an audit record of all generative AI queries performed by any IBM Content Assistant user, as well as traceability back to the business documents that were used to generate the response. An object store administrator can use ACCE to query these objects. This provides the ability to fulfill auditing requirements.
Process to execute a query in ACCE
To execute a query in the Content Engine ACCE tool, select the Search node in the navigation bar for the object store that you wish to search. Then click on ‘New Object Store Search’. You can then select one of the classes mentioned above (for example, GenaiMultiDocumentQuery) from the Class dropdown, and use the controls on the Simple Search screen to refine your search.
Alternatively, you can paste one of the queries in the section below, into the SQL Statement in the SQL View tab, and then edit it as necessary.
Press the Run button when you are done with your changes, to view the query results.
For more information on the Content Engine query syntax, see: https://www.ibm.com/docs/en/filenet-p8-platform/5.6.0?topic=reference-relational-queries
For more information on issuing Content Engine queries programmatically, see; https://www.ibm.com/docs/en/filenet-p8-platform/5.6.0?topic=queries-working#query_procedures__query_syntax
Non-Admin User Examples
The examples in this section illustrate queries that a non-admin user can issue to find their own queries. Note that when logged in as a non-admin user, there is no need to filter by user, or display the user who performed the query, since non-admin users can only see their own queries.
Get all of my document specific queries
For this, we will retrieve the date/time of the query, the question that I asked, and the response from Watson, sorted by most recent first:
SELECT [This],[DateCreated],[GenaiLLMPrompt], [GenaiLLMResponse]
FROM [GenaiMultiDocumentQuery]
ORDER BY DateCreated DESC
While the Content Assistant AddOn includes a GenaiDocumentQuery class to submit queries against a single document, IBM Navigator uses the GenaiMultiDocumentQuery class to submit all queries scoped to one or more documents.
Get all of my whole object store queries for the past 90 days
In this example, we also ask for the property containing the set of document chunks that was used to generate the Watsonx.ai response: GenaiVectorChunks.
SELECT [This], [DateCreated], [GenaiLLMPrompt], [GenaiLLMResponse], [GenaiVectorChunks]
FROM [GenaiVectorQuery]
WHERE DateCreated > Now() - Timespan(90, 'Days')
ORDER BY DateCreated DESC
Note that the GenaiVectorChunks property contains a JSON structure with the chunks of all documents that were passed to Watson with the query. This JSON data structure can be large (over 10K characters in length). This property can be retrieved when querying the GenaiVectorQuery, GenaiDocumentQuery, or GenaiMultiDocumentQuery classes.
Admin User Examples
Some examples of CE queries for Genai Query objects are shown below.
Retrieve all queries for a given user
SELECT [This], [Creator], [DateCreated], [GenaiLLMPrompt], [GenaiLLMResponse]
FROM [GenaiBaseQuery] WITH INCLUDESUBCLASSES
WHERE Creator = 'jraby@us.ibm.com'
ORDER BY DateCreated DESC
Retrieve all queries for a given user on a given day
SELECT [This], [Creator], [DateCreated], [GenaiLLMPrompt], [GenaiLLMResponse]
FROM [GenaiBaseQuery] WITH INCLUDESUBCLASSES
WHERE Creator = 'jraby@us.ibm.com' AND
DateCreated > 2025-04-21T00:00:00-07:00 AND
DateCreated < 2025-04-22T00:00:00-07:00
ORDER BY DateCreated DESC
Retrieve all adhoc summary request for a given user
SELECT [This], [Creator], [DateCreated], [GenaiContextDocuments], [GenaiLLMResponse]
FROM [GenaiAdhocSummary]
WHERE Creator = 'jraby@us.ibm.com'
ORDER BY DateCreated DESC
Retrieve all queries against a specific document
SELECT [This],[DateCreated],[GenaiLLMPrompt], [GenaiContextDocuments], [GenaiLLMResponse]
FROM [GenaiMultiDocumentQuery]
WHERE {95433990-0001-C2AD-AFDE-9198C64D35F9} IN GenaiContextDocuments
ORDER BY DateCreated DESC
Note that the above query searches against the GenaiMultiDocumentQuery class only. This is sufficient when the only client is IBM Content Navigator, as Navigator always uses GenaiMultiDocumentQuery to filter by document, even when only a single document is selected. If an environment uses a different client which queries against the GenaiDocumentQuery class, then the following query would be necessary to find those instances:
SELECT [This],[DateCreated],[GenaiLLMPrompt], [GenaiLLMResponse]
FROM [GenaiDocumentQuery]
WHERE GenaiContextDocument=OBJECT('{95433990-0001-C2AD-AFDE-9198C64D35F9}')
ORDER BY DateCreated DESC
Conclusion
By making the document excerpts that were used to produce the generative AI response available, IBM Content Assistant provides traceability back to the documents that generated the answer.
By saving the Genai query objects that are used to provide Watsonx.ai responses, and by providing the ability to query these Genai query objects, IBM Content Assistant provides auditability for all usage of business documents to provide generative AI responses.