API Connect

 View Only

 How to remove binary data output from `apic products:get`?

Tyler Courtney Hayward's profile image
Tyler Courtney Hayward posted 06/09/26 05:08 AM

Hello all,

We have a few APIs that have WSDLs and XSDs that are zipped together and published along with the API product file and the API yaml.

We need to wrangle some data out of APIC using the apic CLI tool. Currently we can get everything we need with `apic products:get --scope catalog -s $SERVER -c $CATALOG -o $ORG --format json --output - $PRODUCT` and then parse the output. 

This works great until we run into one of our API products that includes a zip file. In that case, `apic products:get` returns the zip file as a binary blob before the API's swagger definition.

Does anyone have an idea how we can strip off the binary blob so we can parse the remaining json? This is all on the CLI, so jq, sed, cut, and other *nix built-ins are all fair game to use.

The data we are mining is the API product and version and all the contained APIs and their respective versions. `apic products:get` makes this very easy to parse out so we can track this data through all our environments, but other approaches and ideas are most welcome.

To give you an idea of what we are dealing with, here is a image of this output opened in a text editor. I've cut it off just after the output changes from binary to json:

image

Thanks in advance!

Kind regards,

Tyler C. Hayward

Chris Dudley's profile image
Chris Dudley

Check the clidocs to be sure but I’m fairly sure there’s a “fields” attribute you can use to only return certain fields in the response. You could then just list the ones you need and avoid the one with the binary data in it?

Tyler Courtney Hayward's profile image
Tyler Courtney Hayward

Chris,

Thanks for the reply. I see the fields option, but where can I find a valid list of field names? The only given example is "add(catalog_product)", which does not make a whole lot of sense to me in trying to suss out what other valid fields there are.

William McEnery's profile image
William McEnery

Hi Tyler,

By default the products:get toolkit command makes multiple requests to APIM backend, the first is to get the product and then it gets the APIs associated with that product. One issue is if you use the --fields flag/query param, the toolkit will only make the request for the product. You would then need to get the APIs separately using apis:get command.If you are only interested in the name and version for the products/APIs you can get all this info in the catalog_product property, so can you try adding --fields "catalog_product" flag to you command and use jq to extract the values

FYI on the fields flag/query parameter, you can use --fields "redact(<fields-to-redact>)" to remove fields from the default output or --fields "add(<fields-to-add>)" to include additional fields to the default output, to return only certain fields something like  --fields "name,version"
The fields should be provided as a list separated by "," e.g. --fields "redact(name,version)" or --fields "add(catalog_product,product)"or --fields "name,version"

To see the fields, see our REST API docs, here is the products:get endpoint, check under the Responses section: https://apic-api.apiconnect.ibmcloud.com/v10/#/IBMAPIConnectPlatformProviderAPI_200/operation/%2Fcatalogs%2F{org}%2F{catalog}%2Fproducts%2F{product-name}%2F{product-version}/get

David Dhanjal's profile image
David Dhanjal

Here’s the whole approach as one flow:


#!/bin/bash
SERVER=...
CATALOG=...
ORG=...
PRODUCT=...

# 1. Pull the raw output (binary zip blob + JSON, concatenated)
apic products:get --scope catalog -s $SERVER -c $CATALOG -o $ORG \
  --format json --output - $PRODUCT > raw.bin

# 2. Find the byte offset of the LAST NUL byte in the file.
#    Zip data is full of 0x00 bytes; valid JSON text never contains them,
#    so this marks the end of the binary blob.
LAST_NUL=$(grep -aboP '\x00' raw.bin | tail -1 | cut -d: -f1)

# 3. Take everything after that...
tail -c +$((LAST_NUL + 2)) raw.bin > tail.bin

# 4. ...then find the first '{' in what remains. That's the true start of the JSON.
FIRST_BRACE=$(grep -abo '{' tail.bin | head -1 | cut -d: -f1)

tail -c +$((FIRST_BRACE + 1)) tail.bin > clean.json

# 5. Now parse normally
jq '{product: .name, version: .version, apis: [.apis[] | {name: .name, version: .version}]}' clean.json

This works regardless of which API in the product is the one carrying the WSDL/XSD zip, the NUL-byte boundary just strips off whatever binary precedes the JSON, leaving you with the same clean document structure your current jq parsing already expects.

One thing to verify on a real sample: confirm there’s only one binary blob per product output (i.e., one API has the zip attachment, but the JSON metadata for the whole product, including all contained APIs and versions comes as a single trailing block). If that’s the case, this script is a drop-in replacement so run it for every product, and for products without any embedded zip, LAST_NUL will simply come back empty/0 and the script degrades gracefully to just passing the whole file through (you may want to add a check: if [ -z "$LAST_NUL" ]; then cp raw.bin clean.json; fi).

If instead each contained API gets its own binary+JSON segment back-to-back, you’d need to split on every NUL-run-to-{ transition and jq each segment separately, then merge results. 

Tyler Courtney Hayward's profile image
Tyler Courtney Hayward

Thank you both William and David! Your answers have got me well on my way to sorting out my scripts.