API Connect

 View Only

Introducing IBM API Connect Config Sync

By Matt Roberts posted Wed December 13, 2023 10:10 AM

  

Today I’m excited to announce the release of the Technical Preview for the new IBM API Connect Config Sync capability, which provides the ability to replicate consumer-side data from an API Connect catalog to one or more target catalogs, in order to support more advanced and flexible scenarios for hybrid deployments, high availability or disaster recovery.

The creation of this capability has been a personal passion project of mine through this year, triggered by a series of conversations that I’ve had with customers across multiple industries and geographies, so I’m excited to see it graduate into Tech Preview, and I look forward to you sharing your input for where and how we should evolve this capability in future!

What is API Connect Config Sync?

Config Sync is a utility that automates the replication of consumer-side data from an API Connect catalog to one or more target catalogs.

This enables users to configure a second (or more) independent API Connect deployment so that API calls can be served by either API Connect cluster interchangeably – which enables a range of hybrid deployments and high-availability or disaster recovery style scenarios.

Currently the user is responsible for configuring the provider-side data in the second cluster, including deployment of APIs and Products that is typically automated via a CI/CD pipeline or equivalent process. Config Sync then handles replication of the consumer state such as Consumer Organizations, Applications, Credentials and Subscriptions that is typically outside the control of the API provider.

Config Sync can be run as frequently as required, and detects the differences between the replicated resource types in the source and target catalogs so that it efficiently replicates only the new changes that have been made in the source instance.

This incremental replication means the utility can be run as frequently as needed - either as a standalone utility, or as a scheduled CronJob inside Kubernetes or OpenShift so that it can be invoked on a regular timed interval such as every 5 or 10 minutes.

Why would I use API Connect Config Sync?

Config Sync helps to enable a range of hybrid cloud API topologies as illustrated in the diagrams below:

  • Independent Disaster Recovery environment – enabling a second API Connect deployment to be kept in sync with the primary on a much more frequent (lower RPO) basis, and with a lower operational overhead
  • Software + SaaS hybrid scenarios – enables different API Connect form factors to be integrated such as for burst-to-cloud and side-by-side migration from on-premises to SaaS
  • Geo-replication – enabling independent API Connect deployments in multiple geographies to maintain near-live subscription and credential state
  • Side-by-side version upgrade – providing an additional option for migrating between different versions of API Connect, such as from v10.0.1 to v10.0.5

How do I get access?

IBM API Connect Config Sync is currently in Tech Preview. 

Show me!

Are you interested to see API Connect Config Sync live in action?  then watch this short video!

12 comments
106 views

Permalink

Comments

Matt Fawcett
Mon July 29, 2024 09:43 AM

Hi @ganesh kaki,

ConfigSync is not supported on the version of APIC that you are using (10.0.1.1).  As you can see from the error snippet provided, the 'expand' query parameter that configSync expects is not supported on the 10.0.1.1 version of the API.

As per our documented support matrix the lowest version that we support is 10.0.1.15 - https://www.ibm.com/docs/en/api-connect/10.0.5.x_lts?topic=imac-technical-preview-using-config-sync-replicate-consumer-side-catalog-data.

Version 10.0.1 of APIC has been EOS for quite some time now so i would strongly recommended moving off of it as soon as possible.

ganesh kaki
Thu July 18, 2024 04:47 AM

Hi Matt,

I'm facing below error while doing the sync. Would you be able to help me?

{"level":"error","timestamp":"2024-07-18T11:40:08.441+0300","msg":"API response","url":"https://abc.com/api/catalogs/pwa/prod/subscriptions?fields=id,name,title,summary,product_url,consumer_org_url,product,plan,plan_title,metadata,app_url,state,url,task_urls&expand=product","responseBody":"{\"status\":400,\"message\":[\"The request contains an unknown query parameter 'expand' with the value 'product'.\"]}"}

APIC Version: 10.0.1.1-1423-eus

Note: Modified the actual endpoint with abc.com. 

Matt Roberts
Mon May 20, 2024 04:11 AM

Hi @ganesh kaki - ok, I'm pleased you were able to track down the cause for the issue you saw.

Currently Config Sync replicates only the consumer-side objects such as Applications, Subscriptions etc and it's expected that you use CI/CD or other technique (outside of Config Sync) to replicate the provider-side resources including APIs, Products, TLS, OAuth profiles etc. We have had interest from other customers as well about expanding Config Sync to other object types in future, but nothing confirmed at this point.

Regards, Matt.

ganesh kaki
Sun May 19, 2024 03:17 AM

Hi Matt, I found the DataPower clsuter peering was broken and the sync is working as expected. I can see one of the DataPower box is holding all the required details. 

To synchronise resources such as TLS, OAuth profiles and etc, Can I expect any feature from config sync? Or it should go as part of CI/CD pipeline only?

Matt Roberts
Wed May 15, 2024 04:09 AM

Hi @ganesh kaki - thanks for your comment; from your description it sounds that the apps + subscriptions have been replicated successfully by Config Sync, but there were not the matching Products/Plans present in the Target system to be able to establish the subscription to the plans.

As illustrated in the diagram in the blog above, the expectation is that the APIs+Products/Plans are recreated in the Target instance using CI/CD or other mechanism - they are not replicated by Config Sync currently, so the first thing to check is whether you have done that recreation of the Products successfully in the Target, with the same name + version as in the Source.

If you confirm the APIs+Products in the Target look to match then the next step is to look in the log of the replication job (e.g. pod log on Kube/OpenShift) to see if there are warning/error messages are written out when the replication takes place.

Regards, Matt.

ganesh kaki
Tue May 14, 2024 10:29 AM

Hi Matt, 

I successfully performed synchronisation using configsync. But only one challenge I have. 

I'm facing 403, api not under any plan. When I checked the datapower gateway subscriptions, I can able to see all subscriptions which is expected. But plans are empty. Is it gonna take time to sync or it's near real time. I can able to see all applications and subscriptions. 

Matt Roberts
Wed February 14, 2024 08:00 AM

Hi @ganesh kaki - thanks for your questions.

In this scenario the Config Sync tech preview provides the ability to replicate consumer state from the primary (e.g. Production) catalog to the secondary (e.g. DR) catalog on a frequent basis such as every 10 minutes - which provides advantages such as lower RPO compared to the typical backup/restore mechanism for replicating state data to a DR environment.

It's important to note that the replication is expected to be in one direction only currently, so if the Production failed you would fail over API traffic to the DR environment, but would need to recover the Production environment to get the primary back in place.  We are intending a future update to allow "reverse sync" such that if you allow write updates in the DR environment while it was serving the live traffic (such as creating new subscriptions) those updates can be re-synchronized back to the original Primary (Production) with affecting the existing subscription state, but that reverse sync capability is not part of the Tech Preview that is available today.

ganesh kaki
Wed February 14, 2024 05:15 AM

I'm trying to explore how can I sync the data between Production and DR. 

One of the scenarios I am checking, I wanted to maintain Production & DR environments in two different data centers like US and UK. DR is going to be same setup as Prod (different project and backup directories).

I'm just seeing how to achieve this.

Can you please provide your thoughts around this. 

Matt Roberts
Tue January 16, 2024 11:41 AM

Hi @Tyler Nelson - thanks, I'm pleased you think this will be useful!

The endpoint information for the source and target API Connect instances are configured as environment variables that are read by the runtime so that you can flexibly control the connectivity.  See Setting environment variables for Config Sync in our public documentation for details.

Matt Roberts
Tue January 16, 2024 11:39 AM

Hi @Purush Das - thanks for your comments.  We are considering the appropriate GA timeline based on customer input, so I've noted that you are interested, thanks!

Tyler Nelson
Tue January 16, 2024 11:07 AM

Very nice demo!  This looks like it will be useful functionality for our Hybrid API Management and possibly DR.

I'm interested in how Config Synch was configured to target the eu-west APIC deployment?

Purush Das
Wed January 10, 2024 06:13 PM

This is a great and a very useful capability for DR.

Looking forward to implement it for independent DR environment. What's the GA release date?

Portal endpoint of Deployment B cannot be same as Deployment A.

During the DR event, we need to change the Dev Portal endpoint after failover to Deployment B to make it same as Deployment A. Otherwise, dev portal endpoint will change and customers does not like changing the dev Portal endpoint after failover.