I'm not sure what is meant by "on an external windows server is the SP behind the RP. " ?
Do you have IDP and SP functionality implemented both on ISVA, or is the SP functionality implemented using different SAML vendor/technology?
I'd recommend getting some ISVA Federation traces (com.tivoli.am.fim.*=ALL) on the IdP side, to see if you can identify where things go wrong.
Do you see it for all browsers?
There was a Chrome/Chromium browser change, that might some some problems with SP initiated SSO (https://www.ibm.com/support/pages/browser-changes-samesite-cookie-handling-and-ibm-security-access-manager)
If IDP & SP are both implemented on ISVA, are they both on the same appliance (= same Federation runtime)? As the Federated SSO functionality relies on session cookies, there is potential for getting this mixed up during the SSO flow.
https://www.ibm.com/docs/en/sva/10.0.2?topic=overview-known-limitations
The STS chain mapping created internally for Identity Provider and Service Provider will have identical ‘issuer’ and ‘applies to’ which can lead to unexpected behavior during runtime flow.
(traces will probably hold the truth :-) )
#Support#SupportMigration#Verify