First 8.7 is old, latest release is 10.1.1. Please upgrade if possible, lots of new features and bug releases, 8.7 is still supported but sometimes the answer from support will be to upgrade if we're hitting a known issue that's been fixed in a later release.
For this specific issue this technote applies
"How to record a non-browser application in Rational Performance Tester"
https://www.ibm.com/support/pages/node/6251929
Not sure what the application in question is. Does it even generate one of the protocols that RPT supports, e.g. HTTP or socket?
If there's s publically available version of this application I could do a recording against I can experiment recording it.
If you get stuck please open a Salesforce ticket
Thank you
#RationalPerformanceTester#Support#SupportMigration