All:

I think I'm zeroing in on something here as I test more.​​​​ First, by setting up a test CM and enabling logging, I can see that even when the connection hangs, the connection manager logs the attempt and onstat -g cmsm shows the connection count go up. So it's making it as far as the CM and hanging and eventually failing on the return trip, after the CM has done the forwarding. I've replicated the behavior with CMs running 4.10.FC12, 4.10.FC14 and 4.50.FC3, so it doesn't appear to have to do with the CM version. Instead, it has to do with the DB server version. Works in 12.10.FC12, not in 12.10.FC14 (I haven't tested FC13 yet).

Here's what I've been able to replicate in controlled tests:

1. If the CM forwards to a different INFORMIXSERVER than the originating one, the connection succeeds. In my tests, if the query originates on the primary and the remote portion hits the HDR secondary, that works, as does the converse. If the query originates on the primary and the CM redirects back to the primary for the remote portion, or if it originates on the HDR secondary and the CM redirects back to the HDR secondary, that's where I _sometimes_ have trouble.
2. I _think_ that Mr. Weis' duplicate entry thing is at play, but I haven't been able to safely replicate that condition.
3. In addition, it looks like the trouble begins when the INFORMIXSERVER value in the sqlhosts file on the originating DB server does not actually match the SLA name on the CM. (We've used aliases in sqlhosts for backward compatibility to emulate decommissioned servers so that old code doesn't need to be rebuilt, for example.) So when the [dbsvr] in SELECT * FROM [db]@[dbsvr]:[table] does not match the SLA name on the CM _and_ the CM redirects back to the originating DB server, the connection successfully gets to the CM, and the CM's redirect attempt hangs for around 30 seconds before returning -908/-27001.
4. HOWEVER, if we reconfigure the SLA as MODE=PROXY and rerun the test listed in 3 above, instead of hanging it fails immediately with "761: INFORMIXSERVER does not match either DBSERVERNAME or DBSERVERALIASES." But only if the redirect points right back to the originating server; if the target server is different than the originating server, the connection succeeds irrespective of whether [dbsvr] and the SLA name match.
5. Repeating again, all of the above tests connect successfully in FC12.

All:

I think I'm zeroing in on something here as I test more.​​​​ First, by setting up a test CM and enabling logging, I can see that even when the connection hangs, the connection manager logs the attempt and onstat -g cmsm shows the connection count go up. So it's making it as far as the CM and hanging and eventually failing on the return trip, after the CM has done the forwarding. I've replicated the behavior with CMs running 4.10.FC12, 4.10.FC14 and 4.50.FC3, so it doesn't appear to have to do with the CM version. Instead, it has to do with the DB server version. Works in 12.10.FC12, not in 12.10.FC14 (I haven't tested FC13 yet).

Here's what I've been able to replicate in controlled tests:

1. If the CM forwards to a different INFORMIXSERVER than the originating one, the connection succeeds. In my tests, if the query originates on the primary and the remote portion hits the HDR secondary, that works, as does the converse. If the query originates on the primary and the CM redirects back to the primary for the remote portion, or if it originates on the HDR secondary and the CM redirects back to the HDR secondary, that's where I _sometimes_ have trouble.
2. I _think_ that Mr. Weis' duplicate entry thing is at play, but I haven't been able to safely replicate that condition.
3. In addition, it looks like the trouble begins when the INFORMIXSERVER value in the sqlhosts file on the originating DB server does not actually match the SLA name on the CM. (We've used aliases in sqlhosts for backward compatibility to emulate decommissioned servers so that old code doesn't need to be rebuilt, for example.) So when the [dbsvr] in SELECT * FROM [db]@[dbsvr]:[table] does not match the SLA name on the CM _and_ the CM redirects back to the originating DB server, the connection successfully gets to the CM, and the CM's redirect attempt hangs for around 30 seconds before returning -908/-27001.
4. HOWEVER, if we reconfigure the SLA as MODE=PROXY and rerun the test listed in 3 above, instead of hanging it fails immediately with "761: INFORMIXSERVER does not match either DBSERVERNAME or DBSERVERALIASES." But only if the redirect points right back to the originating server; if the target server is different than the originating server, the connection succeeds irrespective of whether [dbsvr] and the SLA name match.
5. Repeating again, all of the above tests connect successfully in FC12.

Cart/Horse confusion. The bug was entered in response to my tech support case. ;) HCL is having internal arguments as to whether or not this should be "expected behavior." Which is silly because even if it's not supposed to work, it should fail immediately and with a different error message.

I've replicated the behavior on FC12, FC13 and FC14. It doesn't necessarily have to be an alias, though that's the easiest way to reproduce. Can tweak my repro case and post it if you want to try it yourself.

All:

I think I'm zeroing in on something here as I test more.​​​​ First, by setting up a test CM and enabling logging, I can see that even when the connection hangs, the connection manager logs the attempt and onstat -g cmsm shows the connection count go up. So it's making it as far as the CM and hanging and eventually failing on the return trip, after the CM has done the forwarding. I've replicated the behavior with CMs running 4.10.FC12, 4.10.FC14 and 4.50.FC3, so it doesn't appear to have to do with the CM version. Instead, it has to do with the DB server version. Works in 12.10.FC12, not in 12.10.FC14 (I haven't tested FC13 yet).

Here's what I've been able to replicate in controlled tests:

1. If the CM forwards to a different INFORMIXSERVER than the originating one, the connection succeeds. In my tests, if the query originates on the primary and the remote portion hits the HDR secondary, that works, as does the converse. If the query originates on the primary and the CM redirects back to the primary for the remote portion, or if it originates on the HDR secondary and the CM redirects back to the HDR secondary, that's where I _sometimes_ have trouble.
2. I _think_ that Mr. Weis' duplicate entry thing is at play, but I haven't been able to safely replicate that condition.
3. In addition, it looks like the trouble begins when the INFORMIXSERVER value in the sqlhosts file on the originating DB server does not actually match the SLA name on the CM. (We've used aliases in sqlhosts for backward compatibility to emulate decommissioned servers so that old code doesn't need to be rebuilt, for example.) So when the [dbsvr] in SELECT * FROM [db]@[dbsvr]:[table] does not match the SLA name on the CM _and_ the CM redirects back to the originating DB server, the connection successfully gets to the CM, and the CM's redirect attempt hangs for around 30 seconds before returning -908/-27001.
4. HOWEVER, if we reconfigure the SLA as MODE=PROXY and rerun the test listed in 3 above, instead of hanging it fails immediately with "761: INFORMIXSERVER does not match either DBSERVERNAME or DBSERVERALIASES." But only if the redirect points right back to the originating server; if the target server is different than the originating server, the connection succeeds irrespective of whether [dbsvr] and the SLA name match.
5. Repeating again, all of the above tests connect successfully in FC12.

Cart/Horse confusion. The bug was entered in response to my tech support case. ;) HCL is having internal arguments as to whether or not this should be "expected behavior." Which is silly because even if it's not supposed to work, it should fail immediately and with a different error message.

I've replicated the behavior on FC12, FC13 and FC14. It doesn't necessarily have to be an alias, though that's the easiest way to reproduce. Can tweak my repro case and post it if you want to try it yourself.

All:

I think I'm zeroing in on something here as I test more.​​​​ First, by setting up a test CM and enabling logging, I can see that even when the connection hangs, the connection manager logs the attempt and onstat -g cmsm shows the connection count go up. So it's making it as far as the CM and hanging and eventually failing on the return trip, after the CM has done the forwarding. I've replicated the behavior with CMs running 4.10.FC12, 4.10.FC14 and 4.50.FC3, so it doesn't appear to have to do with the CM version. Instead, it has to do with the DB server version. Works in 12.10.FC12, not in 12.10.FC14 (I haven't tested FC13 yet).

Here's what I've been able to replicate in controlled tests:

1. If the CM forwards to a different INFORMIXSERVER than the originating one, the connection succeeds. In my tests, if the query originates on the primary and the remote portion hits the HDR secondary, that works, as does the converse. If the query originates on the primary and the CM redirects back to the primary for the remote portion, or if it originates on the HDR secondary and the CM redirects back to the HDR secondary, that's where I _sometimes_ have trouble.
2. I _think_ that Mr. Weis' duplicate entry thing is at play, but I haven't been able to safely replicate that condition.
3. In addition, it looks like the trouble begins when the INFORMIXSERVER value in the sqlhosts file on the originating DB server does not actually match the SLA name on the CM. (We've used aliases in sqlhosts for backward compatibility to emulate decommissioned servers so that old code doesn't need to be rebuilt, for example.) So when the [dbsvr] in SELECT * FROM [db]@[dbsvr]:[table] does not match the SLA name on the CM _and_ the CM redirects back to the originating DB server, the connection successfully gets to the CM, and the CM's redirect attempt hangs for around 30 seconds before returning -908/-27001.
4. HOWEVER, if we reconfigure the SLA as MODE=PROXY and rerun the test listed in 3 above, instead of hanging it fails immediately with "761: INFORMIXSERVER does not match either DBSERVERNAME or DBSERVERALIASES." But only if the redirect points right back to the originating server; if the target server is different than the originating server, the connection succeeds irrespective of whether [dbsvr] and the SLA name match.
5. Repeating again, all of the above tests connect successfully in FC12.

I saw this APAR and I am chasing this as well!

Regards,

David.

Cart/Horse confusion. The bug was entered in response to my tech support case. ;) HCL is having internal arguments as to whether or not this should be "expected behavior." Which is silly because even if it's not supposed to work, it should fail immediately and with a different error message.

I've replicated the behavior on FC12, FC13 and FC14. It doesn't necessarily have to be an alias, though that's the easiest way to reproduce. Can tweak my repro case and post it if you want to try it yourself.

All:

I think I'm zeroing in on something here as I test more.​​​​ First, by setting up a test CM and enabling logging, I can see that even when the connection hangs, the connection manager logs the attempt and onstat -g cmsm shows the connection count go up. So it's making it as far as the CM and hanging and eventually failing on the return trip, after the CM has done the forwarding. I've replicated the behavior with CMs running 4.10.FC12, 4.10.FC14 and 4.50.FC3, so it doesn't appear to have to do with the CM version. Instead, it has to do with the DB server version. Works in 12.10.FC12, not in 12.10.FC14 (I haven't tested FC13 yet).

Here's what I've been able to replicate in controlled tests:

1. If the CM forwards to a different INFORMIXSERVER than the originating one, the connection succeeds. In my tests, if the query originates on the primary and the remote portion hits the HDR secondary, that works, as does the converse. If the query originates on the primary and the CM redirects back to the primary for the remote portion, or if it originates on the HDR secondary and the CM redirects back to the HDR secondary, that's where I _sometimes_ have trouble.
2. I _think_ that Mr. Weis' duplicate entry thing is at play, but I haven't been able to safely replicate that condition.
3. In addition, it looks like the trouble begins when the INFORMIXSERVER value in the sqlhosts file on the originating DB server does not actually match the SLA name on the CM. (We've used aliases in sqlhosts for backward compatibility to emulate decommissioned servers so that old code doesn't need to be rebuilt, for example.) So when the [dbsvr] in SELECT * FROM [db]@[dbsvr]:[table] does not match the SLA name on the CM _and_ the CM redirects back to the originating DB server, the connection successfully gets to the CM, and the CM's redirect attempt hangs for around 30 seconds before returning -908/-27001.
4. HOWEVER, if we reconfigure the SLA as MODE=PROXY and rerun the test listed in 3 above, instead of hanging it fails immediately with "761: INFORMIXSERVER does not match either DBSERVERNAME or DBSERVERALIASES." But only if the redirect points right back to the originating server; if the target server is different than the originating server, the connection succeeds irrespective of whether [dbsvr] and the SLA name match.
5. Repeating again, all of the above tests connect successfully in FC12.

All:

I think I'm zeroing in on something here as I test more.​​​​ First, by setting up a test CM and enabling logging, I can see that even when the connection hangs, the connection manager logs the attempt and onstat -g cmsm shows the connection count go up. So it's making it as far as the CM and hanging and eventually failing on the return trip, after the CM has done the forwarding. I've replicated the behavior with CMs running 4.10.FC12, 4.10.FC14 and 4.50.FC3, so it doesn't appear to have to do with the CM version. Instead, it has to do with the DB server version. Works in 12.10.FC12, not in 12.10.FC14 (I haven't tested FC13 yet).

Here's what I've been able to replicate in controlled tests:

1. If the CM forwards to a different INFORMIXSERVER than the originating one, the connection succeeds. In my tests, if the query originates on the primary and the remote portion hits the HDR secondary, that works, as does the converse. If the query originates on the primary and the CM redirects back to the primary for the remote portion, or if it originates on the HDR secondary and the CM redirects back to the HDR secondary, that's where I _sometimes_ have trouble.
2. I _think_ that Mr. Weis' duplicate entry thing is at play, but I haven't been able to safely replicate that condition.
3. In addition, it looks like the trouble begins when the INFORMIXSERVER value in the sqlhosts file on the originating DB server does not actually match the SLA name on the CM. (We've used aliases in sqlhosts for backward compatibility to emulate decommissioned servers so that old code doesn't need to be rebuilt, for example.) So when the [dbsvr] in SELECT * FROM [db]@[dbsvr]:[table] does not match the SLA name on the CM _and_ the CM redirects back to the originating DB server, the connection successfully gets to the CM, and the CM's redirect attempt hangs for around 30 seconds before returning -908/-27001.
4. HOWEVER, if we reconfigure the SLA as MODE=PROXY and rerun the test listed in 3 above, instead of hanging it fails immediately with "761: INFORMIXSERVER does not match either DBSERVERNAME or DBSERVERALIASES." But only if the redirect points right back to the originating server; if the target server is different than the originating server, the connection succeeds irrespective of whether [dbsvr] and the SLA name match.
5. Repeating again, all of the above tests connect successfully in FC12.

I'm attaching a more robust and more generalized reproduction case here. Create the instance ids__3740811 on host1 using the attached onconfig. Create the instance ids__3740811__hdr on host2 using that onconfig, setting it up as an HDR secondary for ids__3740811. Create a connection manager on host3 using the attached cmconfig file. All three get the same sqlhosts file, also attached. Ensure that port 42001 is open on all firewalls.

With that all configured, run pretest.sh (attached) and then test.sh (also attached) from either host1 or host2 (doesn't matter which). You'll see that of the five commands it runs, four of them work, and two out of three that reference the SLA alias tj_kaboom.

Having run those, now reconfigure the cm__3740811 SLA to redirect to the HDR rather than to the primary and rerun both scripts; you'll see that all of the pretest commands still work, while the commands from test.sh have swapped in terms of which succeeds and which fails. In total, six commands that reference tj_kaboom. Four of them work, two fail. And the failures take approximately one minute.

To hear development tell it, the problem is not the two scenarios which fail, but the four which succeed.

Even if we accepted their suggestion -- obviously I do not -- it should fail right away and consistently rather than hanging up for a minute doing who knows what.

One additional note. If you monitor the CM log file while running these tests, you'll see that the CM actually receives the connection request and forwards it even in the cases where it fails. So the issue seems to be on the DB server side.

Now hear this! Bug has allegedly been identified and a fix coded. I've requested a special build. If you need one, here's the APAR:

APAR IT33019 - SPORADIC -908 / -27001 WHEN CONNECTING TO SLA ALIAS

Which is the exact bug @Eric Rowell identified earlier.

Now hear this! Bug has allegedly been identified and a fix coded. I've requested a special build. If you need one, here's the APAR:

APAR IT33019 - SPORADIC -908 / -27001 WHEN CONNECTING TO SLA ALIAS

Which is the exact bug @Eric Rowell identified earlier.