Informix

All:

I wanted to make you aware of something we've hit a couple of times and also find out if someone else has encountered it. Twice in the past few months, I've had DR Ping faliures on our HDR primary which seem to have left the cluster in an unstable state. It's as if the primary lost track of the secondary but doesn't realize that it's not there. In online.log on the primary, it looks like this:

09:33:14  DR: Needed to send a ping message but failed. 109:33:44  DR: Needed to send a ping message but failed. 109:34:14  DR: Needed to send a ping message but failed. 109:34:44  DR: Needed to send a ping message but failed. 109:34:54  DR: ping timeout

On the secondary, it looks like this:

2022-05-23 09:35:04  DR: ping timeout2022-05-23 09:35:04.639  DR: Receive error2022-05-23 09:35:04.640  dr_secrcv thread : asfcode = -25582: oserr = 4: errstr = : Network connection is broken. System error = 4.2022-05-23 09:35:04.640  DR_ERR set to -12022-05-23 09:35:04.640  DR: Receive Btree error2022-05-23 09:35:05.639  DR: Turned off on secondary server

Notice that the "turned off on secondary server" message appears on the secondary but there's no such indication on the primary. An onstat -g cluster from the primary still shows the secondary as connected and active. But it's not.

What happens next is that the next time the primary tries to run a checkpoint, it waits for the secondary to acknowledge it, and that never happens because the secondary thinks it's disconnected while the primary thinks it is still connected. Users can still connect to the primary and even run queries, but as soon as they execute anything that would do a logged write, they get blocked on the pending checkpoint. An onstat - on the primary shows CKPT_INP.

When it gets to this point, the primary will no longer respond to onmode commands. The only way to clear the situation is to take down all the other servers in the cluster, then kill the primary using onclean -ky. At that point, it fires right back up as if nothing happened. And I can re-start the other nodes without incident.

Obviously, this should never happen. If the HDR secondary isn't responding, it should time out and be kicked out of the cluster. The primary shouldn't hang for an eternity waiting around for it.

In case it's relevant, we have DRAUTO set to 3 and we're using NEAR_SYNC mode.

I have a case (TS009442018) open on this. Problem is, support wants me to replicate the issue and grab stats next time it happens, which isn't something I'm super keen on doing in production. (For the time being, to avoid the issue, I've demoted the HDR secondary to an RSS.)

All:

I wanted to make you aware of something we've hit a couple of times and also find out if someone else has encountered it. Twice in the past few months, I've had DR Ping faliures on our HDR primary which seem to have left the cluster in an unstable state. It's as if the primary lost track of the secondary but doesn't realize that it's not there. In online.log on the primary, it looks like this:

09:33:14  DR: Needed to send a ping message but failed. 109:33:44  DR: Needed to send a ping message but failed. 109:34:14  DR: Needed to send a ping message but failed. 109:34:44  DR: Needed to send a ping message but failed. 109:34:54  DR: ping timeout

On the secondary, it looks like this:

2022-05-23 09:35:04  DR: ping timeout2022-05-23 09:35:04.639  DR: Receive error2022-05-23 09:35:04.640  dr_secrcv thread : asfcode = -25582: oserr = 4: errstr = : Network connection is broken. System error = 4.2022-05-23 09:35:04.640  DR_ERR set to -12022-05-23 09:35:04.640  DR: Receive Btree error2022-05-23 09:35:05.639  DR: Turned off on secondary server

Notice that the "turned off on secondary server" message appears on the secondary but there's no such indication on the primary. An onstat -g cluster from the primary still shows the secondary as connected and active. But it's not.

What happens next is that the next time the primary tries to run a checkpoint, it waits for the secondary to acknowledge it, and that never happens because the secondary thinks it's disconnected while the primary thinks it is still connected. Users can still connect to the primary and even run queries, but as soon as they execute anything that would do a logged write, they get blocked on the pending checkpoint. An onstat - on the primary shows CKPT_INP.

When it gets to this point, the primary will no longer respond to onmode commands. The only way to clear the situation is to take down all the other servers in the cluster, then kill the primary using onclean -ky. At that point, it fires right back up as if nothing happened. And I can re-start the other nodes without incident.

Obviously, this should never happen. If the HDR secondary isn't responding, it should time out and be kicked out of the cluster. The primary shouldn't hang for an eternity waiting around for it.

In case it's relevant, we have DRAUTO set to 3 and we're using NEAR_SYNC mode.

I have a case (TS009442018) open on this. Problem is, support wants me to replicate the issue and grab stats next time it happens, which isn't something I'm super keen on doing in production. (For the time being, to avoid the issue, I've demoted the HDR secondary to an RSS.)

All:

I wanted to make you aware of something we've hit a couple of times and also find out if someone else has encountered it. Twice in the past few months, I've had DR Ping faliures on our HDR primary which seem to have left the cluster in an unstable state. It's as if the primary lost track of the secondary but doesn't realize that it's not there. In online.log on the primary, it looks like this:

09:33:14  DR: Needed to send a ping message but failed. 109:33:44  DR: Needed to send a ping message but failed. 109:34:14  DR: Needed to send a ping message but failed. 109:34:44  DR: Needed to send a ping message but failed. 109:34:54  DR: ping timeout

On the secondary, it looks like this:

2022-05-23 09:35:04  DR: ping timeout2022-05-23 09:35:04.639  DR: Receive error2022-05-23 09:35:04.640  dr_secrcv thread : asfcode = -25582: oserr = 4: errstr = : Network connection is broken. System error = 4.2022-05-23 09:35:04.640  DR_ERR set to -12022-05-23 09:35:04.640  DR: Receive Btree error2022-05-23 09:35:05.639  DR: Turned off on secondary server

Notice that the "turned off on secondary server" message appears on the secondary but there's no such indication on the primary. An onstat -g cluster from the primary still shows the secondary as connected and active. But it's not.

What happens next is that the next time the primary tries to run a checkpoint, it waits for the secondary to acknowledge it, and that never happens because the secondary thinks it's disconnected while the primary thinks it is still connected. Users can still connect to the primary and even run queries, but as soon as they execute anything that would do a logged write, they get blocked on the pending checkpoint. An onstat - on the primary shows CKPT_INP.

When it gets to this point, the primary will no longer respond to onmode commands. The only way to clear the situation is to take down all the other servers in the cluster, then kill the primary using onclean -ky. At that point, it fires right back up as if nothing happened. And I can re-start the other nodes without incident.

Obviously, this should never happen. If the HDR secondary isn't responding, it should time out and be kicked out of the cluster. The primary shouldn't hang for an eternity waiting around for it.

In case it's relevant, we have DRAUTO set to 3 and we're using NEAR_SYNC mode.

I have a case (TS009442018) open on this. Problem is, support wants me to replicate the issue and grab stats next time it happens, which isn't something I'm super keen on doing in production. (For the time being, to avoid the issue, I've demoted the HDR secondary to an RSS.)

All:

I wanted to make you aware of something we've hit a couple of times and also find out if someone else has encountered it. Twice in the past few months, I've had DR Ping faliures on our HDR primary which seem to have left the cluster in an unstable state. It's as if the primary lost track of the secondary but doesn't realize that it's not there. In online.log on the primary, it looks like this:

09:33:14  DR: Needed to send a ping message but failed. 109:33:44  DR: Needed to send a ping message but failed. 109:34:14  DR: Needed to send a ping message but failed. 109:34:44  DR: Needed to send a ping message but failed. 109:34:54  DR: ping timeout

On the secondary, it looks like this:

2022-05-23 09:35:04  DR: ping timeout2022-05-23 09:35:04.639  DR: Receive error2022-05-23 09:35:04.640  dr_secrcv thread : asfcode = -25582: oserr = 4: errstr = : Network connection is broken. System error = 4.2022-05-23 09:35:04.640  DR_ERR set to -12022-05-23 09:35:04.640  DR: Receive Btree error2022-05-23 09:35:05.639  DR: Turned off on secondary server

Notice that the "turned off on secondary server" message appears on the secondary but there's no such indication on the primary. An onstat -g cluster from the primary still shows the secondary as connected and active. But it's not.

What happens next is that the next time the primary tries to run a checkpoint, it waits for the secondary to acknowledge it, and that never happens because the secondary thinks it's disconnected while the primary thinks it is still connected. Users can still connect to the primary and even run queries, but as soon as they execute anything that would do a logged write, they get blocked on the pending checkpoint. An onstat - on the primary shows CKPT_INP.

When it gets to this point, the primary will no longer respond to onmode commands. The only way to clear the situation is to take down all the other servers in the cluster, then kill the primary using onclean -ky. At that point, it fires right back up as if nothing happened. And I can re-start the other nodes without incident.

Obviously, this should never happen. If the HDR secondary isn't responding, it should time out and be kicked out of the cluster. The primary shouldn't hang for an eternity waiting around for it.

In case it's relevant, we have DRAUTO set to 3 and we're using NEAR_SYNC mode.

I have a case (TS009442018) open on this. Problem is, support wants me to replicate the issue and grab stats next time it happens, which isn't something I'm super keen on doing in production. (For the time being, to avoid the issue, I've demoted the HDR secondary to an RSS.)

All:

I wanted to make you aware of something we've hit a couple of times and also find out if someone else has encountered it. Twice in the past few months, I've had DR Ping faliures on our HDR primary which seem to have left the cluster in an unstable state. It's as if the primary lost track of the secondary but doesn't realize that it's not there. In online.log on the primary, it looks like this:

09:33:14  DR: Needed to send a ping message but failed. 109:33:44  DR: Needed to send a ping message but failed. 109:34:14  DR: Needed to send a ping message but failed. 109:34:44  DR: Needed to send a ping message but failed. 109:34:54  DR: ping timeout

On the secondary, it looks like this:

2022-05-23 09:35:04  DR: ping timeout2022-05-23 09:35:04.639  DR: Receive error2022-05-23 09:35:04.640  dr_secrcv thread : asfcode = -25582: oserr = 4: errstr = : Network connection is broken. System error = 4.2022-05-23 09:35:04.640  DR_ERR set to -12022-05-23 09:35:04.640  DR: Receive Btree error2022-05-23 09:35:05.639  DR: Turned off on secondary server

Notice that the "turned off on secondary server" message appears on the secondary but there's no such indication on the primary. An onstat -g cluster from the primary still shows the secondary as connected and active. But it's not.

What happens next is that the next time the primary tries to run a checkpoint, it waits for the secondary to acknowledge it, and that never happens because the secondary thinks it's disconnected while the primary thinks it is still connected. Users can still connect to the primary and even run queries, but as soon as they execute anything that would do a logged write, they get blocked on the pending checkpoint. An onstat - on the primary shows CKPT_INP.

When it gets to this point, the primary will no longer respond to onmode commands. The only way to clear the situation is to take down all the other servers in the cluster, then kill the primary using onclean -ky. At that point, it fires right back up as if nothing happened. And I can re-start the other nodes without incident.

Obviously, this should never happen. If the HDR secondary isn't responding, it should time out and be kicked out of the cluster. The primary shouldn't hang for an eternity waiting around for it.

In case it's relevant, we have DRAUTO set to 3 and we're using NEAR_SYNC mode.

I have a case (TS009442018) open on this. Problem is, support wants me to replicate the issue and grab stats next time it happens, which isn't something I'm super keen on doing in production. (For the time being, to avoid the issue, I've demoted the HDR secondary to an RSS.)

All:

I wanted to make you aware of something we've hit a couple of times and also find out if someone else has encountered it. Twice in the past few months, I've had DR Ping faliures on our HDR primary which seem to have left the cluster in an unstable state. It's as if the primary lost track of the secondary but doesn't realize that it's not there. In online.log on the primary, it looks like this:

09:33:14  DR: Needed to send a ping message but failed. 109:33:44  DR: Needed to send a ping message but failed. 109:34:14  DR: Needed to send a ping message but failed. 109:34:44  DR: Needed to send a ping message but failed. 109:34:54  DR: ping timeout

On the secondary, it looks like this:

2022-05-23 09:35:04  DR: ping timeout2022-05-23 09:35:04.639  DR: Receive error2022-05-23 09:35:04.640  dr_secrcv thread : asfcode = -25582: oserr = 4: errstr = : Network connection is broken. System error = 4.2022-05-23 09:35:04.640  DR_ERR set to -12022-05-23 09:35:04.640  DR: Receive Btree error2022-05-23 09:35:05.639  DR: Turned off on secondary server

Notice that the "turned off on secondary server" message appears on the secondary but there's no such indication on the primary. An onstat -g cluster from the primary still shows the secondary as connected and active. But it's not.

What happens next is that the next time the primary tries to run a checkpoint, it waits for the secondary to acknowledge it, and that never happens because the secondary thinks it's disconnected while the primary thinks it is still connected. Users can still connect to the primary and even run queries, but as soon as they execute anything that would do a logged write, they get blocked on the pending checkpoint. An onstat - on the primary shows CKPT_INP.

When it gets to this point, the primary will no longer respond to onmode commands. The only way to clear the situation is to take down all the other servers in the cluster, then kill the primary using onclean -ky. At that point, it fires right back up as if nothing happened. And I can re-start the other nodes without incident.

Obviously, this should never happen. If the HDR secondary isn't responding, it should time out and be kicked out of the cluster. The primary shouldn't hang for an eternity waiting around for it.

In case it's relevant, we have DRAUTO set to 3 and we're using NEAR_SYNC mode.

I have a case (TS009442018) open on this. Problem is, support wants me to replicate the issue and grab stats next time it happens, which isn't something I'm super keen on doing in production. (For the time being, to avoid the issue, I've demoted the HDR secondary to an RSS.)