Informix 11.50.FC6WE, HP-UX 11.31 on PA-RISC

We are running Genero applications on an Apache web server.  The Genero and Apache are running on a Linux box, which connects to our database server via the network.  Our Informix server is configured with both shared memory and network listeners

NETTYPE ipcshm,5,350,CPU
NETTYPE soctcp,8,200,NET

During periods of high activity, such as the morning after a weekend or holiday, we receive a large number of -27001 errors in the session logs of the Genero applications.  Today, we had about 2000 such errors by 10:00 AM.  The typical message looks like:

Found this document on the IBM Support pages (https://www.ibm.com/support/pages/error-27001-unable-connect-tcp-network-connections).  The first item it mentions is that connectivity files are not being removed from /INFORMIXTMP.  I looked in that directory, and there are 71 files named VP.my_instance_name.*.  All but three of these files date back to the day that the instance was started, with times spanning a period of about 10 hours.  The other three are dated the next day, one shortly after midnight, the other two around 08:30 AM.

My questions are, how many of these files should exist normally?  And should they all be from the date/time that the instance started?  Or should they be closer to the current date?  Or any date between the two dates?

The document recommends shutting down the instance and deleting these files.  If I saw files from prior to the date the instance was started, I'd be much more inclined to do that, but if these files typically get created when the instance starts, or shortly after, and are expected to stay there, then I don't want to take an outage for something that is working normally.

The document also mentions other potential causes of this error, including:

> incorrect VP class (CPU) for network connections - as you can see above, we have 8 soctcp listeners running on NET VPs

> kernel parameter problems - the parameters were set using Informix recommendations from the release notes, but if anyone has any suggestions, please share

> network problems (busy or lost packets) - our network group has confirmed that the network was under 50% utilization during the time of the Informix errors

> intermittent hardware failure - possible, but not sure how we'd confirm that

I considered adding more listeners via the 'onmode -P' command, but the number of network sessions seems to have abated somewhat and we are receiving fewer of the -27001 errors now.  But I want to understand what's going on and how to address it if it happens again.

Any other thoughts would be appreciated.

Thanks.


------------------------------
Mark Collins
------------------------------

#Informix

------Original Message------

Informix 11.50.FC6WE, HP-UX 11.31 on PA-RISC

We are running Genero applications on an Apache web server.  The Genero and Apache are running on a Linux box, which connects to our database server via the network.  Our Informix server is configured with both shared memory and network listeners

NETTYPE ipcshm,5,350,CPU
NETTYPE soctcp,8,200,NET

During periods of high activity, such as the morning after a weekend or holiday, we receive a large number of -27001 errors in the session logs of the Genero applications.  Today, we had about 2000 such errors by 10:00 AM.  The typical message looks like:

Found this document on the IBM Support pages (https://www.ibm.com/support/pages/error-27001-unable-connect-tcp-network-connections).  The first item it mentions is that connectivity files are not being removed from /INFORMIXTMP.  I looked in that directory, and there are 71 files named VP.my_instance_name.*.  All but three of these files date back to the day that the instance was started, with times spanning a period of about 10 hours.  The other three are dated the next day, one shortly after midnight, the other two around 08:30 AM.

My questions are, how many of these files should exist normally?  And should they all be from the date/time that the instance started?  Or should they be closer to the current date?  Or any date between the two dates?

The document recommends shutting down the instance and deleting these files.  If I saw files from prior to the date the instance was started, I'd be much more inclined to do that, but if these files typically get created when the instance starts, or shortly after, and are expected to stay there, then I don't want to take an outage for something that is working normally.

The document also mentions other potential causes of this error, including:

> incorrect VP class (CPU) for network connections - as you can see above, we have 8 soctcp listeners running on NET VPs

> kernel parameter problems - the parameters were set using Informix recommendations from the release notes, but if anyone has any suggestions, please share

> network problems (busy or lost packets) - our network group has confirmed that the network was under 50% utilization during the time of the Informix errors

> intermittent hardware failure - possible, but not sure how we'd confirm that

I considered adding more listeners via the 'onmode -P' command, but the number of network sessions seems to have abated somewhat and we are receiving fewer of the -27001 errors now.  But I want to understand what's going on and how to address it if it happens again.

Any other thoughts would be appreciated.

Thanks.


------------------------------
Mark Collins
------------------------------

#Informix

Thanks.  We don't have that parameter set at the moment, so it's just using the default.  I'll put that on the list of things to try when I can schedule an outage.

------------------------------
Mark Collins
------------------------------

Original Message:
Sent: Tue January 21, 2020 12:41 PM
From: Art Kagel
Subject: SQL error -27001

Try increasing NUMFDSERVERS which defaults to 1 in the ONCONFIG file.

Art 

Art S. Kagel, President and Principal Consultant

ASK Database Management

www.askdbmgt.com

Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference.  Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

Original Message------

Informix 11.50.FC6WE, HP-UX 11.31 on PA-RISC

We are running Genero applications on an Apache web server.  The Genero and Apache are running on a Linux box, which connects to our database server via the network.  Our Informix server is configured with both shared memory and network listeners

NETTYPE ipcshm,5,350,CPU
NETTYPE soctcp,8,200,NET

During periods of high activity, such as the morning after a weekend or holiday, we receive a large number of -27001 errors in the session logs of the Genero applications.  Today, we had about 2000 such errors by 10:00 AM.  The typical message looks like:

Program stopped at 'myGeneroApplication.4gl', line number 162.
SQL statement error number -27001.
Read error occurred during connection attempt.

Found this document on the IBM Support pages (https://www.ibm.com/support/pages/error-27001-unable-connect-tcp-network-connections).  The first item it mentions is that connectivity files are not being removed from /INFORMIXTMP.  I looked in that directory, and there are 71 files named VP.my_instance_name.*.  All but three of these files date back to the day that the instance was started, with times spanning a period of about 10 hours.  The other three are dated the next day, one shortly after midnight, the other two around 08:30 AM.

My questions are, how many of these files should exist normally?  And should they all be from the date/time that the instance started?  Or should they be closer to the current date?  Or any date between the two dates?

The document recommends shutting down the instance and deleting these files.  If I saw files from prior to the date the instance was started, I'd be much more inclined to do that, but if these files typically get created when the instance starts, or shortly after, and are expected to stay there, then I don't want to take an outage for something that is working normally.

The document also mentions other potential causes of this error, including:

> incorrect VP class (CPU) for network connections - as you can see above, we have 8 soctcp listeners running on NET VPs

> kernel parameter problems - the parameters were set using Informix recommendations from the release notes, but if anyone has any suggestions, please share

> network problems (busy or lost packets) - our network group has confirmed that the network was under 50% utilization during the time of the Informix errors

> intermittent hardware failure - possible, but not sure how we'd confirm that

I considered adding more listeners via the 'onmode -P' command, but the number of network sessions seems to have abated somewhat and we are receiving fewer of the -27001 errors now.  But I want to understand what's going on and how to address it if it happens again.

Any other thoughts would be appreciated.

Thanks.


------------------------------
Mark Collins
------------------------------

#Informix

I realized I did not post my sqlhosts entries for this.  They are:

my_instance_name onipcshm host1 9000
my_instance_name onsoctcp host1 tcp_my_instancea

Looking at /etc/services, I see that tcp_my_instancea is set to 9000/tcp.  That matches the value of the service_name part of the shared memory connection on the line above.  Is there a conflict between these two?  Should the shared memory entry use a different number?  I know it's not referencing a port on the shared memory line, but I just wanted to make sure that there is no conflict.




------------------------------
Mark Collins
------------------------------

------Original Message------

I realized I did not post my sqlhosts entries for this.  They are:

my_instance_name onipcshm host1 9000
my_instance_name onsoctcp host1 tcp_my_instancea

Looking at /etc/services, I see that tcp_my_instancea is set to 9000/tcp.  That matches the value of the service_name part of the shared memory connection on the line above.  Is there a conflict between these two?  Should the shared memory entry use a different number?  I know it's not referencing a port on the shared memory line, but I just wanted to make sure that there is no conflict.




------------------------------
Mark Collins
------------------------------

#Informix

And I'll add this to the changes to be done at the next scheduled outage as well.

Any information on those VP.instance_name.* files?  How many should there be?  Do they stay there the entire time the instance is up?  Are they automatically deleted during a shutdown?

Thanks.

------------------------------
Mark Collins
------------------------------

Original Message:
Sent: Tue January 21, 2020 12:51 PM
From: Art Kagel
Subject: SQL error -27001

Mark:

Yes, having the shared memory connection line include the same port as one of the TCP connection lines will often cause problems even though the entry is a dummy for shared memory connections and is not supposed to be used in any way. I have witnessed the issue several times on different platforms.

What I normally do is to put the server name or some REAL dummy value like "dummy" or "freddy" in there to avoid conflicts.

Art

Art S. Kagel, President and Principal Consultant

ASK Database Management

www.askdbmgt.com

Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference.  Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

Original Message------

I realized I did not post my sqlhosts entries for this.  They are:

my_instance_name onipcshm host1 9000
my_instance_name onsoctcp host1 tcp_my_instancea

Looking at /etc/services, I see that tcp_my_instancea is set to 9000/tcp.  That matches the value of the service_name part of the shared memory connection on the line above.  Is there a conflict between these two?  Should the shared memory entry use a different number?  I know it's not referencing a port on the shared memory line, but I just wanted to make sure that there is no conflict.




------------------------------
Mark Collins
------------------------------

#Informix

Thank you for the information. Art.
My client's Informix is also configured for ipc connection and 27001 error occurs frequently in online log.
I need to recheck the contents of the sqlhosts and /etc/services files on the client's server.

------------------------------
SangGyu Jeong
Software Engineer
Infrasoft
Seoul Korea, Republic of
------------------------------

Original Message:
Sent: Tue January 21, 2020 12:51 PM
From: Art Kagel
Subject: SQL error -27001

Mark:

Yes, having the shared memory connection line include the same port as one of the TCP connection lines will often cause problems even though the entry is a dummy for shared memory connections and is not supposed to be used in any way. I have witnessed the issue several times on different platforms.

What I normally do is to put the server name or some REAL dummy value like "dummy" or "freddy" in there to avoid conflicts.

Art

Art S. Kagel, President and Principal Consultant

ASK Database Management

www.askdbmgt.com

Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference.  Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

Original Message------

I realized I did not post my sqlhosts entries for this.  They are:

my_instance_name onipcshm host1 9000
my_instance_name onsoctcp host1 tcp_my_instancea

Looking at /etc/services, I see that tcp_my_instancea is set to 9000/tcp.  That matches the value of the service_name part of the shared memory connection on the line above.  Is there a conflict between these two?  Should the shared memory entry use a different number?  I know it's not referencing a port on the shared memory line, but I just wanted to make sure that there is no conflict.




------------------------------
Mark Collins
------------------------------

#Informix