Hello,

I'm using Informix as a container into a kubernetes cluster. I use an IBM cloud K8S cluster and build the Informix container with the offical scripts (https://github.com/informix/informix-server-dockerfiles).

All works well but I would like to enable hugepage to get better performances.

I have followed the IBM cloud documentation to enable hugepages on the node supporting Informix (https://cloud.ibm.com/docs/containers?topic=containers-kernel and https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/).
The yaml scripts used to launch Informix on the k8s cluster are attached to this post.

The Informix pod is launched but the Informix instance refuse to start and don't show any particular issue. Without the hugepages, with the same configuration, Informix works well.
I attached to this post the following informations comming from the pod:

• envvar.txt : the environment variables;
• meminfo.txt : the content of "/proc/meminfo";
• online.log.txt : the Informix log;
• oninit.txt : the output of the "oninit -ivy" command;
• onconfig.txt : the content of the "onconfig" file;
• ipcs.txt : the result of the "ipcs" command.

It seems that Informix correcly see the hugepages, try to use it but has a problem to allocate shared memory. However, I have to little traces to understand the root cause of the problem.

I also tried the same configuration on my laptop with microk8s cluster and all is working fine: Informix is able to start and uses the hugepages.

Could you please help me?
Is it possible to enable more debug traces on Informix?

Thanks a lot for your help.
Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

#Informix

------Original Message------

Hello,

I'm using Informix as a container into a kubernetes cluster. I use an IBM cloud K8S cluster and build the Informix container with the offical scripts (https://github.com/informix/informix-server-dockerfiles).

All works well but I would like to enable hugepage to get better performances.

I have followed the IBM cloud documentation to enable hugepages on the node supporting Informix (https://cloud.ibm.com/docs/containers?topic=containers-kernel and https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/).
The yaml scripts used to launch Informix on the k8s cluster are attached to this post.

The Informix pod is launched but the Informix instance refuse to start and don't show any particular issue. Without the hugepages, with the same configuration, Informix works well.
I attached to this post the following informations comming from the pod:

• envvar.txt : the environment variables;
• meminfo.txt : the content of "/proc/meminfo";
• online.log.txt : the Informix log;
• oninit.txt : the output of the "oninit -ivy" command;
• onconfig.txt : the content of the "onconfig" file;
• ipcs.txt : the result of the "ipcs" command.

It seems that Informix correcly see the hugepages, try to use it but has a problem to allocate shared memory. However, I have to little traces to understand the root cause of the problem.

I also tried the same configuration on my laptop with microk8s cluster and all is working fine: Informix is able to start and uses the hugepages.

Could you please help me?
Is it possible to enable more debug traces on Informix?

Thanks a lot for your help.
Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

#Informix

Hi Paul,

Here is the output of "onstat -m".
For me, there is no more information.

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

Original Message:
Sent: Wed November 06, 2019 03:39 PM
From: Paul Watson
Subject: Hugepages

Anything in onstat –m

 

Cheers

Paul

 

Original Message------

Hello,

I'm using Informix as a container into a kubernetes cluster. I use an IBM cloud K8S cluster and build the Informix container with the offical scripts (https://github.com/informix/informix-server-dockerfiles).

All works well but I would like to enable hugepage to get better performances.

I have followed the IBM cloud documentation to enable hugepages on the node supporting Informix (https://cloud.ibm.com/docs/containers?topic=containers-kernel and https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/).
The yaml scripts used to launch Informix on the k8s cluster are attached to this post.

The Informix pod is launched but the Informix instance refuse to start and don't show any particular issue. Without the hugepages, with the same configuration, Informix works well.
I attached to this post the following informations comming from the pod:

• envvar.txt : the environment variables;
• meminfo.txt : the content of "/proc/meminfo";
• online.log.txt : the Informix log;
• oninit.txt : the output of the "oninit -ivy" command;
• onconfig.txt : the content of the "onconfig" file;
• ipcs.txt : the result of the "ipcs" command.

It seems that Informix correcly see the hugepages, try to use it but has a problem to allocate shared memory. However, I have to little traces to understand the root cause of the problem.

I also tried the same configuration on my laptop with microk8s cluster and all is working fine: Informix is able to start and uses the hugepages.

Could you please help me?
Is it possible to enable more debug traces on Informix?

Thanks a lot for your help.
Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

#Informix

------Original Message------

Hi Paul,

Here is the output of "onstat -m".
For me, there is no more information.

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

#Informix

No, we have only this.


informix@informixdb-hp-0:~$ onstat -m
Your evaluation license will expire on 2019-12-15 11:32:57
shared memory not initialized for INFORMIXSERVER 'informix'

Message Log File: /opt/ibm/data/logs/online.log
21:24:49 Parameter's user-configured value was adjusted. (VPCLASS)
21:24:49 IBM Informix Dynamic Server Started.
21:24:49 Requested shared memory segment size rounded from 8996KB to 10240KB
21:24:49 Shared memory segment will use huge pages.
informix@informixdb-hp-0:~$

------------------------------
Renaud Demarneffe
------------------------------

Original Message:
Sent: Wed November 06, 2019 04:51 PM
From: Paul Watson
Subject: Hugepages

Does this mean there was no output ?

 

Original Message------

Hi Paul,

Here is the output of "onstat -m".
For me, there is no more information.

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

#Informix

This is what this normally should be looking like:

11/05/19 17:19:17 Requested shared memory segment size rounded from 2302208KB to 3145728KB
11/05/19 17:19:17 Shared memory segment will use huge pages.
11/05/19 17:19:17 Segment locked: addr=0x140000000, size=3221225472
11/05/19 17:19:17 Successfully added a bufferpool of page size 2K.

11/05/19 17:19:17 Requested shared memory segment size rounded from 2148608KB to 3145728KB
11/05/19 17:19:17 Shared memory segment will use huge pages.
11/05/19 17:19:18 Segment locked: addr=0x200000000, size=3221225472
11/05/19 17:19:18 Successfully added a bufferpool of page size 8K.

So are you saying, above, that you're missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log?

Huge pages memory allocation can take a while, yet it this doesn't move at all, something might be wrong on OS level?
If this really is just hanging there,

1. can you show us 'ipcs -m' output at this stage?
2. can you retry using strace, as user root:     strace -o /tmp/oninit.trc  oninit
   then post the oninit.trc file?

------------------------------
Andreas Legner
------------------------------

Original Message:
Sent: Thu November 07, 2019 03:34 AM
From: Renaud Demarneffe
Subject: Hugepages

No, we have only this.


informix@informixdb-hp-0:~$ onstat -m
Your evaluation license will expire on 2019-12-15 11:32:57
shared memory not initialized for INFORMIXSERVER 'informix'

Message Log File: /opt/ibm/data/logs/online.log
21:24:49 Parameter's user-configured value was adjusted. (VPCLASS)
21:24:49 IBM Informix Dynamic Server Started.
21:24:49 Requested shared memory segment size rounded from 8996KB to 10240KB
21:24:49 Shared memory segment will use huge pages.
informix@informixdb-hp-0:~$

------------------------------
Renaud Demarneffe
------------------------------


Original Message:
Sent: Wed November 06, 2019 04:51 PM
From: Paul Watson
Subject: Hugepages

Does this mean there was no output ?

 

Original Message------

Hi Paul,

Here is the output of "onstat -m".
For me, there is no more information.

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

#Informix

Hi Andreas,

Yes, I'm missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log.

The output of the "ipcs -m" command, after the oninit hangup is:
informix@informixdb-hp-0:~$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x52564801 262144 root 660 10485760 0


The oninit.trc is attached to the post. As the Informix is installed as a non-root installation (executed by informix user), I need to hack in order to attach to the right process as soon as it started (oninit -ivwy >> $INIT_LOG & oninitpid=$(pidof oninit) ; sudo strace -s 65536 -o /tmp/oninit.trc -p $oninitpid).


Thanks for your help.
Regards,
Renaud



------------------------------
Renaud Demarneffe
------------------------------

Original Message:
Sent: Fri November 08, 2019 11:53 AM
From: Andreas Legner
Subject: Hugepages

This is what this normally should be looking like:

11/05/19 17:19:17 Requested shared memory segment size rounded from 2302208KB to 3145728KB
11/05/19 17:19:17 Shared memory segment will use huge pages.
11/05/19 17:19:17 Segment locked: addr=0x140000000, size=3221225472
11/05/19 17:19:17 Successfully added a bufferpool of page size 2K.

11/05/19 17:19:17 Requested shared memory segment size rounded from 2148608KB to 3145728KB
11/05/19 17:19:17 Shared memory segment will use huge pages.
11/05/19 17:19:18 Segment locked: addr=0x200000000, size=3221225472
11/05/19 17:19:18 Successfully added a bufferpool of page size 8K.

So are you saying, above, that you're missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log?

Huge pages memory allocation can take a while, yet it this doesn't move at all, something might be wrong on OS level?
If this really is just hanging there,

1. can you show us 'ipcs -m' output at this stage?
2. can you retry using strace, as user root:     strace -o /tmp/oninit.trc  oninit
   then post the oninit.trc file?

------------------------------
Andreas Legner
------------------------------


Original Message:
Sent: Thu November 07, 2019 03:34 AM
From: Renaud Demarneffe
Subject: Hugepages

No, we have only this.


informix@informixdb-hp-0:~$ onstat -m
Your evaluation license will expire on 2019-12-15 11:32:57
shared memory not initialized for INFORMIXSERVER 'informix'

Message Log File: /opt/ibm/data/logs/online.log
21:24:49 Parameter's user-configured value was adjusted. (VPCLASS)
21:24:49 IBM Informix Dynamic Server Started.
21:24:49 Requested shared memory segment size rounded from 8996KB to 10240KB
21:24:49 Shared memory segment will use huge pages.
informix@informixdb-hp-0:~$

------------------------------
Renaud Demarneffe


Original Message:
Sent: Wed November 06, 2019 04:51 PM
From: Paul Watson
Subject: Hugepages

Does this mean there was no output ?

 

Original Message------

Hi Paul,

Here is the output of "onstat -m".
For me, there is no more information.

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

#Informix

------Original Message------

Hi Andreas,

Yes, I'm missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log.

The output of the "ipcs -m" command, after the oninit hangup is:
informix@informixdb-hp-0:~$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x52564801 262144 root 660 10485760 0


The oninit.trc is attached to the post. As the Informix is installed as a non-root installation (executed by informix user), I need to hack in order to attach to the right process as soon as it started (oninit -ivwy >> $INIT_LOG & oninitpid=$(pidof oninit) ; sudo strace -s 65536 -o /tmp/oninit.trc -p $oninitpid).


Thanks for your help.
Regards,
Renaud



------------------------------
Renaud Demarneffe
------------------------------

#Informix

Hi Paul,

I made a mistake into my previous response. The Informix instance is a root one.

After some investigations, I think the problem comes from the K8S cluster that does not allow access to huge pages even if it mount it into the pod.
Indeed, the same docker running on my PC with a microk8s cluster is working while it is not running on an IKS cluster (public IBM K8S cluster).

I created a docker with a small program that tests access to huge pages via shm (shmget function). This docker runs smoothly on microk8s but not on IKS. On IKS, it returns the following error:
--- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7f7f71400000} ---
+++ killed by SIGBUS (core dumped) +++
The files for this test are attached.

The error of this program is the same as the Informix one.
I think the problem comes from this sentence that is in the Kubernetes documentation: Applications that consume huge pages via shmget() with SHM_HUGETLB must run with a supplemental group that matches proc/sys/vm/hugetlb_shm_group.
Or comes from a security option of the IKS.

But I don't understand how to solve it:
* I'm running the "hugepage-shm" program as root user (uid=0, gid=0);
* The content of "/proc/sys/vm/hugetlb_shm_group" is "0".
Maybe I need to adapt the pod's scuritycontext or apply a particular Pod Security Policy but I do not see which one?

I opened a support ticket on IBM. Let's see what they will say.

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

Original Message:
Sent: Sun November 10, 2019 10:09 AM
From: Paul Watson
Subject: Hugepages

Huge pages and a non-root install doesn't work - or I have never been able to get it to work, change to a roof installation with no config changes and it should just work

Cheers

Paul

Paul Watson

Oninit LLC

+1-913-387-7529

www.oninit.com

Oninit®️ is a registered trademark of Oninit LLC

Original Message------

Hi Andreas,

Yes, I'm missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log.

The output of the "ipcs -m" command, after the oninit hangup is:
informix@informixdb-hp-0:~$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x52564801 262144 root 660 10485760 0


The oninit.trc is attached to the post. As the Informix is installed as a non-root installation (executed by informix user), I need to hack in order to attach to the right process as soon as it started (oninit -ivwy >> $INIT_LOG & oninitpid=$(pidof oninit) ; sudo strace -s 65536 -o /tmp/oninit.trc -p $oninitpid).


Thanks for your help.
Regards,
Renaud



------------------------------
Renaud Demarneffe
------------------------------

#Informix

Support ticket directed to which part of IBM? I guess not Informix support?

Could you also post output of your test prog when it fails?  I'm wondering where exactly it fails - with SIGBUS?

------------------------------
Andreas Legner
------------------------------

Original Message:
Sent: Tue November 12, 2019 02:22 PM
From: Renaud Demarneffe
Subject: Hugepages

Hi Paul,

I made a mistake into my previous response. The Informix instance is a root one.

After some investigations, I think the problem comes from the K8S cluster that does not allow access to huge pages even if it mount it into the pod.
Indeed, the same docker running on my PC with a microk8s cluster is working while it is not running on an IKS cluster (public IBM K8S cluster).

I created a docker with a small program that tests access to huge pages via shm (shmget function). This docker runs smoothly on microk8s but not on IKS. On IKS, it returns the following error:
--- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7f7f71400000} ---
+++ killed by SIGBUS (core dumped) +++
The files for this test are attached.

The error of this program is the same as the Informix one.
I think the problem comes from this sentence that is in the Kubernetes documentation: Applications that consume huge pages via shmget() with SHM_HUGETLB must run with a supplemental group that matches proc/sys/vm/hugetlb_shm_group.
Or comes from a security option of the IKS.

But I don't understand how to solve it:
* I'm running the "hugepage-shm" program as root user (uid=0, gid=0);
* The content of "/proc/sys/vm/hugetlb_shm_group" is "0".
Maybe I need to adapt the pod's scuritycontext or apply a particular Pod Security Policy but I do not see which one?

I opened a support ticket on IBM. Let's see what they will say.

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------


Original Message:
Sent: Sun November 10, 2019 10:09 AM
From: Paul Watson
Subject: Hugepages

Huge pages and a non-root install doesn't work - or I have never been able to get it to work, change to a roof installation with no config changes and it should just work

Cheers

Paul

Paul Watson

Oninit LLC

+1-913-387-7529

www.oninit.com

Oninit®️ is a registered trademark of Oninit LLC

Original Message------

Hi Andreas,

Yes, I'm missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log.

The output of the "ipcs -m" command, after the oninit hangup is:
informix@informixdb-hp-0:~$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x52564801 262144 root 660 10485760 0


The oninit.trc is attached to the post. As the Informix is installed as a non-root installation (executed by informix user), I need to hack in order to attach to the right process as soon as it started (oninit -ivwy >> $INIT_LOG & oninitpid=$(pidof oninit) ; sudo strace -s 65536 -o /tmp/oninit.trc -p $oninitpid).


Thanks for your help.
Regards,
Renaud



------------------------------
Renaud Demarneffe
------------------------------

#Informix

The ticket is directed to Kubernetes (Containers) support. 

The error is when the program tries to write into the huge page.
I join the strace output.


------------------------------
Renaud Demarneffe
------------------------------

Original Message:
Sent: Tue November 12, 2019 05:39 PM
From: Andreas Legner
Subject: Hugepages

Support ticket directed to which part of IBM? I guess not Informix support?

Could you also post output of your test prog when it fails?  I'm wondering where exactly it fails - with SIGBUS?

------------------------------
Andreas Legner
------------------------------


Original Message:
Sent: Tue November 12, 2019 02:22 PM
From: Renaud Demarneffe
Subject: Hugepages

Hi Paul,

I made a mistake into my previous response. The Informix instance is a root one.

After some investigations, I think the problem comes from the K8S cluster that does not allow access to huge pages even if it mount it into the pod.
Indeed, the same docker running on my PC with a microk8s cluster is working while it is not running on an IKS cluster (public IBM K8S cluster).

I created a docker with a small program that tests access to huge pages via shm (shmget function). This docker runs smoothly on microk8s but not on IKS. On IKS, it returns the following error:
--- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7f7f71400000} ---
+++ killed by SIGBUS (core dumped) +++
The files for this test are attached.

The error of this program is the same as the Informix one.
I think the problem comes from this sentence that is in the Kubernetes documentation: Applications that consume huge pages via shmget() with SHM_HUGETLB must run with a supplemental group that matches proc/sys/vm/hugetlb_shm_group.
Or comes from a security option of the IKS.

But I don't understand how to solve it:
* I'm running the "hugepage-shm" program as root user (uid=0, gid=0);
* The content of "/proc/sys/vm/hugetlb_shm_group" is "0".
Maybe I need to adapt the pod's scuritycontext or apply a particular Pod Security Policy but I do not see which one?

I opened a support ticket on IBM. Let's see what they will say.

Regards,
Renaud

------------------------------
Renaud Demarneffe


Original Message:
Sent: Sun November 10, 2019 10:09 AM
From: Paul Watson
Subject: Hugepages

Huge pages and a non-root install doesn't work - or I have never been able to get it to work, change to a roof installation with no config changes and it should just work

Cheers

Paul

Paul Watson

Oninit LLC

+1-913-387-7529

www.oninit.com

Oninit®️ is a registered trademark of Oninit LLC

Original Message------

Hi Andreas,

Yes, I'm missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log.

The output of the "ipcs -m" command, after the oninit hangup is:
informix@informixdb-hp-0:~$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x52564801 262144 root 660 10485760 0


The oninit.trc is attached to the post. As the Informix is installed as a non-root installation (executed by informix user), I need to hack in order to attach to the right process as soon as it started (oninit -ivwy >> $INIT_LOG & oninitpid=$(pidof oninit) ; sudo strace -s 65536 -o /tmp/oninit.trc -p $oninitpid).


Thanks for your help.
Regards,
Renaud



------------------------------
Renaud Demarneffe
------------------------------

#Informix

Hello,

I found the solution.
In addition to configuring the worker node to create the huge pages in the kernel, I have to modify the cgroup permissions to allow the pod to write in the huge pages. This is done into the daemonset applied on the worker node (file worker-node-kernel-settings.yaml.txt) where the command to apply on the worker node is now:
sysctl -w vm.nr_hugepages=800; echo 9223372036854771712 | tee /sys/fs/cgroup/hugetlb/kubepods/hugetlb.2MB.limit_in_bytes;

------------------------------
Renaud Demarneffe
------------------------------

Original Message:
Sent: Wed November 13, 2019 04:36 AM
From: Renaud Demarneffe
Subject: Hugepages

The ticket is directed to Kubernetes (Containers) support. 

The error is when the program tries to write into the huge page.
I join the strace output.


------------------------------
Renaud Demarneffe
------------------------------


Original Message:
Sent: Tue November 12, 2019 05:39 PM
From: Andreas Legner
Subject: Hugepages

Support ticket directed to which part of IBM? I guess not Informix support?

Could you also post output of your test prog when it fails?  I'm wondering where exactly it fails - with SIGBUS?

------------------------------
Andreas Legner


Original Message:
Sent: Tue November 12, 2019 02:22 PM
From: Renaud Demarneffe
Subject: Hugepages

Hi Paul,

I made a mistake into my previous response. The Informix instance is a root one.

After some investigations, I think the problem comes from the K8S cluster that does not allow access to huge pages even if it mount it into the pod.
Indeed, the same docker running on my PC with a microk8s cluster is working while it is not running on an IKS cluster (public IBM K8S cluster).

I created a docker with a small program that tests access to huge pages via shm (shmget function). This docker runs smoothly on microk8s but not on IKS. On IKS, it returns the following error:
--- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7f7f71400000} ---
+++ killed by SIGBUS (core dumped) +++
The files for this test are attached.

The error of this program is the same as the Informix one.
I think the problem comes from this sentence that is in the Kubernetes documentation: Applications that consume huge pages via shmget() with SHM_HUGETLB must run with a supplemental group that matches proc/sys/vm/hugetlb_shm_group.
Or comes from a security option of the IKS.

But I don't understand how to solve it:
* I'm running the "hugepage-shm" program as root user (uid=0, gid=0);
* The content of "/proc/sys/vm/hugetlb_shm_group" is "0".
Maybe I need to adapt the pod's scuritycontext or apply a particular Pod Security Policy but I do not see which one?

I opened a support ticket on IBM. Let's see what they will say.

Regards,
Renaud

------------------------------
Renaud Demarneffe


Original Message:
Sent: Sun November 10, 2019 10:09 AM
From: Paul Watson
Subject: Hugepages

Huge pages and a non-root install doesn't work - or I have never been able to get it to work, change to a roof installation with no config changes and it should just work

Cheers

Paul

Paul Watson

Oninit LLC

+1-913-387-7529

www.oninit.com

Oninit®️ is a registered trademark of Oninit LLC

Original Message------

Hi Andreas,

Yes, I'm missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log.

The output of the "ipcs -m" command, after the oninit hangup is:
informix@informixdb-hp-0:~$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x52564801 262144 root 660 10485760 0


The oninit.trc is attached to the post. As the Informix is installed as a non-root installation (executed by informix user), I need to hack in order to attach to the right process as soon as it started (oninit -ivwy >> $INIT_LOG & oninitpid=$(pidof oninit) ; sudo strace -s 65536 -o /tmp/oninit.trc -p $oninitpid).


Thanks for your help.
Regards,
Renaud



------------------------------
Renaud Demarneffe
------------------------------

#Informix

Did you shut down properly, before attempting to restart?

This is your (current) problem:
19:58:58 Requested shared memory segment size rounded from 8996KB to 10240KB
19:58:58 Shared memory segment will use huge pages.
19:58:58 shmget: [EEXIST][17]: key 52564801: shared memory already exists

A simple 'onmode -ky' might remove this leftover segment (and others).

Doesn't immediately look like a huge pages problem, yet might be the result of your real problem
So I'd suggest a retry after removing those existing segments, then re-posting online.log.

------------------------------
Andreas Legner
------------------------------

Hi Andreas,

Here is the new online.log after removing the shm segments.
The problem is still there. The shm segment appears after the failed "oninit".

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

Original Message:
Sent: Wed November 06, 2019 03:49 PM
From: Andreas Legner
Subject: Hugepages

Did you shut down properly, before attempting to restart?

This is your (current) problem:
19:58:58 Requested shared memory segment size rounded from 8996KB to 10240KB
19:58:58 Shared memory segment will use huge pages.
19:58:58 shmget: [EEXIST][17]: key 52564801: shared memory already exists

A simple 'onmode -ky' might remove this leftover segment (and others).

Doesn't immediately look like a huge pages problem, yet might be the result of your real problem
So I'd suggest a retry after removing those existing segments, then re-posting online.log.

------------------------------
Andreas Legner
------------------------------

Hi Renaud,

there should be many more messages in the online.log if the server didn't start. Are you sure that the server isn't running. Please have a look on your oninit processes. What does onstat without any parameter say?

It is possible you get the warning from oninit (WARNING: server initialization failed or timed out.) even though the server is running. We experience this frequently due to a long lasting check of thousands of chunks during startup of the server.

Regards,
Reinhard.

------------------------------
ARZ Emmendingen Support
------------------------------

Original Message:
Sent: Wed November 06, 2019 04:50 PM
From: Renaud Demarneffe
Subject: Hugepages

Hi Andreas,

Here is the new online.log after removing the shm segments.
The problem is still there. The shm segment appears after the failed "oninit".

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------


Original Message:
Sent: Wed November 06, 2019 03:49 PM
From: Andreas Legner
Subject: Hugepages

Did you shut down properly, before attempting to restart?

This is your (current) problem:
19:58:58 Requested shared memory segment size rounded from 8996KB to 10240KB
19:58:58 Shared memory segment will use huge pages.
19:58:58 shmget: [EEXIST][17]: key 52564801: shared memory already exists

A simple 'onmode -ky' might remove this leftover segment (and others).

Doesn't immediately look like a huge pages problem, yet might be the result of your real problem
So I'd suggest a retry after removing those existing segments, then re-posting online.log.

------------------------------
Andreas Legner

Hi Reinhard,

Sorry for the late response.

I retested the solution on a new k8s node with a new pod. I got the same error and I have no more information on the online.log.
I let the pod start for more than 1 hour before reading the logs and executing the onstat cmd.

Here is the content of the online.log file:

informix@informixdb-hp-0:~$ cat /opt/ibm/data/logs/online.log 
21:24:49  Parameter's user-configured value was adjusted. (VPCLASS)
21:24:49  IBM Informix Dynamic Server Started.
21:24:49  Requested shared memory segment size rounded from 8996KB to 10240KB
21:24:49  Shared memory segment will use huge pages.
15:54:22  Parameter's user-configured value was adjusted. (VPCLASS)
15:54:22  IBM Informix Dynamic Server Started.
15:54:22  Requested shared memory segment size rounded from 8996KB to 10240KB
15:54:22  Shared memory segment will use huge pages.
informix@informixdb-hp-0:~$ 

The output of the onstat command is the following:

informix@informixdb-hp-0:~$ onstat
Your evaluation license will expire on 2019-12-15 11:32:57
shared memory not initialized for INFORMIXSERVER 'informix'
informix@informixdb-hp-0:~$ 

I also checked with "ps -aux" the state of the "oninit" process and it's defunct.

informix@informixdb-hp-0:~$ ps -aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
informix     1  0.0  0.0   4084   692 ?        Ss   Nov07   0:00 /opt/ibm/scripts/dinit /opt/ibm/scripts/informix_entry.sh 0x73 /opt/ibm/scripts/informix_stop.sh 
informix     8  0.0  0.0   4340   812 ?        S    Nov07   0:00 sh -c /opt/ibm/scripts/informix_entry.sh 0x73
informix     9  0.0  0.0  20620  3512 ?        S    Nov07   0:00 /bin/bash /opt/ibm/scripts/informix_entry.sh 0x73
root       225  0.0  0.0      0     0 ?        Z    Nov07   0:00 [oninit] <defunct>
informix   305  0.0  0.0   4276   760 ?        S    Nov07   0:03 tail -f /opt/ibm/data/logs/online.log
informix   318  0.0  0.0  20288  3296 pts/0    Ss   16:48   0:00 /bin/bash
informix   335  0.0  0.0  17508  2132 pts/0    R+   16:50   0:00 ps -aux
informix@informixdb-hp-0:~$ 

I don't understand why Informix got into timeout and why the process is killed.

Regards,
Renaud

------------------------------
Renaud Demarneffe
------------------------------

Original Message:
Sent: Thu November 07, 2019 04:54 AM
From: ARZ Emmendingen Support
Subject: Hugepages

Hi Renaud,

there should be many more messages in the online.log if the server didn't start. Are you sure that the server isn't running. Please have a look on your oninit processes. What does onstat without any parameter say?

It is possible you get the warning from oninit (WARNING: server initialization failed or timed out.) even though the server is running. We experience this frequently due to a long lasting check of thousands of chunks during startup of the server.

Regards,
Reinhard.

------------------------------
ARZ Emmendingen Support
------------------------------


Original Message:
Sent: Wed November 06, 2019 04:50 PM
From: Renaud Demarneffe
Subject: Hugepages

Hi Andreas,

Here is the new online.log after removing the shm segments.
The problem is still there. The shm segment appears after the failed "oninit".

Regards,
Renaud

------------------------------
Renaud Demarneffe


Original Message:
Sent: Wed November 06, 2019 03:49 PM
From: Andreas Legner
Subject: Hugepages

Did you shut down properly, before attempting to restart?

This is your (current) problem:
19:58:58 Requested shared memory segment size rounded from 8996KB to 10240KB
19:58:58 Shared memory segment will use huge pages.
19:58:58 shmget: [EEXIST][17]: key 52564801: shared memory already exists

A simple 'onmode -ky' might remove this leftover segment (and others).

Doesn't immediately look like a huge pages problem, yet might be the result of your real problem
So I'd suggest a retry after removing those existing segments, then re-posting online.log.

------------------------------
Andreas Legner