Hi,

I just killed a perl script that reads command output and sends it to syslog server.
The process was using several gigabytes of memory, when I killed this process, drop in memory usage was easily n in export NMON=m ; nmon output.

I made a dummy payload shell script that prints lines for ever.

I stripped down the perl script and changed file handle to read the output from the dummy payload shell script.
Then I ran it

time ./to.syslog.pl
Out of memory!

real 5m37.11s
user 0m7.01s
sys 0m0.45s

I ran following loop in a another terminal

while svmon -P 28836218 -O commandline=on,format=nolimit,unit=MB,segment=on
do
sleep 1
echo
done

Last output before perl ran out of memory was
Command line : svmon -P 28836218 -O commandline=on,format=nolimit,unit=MB,segment=on
Unit: MB

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual
28836218 perl 204.16 49.3 0 204.04

Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
837283 11 work text data BSS heap sm 127.88 0 0 127.88
20002 0 work kernel segment m 52.6 49.1 0 52.6
990019 90000000 work shared library text m 20.1 0 0 20.1
a000a 9ffffffd work shared library sm 1.36 0.02 0 1.36
9b001b 90020014 work shared library s 0.85 0 0 0.85
b06830 9001000a work shared library data sm 0.59 0 0 0.59
bb72bb f00000002 work process private m 0.31 0.19 0 0.31
a66766 80020014 work USLA heap sm 0.24 0 0 0.24
d000d 9ffffffe work shared library sm 0.07 0 0 0.07
b20132 9fffffff clnt USLA text,/dev/hd2:4238 s 0.07 0 - -
907210 10 clnt text data BSS heap, s 0.05 0 - -
/dev/hd10opt:713
af732f ffffffff work application stack sm 0.03 0 0 0.03
aa736a 8fffffff work private load data s 0.02 0 0 0.02

Then I ran this command
./to.stdout.pl > /dev/null

And on another terminal I ran
sleep 600 ; svmon -P 20578644 -O commandline=on,format=nolimit,unit=MB,segment=on

Output from that command was
Command line : svmon -P 20578644 -O commandline=on,format=nolimit,unit=MB,segment=on
Unit: MB

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual
20578644 perl 84.9 49.3 0 84.8

Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
20002 0 work kernel segment m 52.6 49.1 0 52.6
990019 90000000 work shared library text m 24.6 0 0 24.6
a01a20 11 work text data BSS heap sm 4.00 0 0 4.00
a000a 9ffffffd work shared library sm 1.36 0.02 0 1.36
9b001b 90020014 work shared library s 1.01 0 0 1.01
aa732a 9001000a work shared library data sm 0.59 0 0 0.59
857345 f00000002 work process private m 0.31 0.19 0 0.31
a57325 80020014 work USLA heap sm 0.24 0 0 0.24
d000d 9ffffffe work shared library sm 0.08 0 0 0.08
b20132 9fffffff clnt USLA text,/dev/hd2:4238 s 0.07 0 - -
907210 10 clnt text data BSS heap, s 0.05 0 - -
/dev/hd10opt:713
987298 ffffffff work application stack sm 0.03 0 0 0.03
917251 8fffffff work private load data s 0.02 0 0 0.02

The only difference between these two perl scripts is that in the script first script has syslog('local0|notice', "$line"); inside the while() loop, while the script second script has print "$line"; inside the while() loop.

I've ran in to this problem using both system perl /usr/bin/perl and AIX toolbox perl /opt/freeware/bin/perl.
Older 5.8 and I suspect 5.10 perl versions were not affected by this memory leak, but newer such as 5.28 and 5.34 are affected.

------------------------------
Esa Kärkkäinen
------------------------------
#AIXOpenSource

Hi Esa,

Please share the shell script and perl script you are using.

------------------------------
SANGAMESH
------------------------------

Original Message:
Sent: Thu January 12, 2023 07:02 AM
From: Esa Kärkkäinen
Subject: Memory leak in perl syslog function

Hi,

I just killed a perl script that reads command output and sends it to syslog server.
The process was using several gigabytes of memory, when I killed this process, drop in memory usage was easily n in export NMON=m ; nmon output.

I made a dummy payload shell script that prints lines for ever.

I stripped down the perl script and changed file handle to read the output from the dummy payload shell script.
Then I ran it

time ./to.syslog.pl
Out of memory!

real 5m37.11s
user 0m7.01s
sys 0m0.45s

I ran following loop in a another terminal

while svmon -P 28836218 -O commandline=on,format=nolimit,unit=MB,segment=on
do
sleep 1
echo
done

Last output before perl ran out of memory was
Command line : svmon -P 28836218 -O commandline=on,format=nolimit,unit=MB,segment=on
Unit: MB

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual
28836218 perl 204.16 49.3 0 204.04

Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
837283 11 work text data BSS heap sm 127.88 0 0 127.88
20002 0 work kernel segment m 52.6 49.1 0 52.6
990019 90000000 work shared library text m 20.1 0 0 20.1
a000a 9ffffffd work shared library sm 1.36 0.02 0 1.36
9b001b 90020014 work shared library s 0.85 0 0 0.85
b06830 9001000a work shared library data sm 0.59 0 0 0.59
bb72bb f00000002 work process private m 0.31 0.19 0 0.31
a66766 80020014 work USLA heap sm 0.24 0 0 0.24
d000d 9ffffffe work shared library sm 0.07 0 0 0.07
b20132 9fffffff clnt USLA text,/dev/hd2:4238 s 0.07 0 - -
907210 10 clnt text data BSS heap, s 0.05 0 - -
/dev/hd10opt:713
af732f ffffffff work application stack sm 0.03 0 0 0.03
aa736a 8fffffff work private load data s 0.02 0 0 0.02

Then I ran this command
./to.stdout.pl > /dev/null

And on another terminal I ran
sleep 600 ; svmon -P 20578644 -O commandline=on,format=nolimit,unit=MB,segment=on

Output from that command was
Command line : svmon -P 20578644 -O commandline=on,format=nolimit,unit=MB,segment=on
Unit: MB

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual
20578644 perl 84.9 49.3 0 84.8

Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
20002 0 work kernel segment m 52.6 49.1 0 52.6
990019 90000000 work shared library text m 24.6 0 0 24.6
a01a20 11 work text data BSS heap sm 4.00 0 0 4.00
a000a 9ffffffd work shared library sm 1.36 0.02 0 1.36
9b001b 90020014 work shared library s 1.01 0 0 1.01
aa732a 9001000a work shared library data sm 0.59 0 0 0.59
857345 f00000002 work process private m 0.31 0.19 0 0.31
a57325 80020014 work USLA heap sm 0.24 0 0 0.24
d000d 9ffffffe work shared library sm 0.08 0 0 0.08
b20132 9fffffff clnt USLA text,/dev/hd2:4238 s 0.07 0 - -
907210 10 clnt text data BSS heap, s 0.05 0 - -
/dev/hd10opt:713
987298 ffffffff work application stack sm 0.03 0 0 0.03
917251 8fffffff work private load data s 0.02 0 0 0.02

The only difference between these two perl scripts is that in the script first script has syslog('local0|notice', "$line"); inside the while() loop, while the script second script has print "$line"; inside the while() loop.

I've ran in to this problem using both system perl /usr/bin/perl and AIX toolbox perl /opt/freeware/bin/perl.
Older 5.8 and I suspect 5.10 perl versions were not affected by this memory leak, but newer such as 5.28 and 5.34 are affected.

------------------------------
Esa Kärkkäinen
------------------------------
#AIXOpenSource

Hi Sangamesh,

Please find both scripts attached to this post.
FWIW I had to change the perl script name from to.syslog.pl to to.syslog.sh because otherwise the file type was not supported.

The only difference between to.syslog.pl and to.stdout.pl is which line is commented and which is not.

This is to syslog

# print "$line";
syslog('local0|notice', "$line");

And this is to stdout

print "$line";
# syslog('local0|notice', "$line");

------------------------------
Esa Kärkkäinen
------------------------------

Original Message:
Sent: Fri January 13, 2023 06:16 AM
From: SANGAMESH MALLAYYA
Subject: Memory leak in perl syslog function

Hi Esa,

Please share the shell script and perl script you are using.

------------------------------
SANGAMESH

Original Message:
Sent: Thu January 12, 2023 07:02 AM
From: Esa Kärkkäinen
Subject: Memory leak in perl syslog function

Hi,

I just killed a perl script that reads command output and sends it to syslog server.
The process was using several gigabytes of memory, when I killed this process, drop in memory usage was easily n in export NMON=m ; nmon output.

I made a dummy payload shell script that prints lines for ever.

I stripped down the perl script and changed file handle to read the output from the dummy payload shell script.
Then I ran it

time ./to.syslog.pl
Out of memory!

real 5m37.11s
user 0m7.01s
sys 0m0.45s

I ran following loop in a another terminal

while svmon -P 28836218 -O commandline=on,format=nolimit,unit=MB,segment=on
do
sleep 1
echo
done

Last output before perl ran out of memory was
Command line : svmon -P 28836218 -O commandline=on,format=nolimit,unit=MB,segment=on
Unit: MB

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual
28836218 perl 204.16 49.3 0 204.04

Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
837283 11 work text data BSS heap sm 127.88 0 0 127.88
20002 0 work kernel segment m 52.6 49.1 0 52.6
990019 90000000 work shared library text m 20.1 0 0 20.1
a000a 9ffffffd work shared library sm 1.36 0.02 0 1.36
9b001b 90020014 work shared library s 0.85 0 0 0.85
b06830 9001000a work shared library data sm 0.59 0 0 0.59
bb72bb f00000002 work process private m 0.31 0.19 0 0.31
a66766 80020014 work USLA heap sm 0.24 0 0 0.24
d000d 9ffffffe work shared library sm 0.07 0 0 0.07
b20132 9fffffff clnt USLA text,/dev/hd2:4238 s 0.07 0 - -
907210 10 clnt text data BSS heap, s 0.05 0 - -
/dev/hd10opt:713
af732f ffffffff work application stack sm 0.03 0 0 0.03
aa736a 8fffffff work private load data s 0.02 0 0 0.02

Then I ran this command
./to.stdout.pl > /dev/null

And on another terminal I ran
sleep 600 ; svmon -P 20578644 -O commandline=on,format=nolimit,unit=MB,segment=on

Output from that command was
Command line : svmon -P 20578644 -O commandline=on,format=nolimit,unit=MB,segment=on
Unit: MB

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual
20578644 perl 84.9 49.3 0 84.8

Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
20002 0 work kernel segment m 52.6 49.1 0 52.6
990019 90000000 work shared library text m 24.6 0 0 24.6
a01a20 11 work text data BSS heap sm 4.00 0 0 4.00
a000a 9ffffffd work shared library sm 1.36 0.02 0 1.36
9b001b 90020014 work shared library s 1.01 0 0 1.01
aa732a 9001000a work shared library data sm 0.59 0 0 0.59
857345 f00000002 work process private m 0.31 0.19 0 0.31
a57325 80020014 work USLA heap sm 0.24 0 0 0.24
d000d 9ffffffe work shared library sm 0.08 0 0 0.08
b20132 9fffffff clnt USLA text,/dev/hd2:4238 s 0.07 0 - -
907210 10 clnt text data BSS heap, s 0.05 0 - -
/dev/hd10opt:713
987298 ffffffff work application stack sm 0.03 0 0 0.03
917251 8fffffff work private load data s 0.02 0 0 0.02

The only difference between these two perl scripts is that in the script first script has syslog('local0|notice', "$line"); inside the while() loop, while the script second script has print "$line"; inside the while() loop.

I've ran in to this problem using both system perl /usr/bin/perl and AIX toolbox perl /opt/freeware/bin/perl.
Older 5.8 and I suspect 5.10 perl versions were not affected by this memory leak, but newer such as 5.28 and 5.34 are affected.

------------------------------
Esa Kärkkäinen
------------------------------
#AIXOpenSource