Originally posted by: Oshkosh AIX

When upgrading httpd from 2.4.39-1 to the AIX Toolbox 2.4.39-2, I get the following error when I try to restart the httpd daemon (we are running AIX 7200-03-03):

rtld: 0712-001 Symbol _GLOBAL__AIXI_libexpat_so was referenced
from module httpd(), but a runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol _GLOBAL__AIXD_libexpat_so was referenced
from module httpd(), but a runtime definition
of the symbol was not found.

Steps to Reproduce:
I looked at the expat package installed and we are the latest level (AIX Toolbox 2.2.6-1).

When I remove the new version of http and reinstall the old version (2.4.39-1), the issue goes away and httpd starts normally.

If I try to update the package again, the error presents itself when I try to run the httpd start command. The outcome is that the daemon will not start.

There were no entries in the httpd error_log, only the successful starts I had when I reinstalled the older version. 

Originally posted by: Oshkosh AIX

An additional note.

I ran the dump -Tv on the libexpat.a and found that the symbols were listed:

/opt/freeware/lib/libexpat.a[libexpat.so.1]:

                        ***Loader Section***

                        ***Loader Symbol Table Information***
[Index]      Value      Scn     IMEX Sclass   Type           IMPid Name

[0]     0x20001f34    .data              RW SECdef        [noIMid] __rtinit
[1]     0x00000000    undef      IMP     DS EXTref libgcc_s.a(shr.o) __cxa_finalize
[2]     0x00000000    undef      IMP     DS EXTref libgcc_s.a(shr.o) __register_frame_info_table_bases
[3]     0x00000000    undef      IMP     DS EXTref libgcc_s.a(shr.o) __deregister_frame_info
[4]     0x00000000    undef      IMP     DS EXTref libgcc_s.a(shr.o) _GLOBAL__AIXI_shr_o
[5]     0x00000000    undef      IMP     DS EXTref libgcc_s.a(shr.o) _GLOBAL__AIXD_shr_o
[6]     0x0000e008    undef      IMP     XO EXTref   libc.a(shr.o) ___memset
[7]     0x0000f000    undef      IMP     XO EXTref   libc.a(shr.o) ___memmove
[8]     0x00000000    undef      IMP     UA EXTref   libc.a(shr.o) errno
[9]     0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) close
[10]    0x00000000    undef      IMP     RW EXTref   libc.a(shr.o) _iob
[11]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) malloc
[12]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) realloc
[13]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) free
[14]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) open64
[15]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) fprintf
[16]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) getenv
[17]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) getpid
[18]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) gettimeofday
[19]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) __assert
[20]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) read
[21]    0x00000000    undef      IMP     DS EXTref   libc.a(shr.o) memcmp
[22]    0x10026d30    .text      EXP     RO   Ldef        [noIMid] _GLOBAL__F_xmlparse.c_4503BEFD_0x75ac9fd96ea5b1
[23]    0x100287f0    .text      EXP     RO   Ldef        [noIMid] _GLOBAL__F_xmltok.c_EEA89552_0xadb8480e4b5aa7
[24]    0x1002a140    .text      EXP     RO   Ldef        [noIMid] _GLOBAL__F_xmlrole.c_468738AA_0xa40f3df7cfc0b6
[25]    0x20002304    .data      EXP     DS   Ldef        [noIMid] XmlInitUnknownEncoding
[26]    0x20002310    .data      EXP     DS   Ldef        [noIMid] XmlInitUnknownEncodingNS
[27]    0x2000231c    .data      EXP     DS   Ldef        [noIMid] XmlParseXmlDeclNS
[28]    0x20002328    .data      EXP     DS   Ldef        [noIMid] XmlParseXmlDecl
[29]    0x20002340    .data      EXP     DS   Ldef        [noIMid] XmlInitEncoding
[30]    0x2000234c    .data      EXP     DS   Ldef        [noIMid] XmlInitEncodingNS
[31]    0x200026a0    .data      EXP     DS   Ldef        [noIMid] _GLOBAL__AIXI_libexpat_so
[32]    0x200026ac    .data      EXP     DS   Ldef        [noIMid] _GLOBAL__AIXD_libexpat_so

...
 

Originally posted by: AyappanP

Do you have LIBPATH exported ?

I think httpd is not finding the correct expat library. 

Paste the output of "ldd /opt/freeware/sbin/httpd"

Originally posted by: Oshkosh AIX

LIBPATH is exported:

#/ >echo $LIBPATH
/usr/lib:/opt/freeware/lib:lib

#/ >ldd /opt/freeware/sbin/httpd
/opt/freeware/sbin/httpd needs:
         /usr/lib/libpcre.a(libpcre.so.1)
         /opt/freeware/lib/libaprutil-1.so
         /opt/freeware/lib/libapr-1.so
         /usr/lib/libpthread.a(shr_xpg5.o)
         /usr/lib/libc.a(shr.o)
         /usr/lib/librtl.a(shr.o)
         /opt/freeware/lib/libgcc_s.a(shr.o)
         /usr/lib/libexpat.a(libexpat.so.1)
         /usr/lib/libiconv.a(libiconv.so.2)
ar: 0707-109 Member name libiconv.so.2 does not exist.
dump: /tmp/tmpdir9699732/extract/libiconv.so.2: 0654-106 Cannot open the specified file.
         /unix
         /usr/lib/libpthreads.a(shr_comm.o)
         /usr/lib/libcrypt.a(shr.o)

What I find interesting is that everything works fine running httpd (2.4.39-1), but does not when I update to the newer version (2.4.39-2).

Originally posted by: AyappanP

Paste the below outputs.

ls -l /usr/lib/libexpat.a

dump -Tov /usr/lib/libexpat.a | grep GLOBAL

Originally posted by: Oshkosh AIX

Here is the output from those two commands:

#: / >ls -l /usr/lib/libexpat.a
lrwxrwxrwx    1 root     system           33 Sep 04 10:39 /usr/lib/libexpat.a -> ../../opt/freeware/lib/libexpat.a

#: / >dump -Tov /usr/lib/libexpat.a | grep GLOBAL
[4]     0x00000000    undef      IMP     DS EXTref libgcc_s.a(shr.o) _GLOBAL__AIXI_shr_o
[5]     0x00000000    undef      IMP     DS EXTref libgcc_s.a(shr.o) _GLOBAL__AIXD_shr_o
[22]    0x10026d30    .text      EXP     RO   Ldef        [noIMid] _GLOBAL__F_xmlparse.c_4503BEFD_0x75ac9fd96ea5b1
[23]    0x100287f0    .text      EXP     RO   Ldef        [noIMid] _GLOBAL__F_xmltok.c_EEA89552_0xadb8480e4b5aa7
[24]    0x1002a140    .text      EXP     RO   Ldef        [noIMid] _GLOBAL__F_xmlrole.c_468738AA_0xa40f3df7cfc0b6
[31]    0x200026a0    .data      EXP     DS   Ldef        [noIMid] _GLOBAL__AIXI_libexpat_so
[32]    0x200026ac    .data      EXP     DS   Ldef        [noIMid] _GLOBAL__AIXD_libexpat_so

NOTE: All these outputs are from after I rolled back to http version 2.4.39-1

Originally posted by: AyappanP

You seem to have the proper expat library. 

Not sure why httpd couldn't able to recognize the symbols. 

Just make sure you don't have any other expat library present in the system. 

Originally posted by: Oshkosh AIX

Confirmed that there is only one version of expat installed on the server and that it is the AIX Toolbox version 2.2.6-1. Is there a resource I can contact to see if this is a bug with the new version of AIX Toolbox httpd?

Originally posted by: AyappanP

Try the below things.

Stop httpd and then run "slibclean" and start it again.

unset LIBPATH and try starting it again

Originally posted by: Oshkosh AIX

Just to confirm. Should I take these steps prior to trying the upgrade or immediately after the upgrade. 

Thanks.

Originally posted by: AyappanP

Immediately after the upgrade. 

Originally posted by: Oshkosh AIX

The running of the slibclean command appears to have resolved the issue. I ran the update of httpd, ran slibclean, and was able to restart the httpd daemon successfully on both our test and production http servers.

Thanks again for the assistance.