Originally posted by: FlorianPommerening

Hi,

I'm on a 64 bit machine running Ubuntu 16.04 and want to compile a 32 bit binary. In older Ubuntu versions, this worked fine using g++-multilib, but now my binaries crash with a segfault. I narrowed it down to the following program:

#include "cplex.h"

using namespace std;

int main() {
    int err = 0;
    CPXENVptr env_ = CPXopenCPLEX( &err );

    return 0;
}

Compiling and running the program as a 64 bit binary works fine, but when I add the flag -m32, the resulting binary crashes in the call to CPXopenCPLEX.

The problem occurs with the 32 bit versions of cplex 12.5.1 and 12.6.2 and 12.6.3. I tried this on an older machine with an older version of libc and libpthreads and it worked. The binary that was compiled on the older machine also runs on the new machine without crashing; the binary compiled on the new machine also crashes on the old machine.

I thought that this might be caused by a difference in header files between the machines, so I copied all (directly and indirectly) included headers  from the old to the new machine, and forced the compiler to use those headers. The error still occurred, so I guess the reason must be something else. 

I also ran strace on both the working and the non-working version and both produced nearly identical output (except for specific pointer values and the end of course). The working binary ends with the lines

futex(0x8da4ba8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
brk(NULL)                               = 0x953c000
brk(0x955d000)                          = 0x955d000
time(NULL)                              = 1475841109
open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3
read(3, "0-3\n", 8192)                  = 4
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

while the non-working version ends in

futex(0x8da5ba8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
brk(NULL)                               = 0x901e000
brk(0x903f000)                          = 0x903f000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x11b0ff48} ---
+++ killed by SIGSEGV (core dumped) +++

Any ideas how to solve this?

Originally posted by: EdKlotz

It sounds like you are mixing some 64 bit .o files with some 32 bit libraries, or vice versa.

> Compiling and running the program as a 64 bit binary works fine, but when I add the flag -m32, the resulting binary crashes in the call to CPXopenCPLEX.

If you just take a 64 bit version of CPLEX and add -m32 to the compile and link step, that certainly can happen.   You will link a 32 bit .o file to a 64 bit libcplex.a,

and the resulting executable won't work.   Make sure you are compiling and linking with a 32 bit version.

> The problem occurs with the 32 bit versions of cplex 12.5.1 and 12.6.2 and 12.6.3. I tried this on an older machine with an older version of libc and libpthreads and it > worked. The binary that was compiled on the older machine also runs on the new machine without crashing; the binary compiled on the new machine also crashes > on the old machine.

This, however, sounds different since it sounds like you are compile and linking 32 bit .o and .a files.   However, you still should check that there are no -m64 foptions anywhere in the compile and link statements.

Also, since you have simplified what presumably was a C++ program to essentially  a C program, see what happens if you use gcc instead of g++.   For example, try building the lpex1 C example and see if it runs correctly.

Also

Originally posted by: FlorianPommerening

Thanks for the response. I'm using the appropriate cplex libraries for the build (32 bit libraries for the 32 bit build and 64 bit libraries for the 64 bit build). I don't think the program would link correctly otherwise.

I tried compiling lpex1.c and got the same issue. My compile call for this was;

gcc-4.8 -m32 lpex1.c /path/to/CPLEX_Studio1251/cplex/lib/x86_sles10_4.1/static_pic/libcplex.a -lm -lpthread -I/path/to/CPLEX_Studio1251/cplex/include/

The C program can be narrowed down to the same call:

#include <ilcplex/cplex.h>

int main(int argc, char **argv) {
   int status = 0;
   CPXopenCPLEX(&status);
   return (status);
}

running this also crashes with a segfault.

Originally posted by: FlorianPommerening

Running with strace also shows the following successful calls to open:

open("/lib/i386-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

So it looks like the 32 bit versions of libm, libpthread and libc are used.

Let me make sure I understand correctly:

1. This used to work.
2. You only changed Ubuntu version, nothing else, and all of a sudden it no longer works.
3. You run the 32bit binary on a 64bit machine (and that used to work before as well).
4. Can you provide a backtrace from gdb?

Originally posted by: FlorianPommerening

1. Yes, I routinely compiled with -m32 on a 64 bit machine. It still works on the other machine.

2. I cannot guarantee that I didn't change anything else. There was some time between the working and not working state where I use my computer for other things. But I think we can limit the source of this anyway: the old PC can still compile a working binary, and this binary then works on the new PC; and the binary compiled on the new PC also crashes on the old PC. I think this limits it to something that g++/gcc does while creating the binary. As far as I know, the output of gcc only depends on (a) the source code, (b) the included system headers, (c) the compiler version, and (d) any statically linked libraries. Am I missing anything here? I think my tests exclude (a) and (b). The compiler was g++ 4.8 on both machines: 4.8.5 on the new and something between 4.8.0 and 4.8.4 on the old (I don't remember the details and cannot check at the moment). The only library that I explicitly linked statically was cplex and I also copied that, so both the old and the new machine used exactly the same file. The libraries libc, m, and pthreads are linked dynamically. Is anything linked statically in an implicit way?

3. Yes and yes. Also still works like this on the old PC.

4. Sure:

#0  0x0862bb56 in ucnv_createConverter_44_cplex ()
#1  0x2d4f5349 in ?? ()
#2  0x39353838 in ?? ()
#3  0x0000312d in ?? ()
#4  0xf7dd531c in __default_morecore () from /lib/i386-linux-gnu/libc.so.6
#5  0xf7dd0f92 in ?? () from /lib/i386-linux-gnu/libc.so.6
#6  0xf7dd1b29 in ?? () from /lib/i386-linux-gnu/libc.so.6
#7  0x00000000 in ?? ()

EDIT: This was for the c++ program I originally posted, compiled g++-4.8.5 with headers and library copied from the old machine. Using the headers from the new computer leads to identical output. Using g++ 5.4.0 instead of 4.8.5 also gave similar output (only the specific addresses changed). The C program (compiled with gcc) also has the same trace and different addresses.

Thanks for the backtrace. I will try to figure out what is going wrong but actually I am not very optimistic.

I did what you described on Ubuntu 14.04 (gcc 4.8.4) here and everything works without a flaw. I don't have Ubuntu 16.04 here right now, so I cannot easily reproduce exactly what you are doing.

Originally posted by: FlorianPommerening

I did some more tests to pinpoint the error. I tried to reduce the differences between the machine that can produce a working binary and the machine that cannot. As before, when I say "working binary", I mean that it runs without segfault on both machines.

(1) Compiling without assembling lead to identical GAS code (except for the string containing the name of the compiler: "4.8.4" vs. "4.8.5").

(2) Compiling to *.o files on the new machine, then copying them to the old machine and linking there, lead to a working binary. This points to a linker error.

(3) Running the linker with verbose output showed the actual linker call (which is identical on both machines). Here it is minus some unused -L statements and with resolved relative paths:

/usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 \
--sysroot=/ --build-id --eh-frame-hdr -m elf_i386 --hash-style=gnu \
--as-needed -dynamic-linker /lib/ld-linux.so.2 -z relro \
/usr/lib32/crt1.o /usr/lib32/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtbegin.o \
-L/usr/lib/gcc/x86_64-linux-gnu/4.8/32 \
main.o \
/path/to/CPLEX_Studio1251/cplex/lib/x86_sles10_4.1/static_pic/libcplex.a \
-lpthread -lstdc++ -lm -lc -lgcc_s -lgcc \
/usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtend.o /usr/lib32/crtn.o

(4) to exclude the influence of linked system libraries, I copied all of them from the old machine to the new and used them in the linker instead of the global paths:

/usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 \
--sysroot=/ --build-id --eh-frame-hdr -m elf_i386 --hash-style=gnu \
--as-needed -dynamic-linker copied/ld-linux.so.2 -z relro \
copied/crt1.o copied/crti.o copied/crtbegin.o \
copied/main.o \
copied/libcplex.a \
copied/libpthread.so copied/libstdc++.a copied/libm.a \
copied/libc.so copied/libgcc_s.so copied/libgcc.a \
copied/crtend.o copied/crtn.o

This linker call produces a working binary on the old machine and a non-working one on the new machine using identical files in the directory "copied". I don't understand this; I thought using the same input to the linker should produce the same output. Maybe its a difference between 4.8.4 and 4.8.5?

Honestly, I have no clue what is going on. I stared hard at the source code of the offending function and could not see anything that might go wrong.

I am not sure your experiment in step 4 was correct. You were linking with libc.so, a shared library. Are you sure that when running the library is picked up from there? Or could it be the runtime linker picks up the C library from the system instead of from the copied directory. Maybe you should link statically, using -static? I am wondering whether a static link will produce a binary that works on all machines?

Could you run your program through gdb again. When it crashes type 'disassemble' and 'info registers' and attach the output here. I am still not optimistic that we can figure out the problem from there but it is more or less the last thing I can think of.

Originally posted by: FlorianPommerening

I'll try static builds next week. For now, here is the output of `disassemble` and `info registers`:

Program received signal SIGSEGV, Segmentation fault.
0x08773266 in ucnv_createConverter_44_cplex ()
(gdb) disassemble
Dump of assembler code for function ucnv_createConverter_44_cplex:
   0x08773090 <+0>:       push   %edi
   0x08773091 <+1>:       push   %esi
   0x08773092 <+2>:       push   %ebp
   0x08773093 <+3>:       push   %ebx
   0x08773094 <+4>:       sub    $0x118,%esp
   0x0877309a <+10>:      mov    0x12c(%esp),%eax
   0x087730a1 <+17>:      mov    %eax,0x114(%esp)
   0x087730a8 <+24>:      call   0x87730ad <ucnv_createConverter_44_cplex+29>
   0x087730ad <+29>:      pop    %edi
   0x087730ae <+30>:      lea    0x76ef53(%edi),%edi
   0x087730b4 <+36>:      mov    -0x4d3b98(%edi),%esi
   0x087730ba <+42>:      mov    -0x4d3b94(%edi),%ebp
   0x087730c0 <+48>:      mov    -0x4d3b90(%edi),%ebx
   0x087730c6 <+54>:      mov    -0x4d3b8c(%edi),%ecx
   0x087730cc <+60>:      mov    -0x4d3b88(%edi),%edx
   0x087730d2 <+66>:      mov    -0x4d3b84(%edi),%eax
   0x087730d8 <+72>:      mov    %edi,0x110(%esp)
   0x087730df <+79>:      mov    -0x4d3b80(%edi),%edi
   0x087730e5 <+85>:      mov    %edi,0x10(%esp)
   0x087730e9 <+89>:      mov    0x134(%esp),%edi
   0x087730f0 <+96>:      cmpl   $0x0,(%edi)
   0x087730f3 <+99>:      mov    0x10(%esp),%edi
   0x087730f7 <+103>:     jle    0x8773106 <ucnv_createConverter_44_cplex+118>
   0x087730f9 <+105>:     xor    %eax,%eax
   0x087730fb <+107>:     add    $0x118,%esp
   0x08773101 <+113>:     pop    %ebx
   0x08773102 <+114>:     pop    %ebp
   0x08773103 <+115>:     pop    %esi
   0x08773104 <+116>:     pop    %edi
   0x08773105 <+117>:     ret    
   0x08773106 <+118>:     mov    %edi,0x108(%esp)
   0x0877310d <+125>:     mov    %eax,0x104(%esp)
   0x08773114 <+132>:     mov    0x130(%esp),%eax
   0x0877311b <+139>:     mov    %edx,0x100(%esp)
   0x08773122 <+146>:     mov    %ecx,0xfc(%esp)
   0x08773129 <+153>:     mov    %ebx,0xf8(%esp)
   0x08773130 <+160>:     mov    0x134(%esp),%ebx
   0x08773137 <+167>:     mov    %ebp,0xf4(%esp)
   0x0877313e <+174>:     mov    %esi,0xf0(%esp)
   0x08773145 <+181>:     mov    %ebx,0xc(%esp)
   0x08773149 <+185>:     lea    0x10(%esp),%edx
   0x0877314d <+189>:     lea    0xf0(%esp),%ecx
   0x08773154 <+196>:     call   0x8774b5c <ucnv_loadSharedData_44_cplex.>
   0x08773159 <+201>:     mov    %eax,%ebp
   0x0877315b <+203>:     mov    0x134(%esp),%eax
   0x08773162 <+210>:     mov    (%eax),%edx
   0x08773164 <+212>:     test   %edx,%edx
   0x08773166 <+214>:     jle    0x87731ed <ucnv_createConverter_44_cplex+349>
   0x0877316c <+220>:     test   %ebp,%ebp
   0x0877316e <+222>:     je     0x877332d <ucnv_createConverter_44_cplex+669>
   0x08773174 <+228>:     cmpl   $0xffffffff,0x4(%ebp)
   0x08773178 <+232>:     je     0x877332d <ucnv_createConverter_44_cplex+669>
   0x0877317e <+238>:     mov    0x110(%esp),%ebx
   0x08773185 <+245>:     lea    0x26268(%ebx),%esi
   0x0877318b <+251>:     push   %esi
   0x0877318c <+252>:     call   0x879aeb0 <umtx_lock_44_cplex>
   0x08773191 <+257>:     pop    %ecx
   0x08773192 <+258>:     mov    0x4(%ebp),%eax
   0x08773195 <+261>:     test   %eax,%eax
   0x08773197 <+263>:     jbe    0x877319f <ucnv_createConverter_44_cplex+271>
   0x08773199 <+265>:     dec    %eax
   0x0877319a <+266>:     mov    %eax,0x4(%ebp)
   0x0877319d <+269>:     jne    0x87731d8 <ucnv_createConverter_44_cplex+328>
   0x0877319f <+271>:     movsbl 0x14(%ebp),%eax
   0x087731a3 <+275>:     test   %eax,%eax
   0x087731a5 <+277>:     jne    0x87731d8 <ucnv_createConverter_44_cplex+328>
   0x087731a7 <+279>:     mov    0x18(%ebp),%eax
   0x087731aa <+282>:     mov    0x8(%eax),%eax
   0x087731ad <+285>:     test   %eax,%eax
   0x087731af <+287>:     je     0x87731b5 <ucnv_createConverter_44_cplex+293>
   0x087731b1 <+289>:     push   %ebp
   0x087731b2 <+290>:     call   *%eax
   0x087731b4 <+292>:     pop    %ecx
   0x087731b5 <+293>:     mov    0x8(%ebp),%eax
   0x087731b8 <+296>:     test   %eax,%eax
   0x087731ba <+298>:     je     0x87731c3 <ucnv_createConverter_44_cplex+307>
   0x087731bc <+300>:     push   %eax
   0x087731bd <+301>:     call   0x87964d0 <udata_close_44_cplex>
   0x087731c2 <+306>:     pop    %ecx
   0x087731c3 <+307>:     mov    0xc(%ebp),%eax
   0x087731c6 <+310>:     test   %eax,%eax
   0x087731c8 <+312>:     je     0x87731d1 <ucnv_createConverter_44_cplex+321>
   0x087731ca <+314>:     push   %eax
   0x087731cb <+315>:     call   0x886b540 <uprv_free_44_cplex>
   0x087731d0 <+320>:     pop    %ecx
   0x087731d1 <+321>:     push   %ebp
   0x087731d2 <+322>:     call   0x886b540 <uprv_free_44_cplex>
   0x087731d7 <+327>:     pop    %ecx
   0x087731d8 <+328>:     push   %esi
   0x087731d9 <+329>:     call   0x879af10 <umtx_unlock_44_cplex>
   0x087731de <+334>:     pop    %ecx
   0x087731df <+335>:     mov    0x134(%esp),%eax
   0x087731e6 <+342>:     mov    (%eax),%edx
   0x087731e8 <+344>:     jmp    0x877332d <ucnv_createConverter_44_cplex+669>
   0x087731ed <+349>:     cmpl   $0x0,0x114(%esp)
   0x087731f5 <+357>:     je     0x8773347 <ucnv_createConverter_44_cplex+695>
   0x087731fb <+363>:     movl   $0x1,0x10c(%esp)
   0x08773206 <+374>:     push   $0x104
   0x0877320b <+379>:     push   $0x0
   0x0877320d <+381>:     pushl  0x11c(%esp)
   0x08773214 <+388>:     mov    0x11c(%esp),%ebx
   0x0877321b <+395>:     call   0x804adb0 <memset@plt>
   0x08773220 <+400>:     add    $0xc,%esp
   0x08773223 <+403>:     mov    0x10c(%esp),%eax
   0x0877322a <+410>:     mov    0x114(%esp),%ecx
   0x08773231 <+417>:     mov    %al,0x21(%ecx)
   0x08773234 <+420>:     mov    %ebp,0x18(%ecx)
   0x08773237 <+423>:     mov    0xfc(%esp),%edx
   0x0877323e <+430>:     mov    %edx,0x1c(%ecx)
   0x08773241 <+433>:     movsbl 0xf8(%esp),%ebx
   0x08773249 <+441>:     test   %ebx,%ebx
   0x0877324b <+443>:     jne    0x87732c3 <ucnv_createConverter_44_cplex+563>
   0x0877324d <+445>:     mov    %ecx,%esi
   0x0877324f <+447>:     mov    0x110(%esp),%ebx
   0x08773256 <+454>:     movl   $0xffffffff,0xb4(%esi)
    0x08773260 <+464>:    lea    0x8ee1f44,%edi
=> 0x08773266 <+470>:  mov    (%ebx,%edi,1),%edx
   0x08773269 <+473>:     mov    %edx,0x4(%esi)
   0x0877326c <+476>:     lea    0x8ee1f08,%ecx
   0x08773272 <+482>:     mov    (%ebx,%ecx,1),%eax
   0x08773275 <+485>:     mov    %eax,(%esi)
   0x08773277 <+487>:     mov    0x1c(%ebp),%edi
   0x0877327a <+490>:     mov    %edi,0x2c(%esi)
   0x0877327d <+493>:     mov    0x10(%ebp),%edx
   0x08773280 <+496>:     movzbl 0x47(%edx),%ecx
   0x08773284 <+500>:     mov    %cl,0x3c(%esi)
   0x08773287 <+503>:     mov    0x10(%ebp),%eax
   0x0877328a <+506>:     movzbl 0x50(%eax),%edx
   0x0877328e <+510>:     mov    %dl,0x42(%esi)
   0x08773291 <+513>:     mov    0x10(%ebp),%ecx
   0x08773294 <+516>:     movsbl 0x4c(%ecx),%eax
   0x08773298 <+520>:     push   %eax
   0x08773299 <+521>:     mov    %al,0x3d(%esi)
   0x0877329c <+524>:     lea    0x6c(%esi),%edi
   0x0877329f <+527>:     mov    %edi,0x14(%esi)
   0x087732a2 <+530>:     mov    0x10(%ebp),%esi
   0x087732a5 <+533>:     add    $0x48,%esi
   0x087732a8 <+536>:     push   %esi
   0x087732a9 <+537>:     push   %edi
   0x087732aa <+538>:     call   0x804b000 <memcpy@plt>
   0x087732af <+543>:     add    $0xc,%esp
   0x087732b2 <+546>:     mov    0x114(%esp),%eax
   0x087732b9 <+553>:     movl   $0x1,0x100(%eax)
   0x087732c3 <+563>:     mov    0x18(%ebp),%eax
   0x087732c6 <+566>:     mov    0xc(%eax),%edx
   0x087732c9 <+569>:     test   %edx,%edx
   0x087732cb <+571>:     je     0x87731df <ucnv_createConverter_44_cplex+335>
   0x087732d1 <+577>:     pushl  0x134(%esp)
   0x087732d8 <+584>:     lea    0xf4(%esp),%eax
   0x087732df <+591>:     push   %eax
   0x087732e0 <+592>:     pushl  0x11c(%esp)
   0x087732e7 <+599>:     call   *%edx
   0x087732e9 <+601>:     add    $0xc,%esp
   0x087732ec <+604>:     mov    0x134(%esp),%eax
   0x087732f3 <+611>:     mov    (%eax),%edx
   0x087732f5 <+613>:     test   %edx,%edx
   0x087732f7 <+615>:     jle    0x877332d <ucnv_createConverter_44_cplex+669>
   0x087732f9 <+617>:     movsbl 0xf8(%esp),%eax
   0x08773301 <+625>:     test   %eax,%eax
   0x08773303 <+627>:     jne    0x877332d <ucnv_createConverter_44_cplex+669>
   0x08773305 <+629>:     pushl  0x114(%esp)
   0x0877330c <+636>:     mov    0x114(%esp),%ebx
   0x08773313 <+643>:     call   0x8771f90 <ucnv_close_44_cplex>
   0x08773318 <+648>:     pop    %ecx
   0x08773319 <+649>:     mov    0x134(%esp),%eax
   0x08773320 <+656>:     mov    (%eax),%edx
   0x08773322 <+658>:     movl   $0x0,0x114(%esp)
   0x0877332d <+669>:     test   %edx,%edx
   0x0877332f <+671>:     jg     0x87730f9 <ucnv_createConverter_44_cplex+105>
   0x08773335 <+677>:     mov    0x114(%esp),%eax
   0x0877333c <+684>:     add    $0x118,%esp
   0x08773342 <+690>:     pop    %ebx
   0x08773343 <+691>:     pop    %ebp
   0x08773344 <+692>:     pop    %esi
   0x08773345 <+693>:     pop    %edi
   0x08773346 <+694>:     ret    
   0x08773347 <+695>:     push   $0x104
   0x0877334c <+700>:     mov    0x114(%esp),%ebx
   0x08773353 <+707>:     call   0x886b4f0 <uprv_malloc_44_cplex>
   0x08773358 <+712>:     pop    %ecx
   0x08773359 <+713>:     mov    %eax,0x114(%esp)
   0x08773360 <+720>:     cmpl   $0x0,0x114(%esp)
   0x08773368 <+728>:     je     0x877337a <ucnv_createConverter_44_cplex+746>
   0x0877336a <+730>:     movl   $0x0,0x10c(%esp)
   0x08773375 <+741>:     jmp    0x8773206 <ucnv_createConverter_44_cplex+374>
   0x0877337a <+746>:     mov    0x134(%esp),%eax
   0x08773381 <+753>:     movl   $0x7,(%eax)
   0x08773387 <+759>:     mov    $0x7,%edx
   0x0877338c <+764>:     test   %ebp,%ebp
   0x0877338e <+766>:     je     0x87733f9 <ucnv_createConverter_44_cplex+873>
   0x08773390 <+768>:     cmpl   $0xffffffff,0x4(%ebp)
   0x08773394 <+772>:     je     0x87733f9 <ucnv_createConverter_44_cplex+873>
   0x08773396 <+774>:     lea    0x26268(%ebx),%esi
   0x0877339c <+780>:     push   %esi
   0x0877339d <+781>:     call   0x879aeb0 <umtx_lock_44_cplex>
   0x087733a2 <+786>:     pop    %ecx
   0x087733a3 <+787>:     mov    0x4(%ebp),%eax
   0x087733a6 <+790>:     test   %eax,%eax
   0x087733a8 <+792>:     jbe    0x87733b0 <ucnv_createConverter_44_cplex+800>
   0x087733aa <+794>:     dec    %eax
   0x087733ab <+795>:     mov    %eax,0x4(%ebp)
   0x087733ae <+798>:     jne    0x87733e9 <ucnv_createConverter_44_cplex+857>
   0x087733b0 <+800>:     movsbl 0x14(%ebp),%eax
   0x087733b4 <+804>:     test   %eax,%eax
   0x087733b6 <+806>:     jne    0x87733e9 <ucnv_createConverter_44_cplex+857>
   0x087733b8 <+808>:     mov    0x18(%ebp),%eax
   0x087733bb <+811>:     mov    0x8(%eax),%eax
   0x087733be <+814>:     test   %eax,%eax
   0x087733c0 <+816>:     je     0x87733c6 <ucnv_createConverter_44_cplex+822>
   0x087733c2 <+818>:     push   %ebp
   0x087733c3 <+819>:     call   *%eax
   0x087733c5 <+821>:     pop    %ecx
   0x087733c6 <+822>:     mov    0x8(%ebp),%eax
   0x087733c9 <+825>:     test   %eax,%eax
   0x087733cb <+827>:     je     0x87733d4 <ucnv_createConverter_44_cplex+836>
   0x087733cd <+829>:     push   %eax
   0x087733ce <+830>:     call   0x87964d0 <udata_close_44_cplex>
   0x087733d3 <+835>:     pop    %ecx
   0x087733d4 <+836>:     mov    0xc(%ebp),%eax
   0x087733d7 <+839>:     test   %eax,%eax
   0x087733d9 <+841>:     je     0x87733e2 <ucnv_createConverter_44_cplex+850>
   0x087733db <+843>:     push   %eax
   0x087733dc <+844>:     call   0x886b540 <uprv_free_44_cplex>
   0x087733e1 <+849>:     pop    %ecx
   0x087733e2 <+850>:     push   %ebp
   0x087733e3 <+851>:     call   0x886b540 <uprv_free_44_cplex>
   0x087733e8 <+856>:     pop    %ecx
   0x087733e9 <+857>:     push   %esi
   0x087733ea <+858>:     call   0x879af10 <umtx_unlock_44_cplex>
   0x087733ef <+863>:     pop    %ecx
   0x087733f0 <+864>:     mov    0x134(%esp),%eax
   0x087733f7 <+871>:     mov    (%eax),%edx
   0x087733f9 <+873>:     movl   $0x0,0x114(%esp)
   0x08773404 <+884>:     jmp    0x877332d <ucnv_createConverter_44_cplex+669>
   0x08773409 <+889>:     lea    0x0(%esi,%eiz,1),%esi
End of assembler dump. 

And the output of `info registers`:

(gdb) info registers
eax            0x0      0
ecx            0x8f10850        150014032
edx            0x0      0
ebx            0x8ee2000        149823488
esp            0xffffc740       0xffffc740
ebp            0x8ef55a0        0x8ef55a0 <_Latin1Data_44_cplex>
esi            0x8f10850        150014032
edi            0x8ee1f44        149823300
eip            0x8773266        0x8773266 <ucnv_createConverter_44_cplex+470>
eflags         0x10246  [ PF ZF IF RF ]
cs             0x23     35
ss             0x2b     43
ds             0x2b     43
es             0x2b     43
fs             0x0      0
gs             0x63     99

Originally posted by: FlorianPommerening

Just to report back on the static build: it behaves the same as a default build, which links some libraries static and others dynamic (see linker call in my previous prost). I used the additional flags `-static -static-libgcc -static-libstdc++`. The resulting binary still crashes with a segfault.

Also, it seems you were right about my step (4). It's probably wrong to assume that binaries (*.o, *.a, *.so) can be linked with a different linker version and expected to work. For example, for the static build the link call fails with the copied files from the other computer.

Originally posted by: FlorianPommerening

I had to re-install my OS and thought I'd try again on a relatively clean installation. I upgraded to ubuntu 16.10 which uses g++ 6.2.0 by default. I installed the 32 bit version of CPLEX 12.6.3 in '/opt/cplex-1263-32' and then ran the following compiler call

g++ -m32 main.cpp -I /opt/cplex-1263-32/cplex/include/ilcplex /opt/cplex-1263-32/cplex/lib/x86_linux/static_pic/libcplex.a -lpthread

The cpp file is still the one from my first post. With g++ 6.2.0 I now get an error message while linking:

/usr/bin/ld: /opt/cplex-1263-32/cplex/lib/x86_linux/static_pic/libcplex.a(ucnv.ao): direct GOT relocation R_386_GOT32 against `UCNV_FROM_U_CALLBACK_STOP_44_cplex' without base register can not be used when making a shared object
/usr/bin/ld: final link failed: Bad value

With g++ 4.8.5 I get no linker error but still the segmentation fault mentioned in the first post. I thought about opening a new question about this since the symptoms are different, but it could be related. For example, g++ 4.8 maybe just does not notice the error at compile time and then runs into it at runtime. Let me know if you think this is a separate issue that I should move to a new topic.

Originally posted by: FlorianPommerening

Forgot say: adding --static to the build options leads to a successfully linked build that segfaults at runtime.
 

Originally posted by: LaszloLadanyi

Apparently binutils (which contains the runtime loader and the linker) was a bit more lax with 32-bit libs/executables in the past. As you can see from HJ Lu's commit into binutils (http://www.sourceware.org/ml/binutils-cvs/2016-06/msg00091.html) it became stricter fairly recently. It look like the compiler that was used to compile that portion of the cplex code generated code that does not satisfy that scrictness. Since in the latest cplex version (12.7) 32-bit linux platform is not even supported I see practically no chance that you can run 32-bit cplex on a linux distribution that has this binutils change. Just curiosity: why don't you switch to 64-bit?

--Laci

Originally posted by: FlorianPommerening

Thanks! What I still find surprising is that (much) older versions of binutils work fine with the same cplex code. Well, like you said, there is probably no way around this.

I didn't know that 12.7 no longer supported 32 bit. This is bad news for us. The project we work on usually runs with memory limited to 2GB and scales further with more available memory, switching to 64 bit would mean that we lose a lot of memory because all pointers are twice as big. We considered switching to 64 bit before but last time we measured, our benchmarks were about 10% faster but used about 50% more memory. Overall, our performance dropped significantly because of that.

Originally posted by: LaszloLadanyi

There are several things you can do about memory consumption:

1) If you use an API (as opposed to reading problem instances from a file) then do not give names to the variables and constraints, if possible. Depending on the API you use this may or may not be possible. Also, you may want to have names in callbacks. Anyway, for various reasons cplex keeps the names around with every copy of the problem instance in the memory (and there are quite a few of them: original, presolved, row-wise, column-wise, etc.). If you don't use names that will save quite a bit of memory.

2) Set a couple of parameters. You can try to set CPX_PARAM_MEMORYEMPHASIS to CPX_ON, but that is a fairly drastic change and will affect a number of algorithmic decisions. However, take a look at the page https://www.ibm.com/support/knowledgecenter/SSSA5P_12.6.3/ilog.odms.cplex.help/CPLEX/UsrMan/topics/discr_optim/mip/troubleshoot/61_mem_gone.html. There a few parameters are described that will help you to save part of the search tree into files. File access is much slower than memory access, but it's still better than running out of memory :-), and cplex does a reasonable job at handling these files efficiently. I'd suggest play with these parameters a bit.

Good luck!

--Laci

Originally posted by: FlorianPommerening

I should have been more explicit in my response: the part that eats up our memory is (usually) not cplex. We perform a search where we use cplex to come up with heuristic values for each search node. Storing the search nodes is mostly the memory intense part. We build up our cplex model at runtime through the OSI interface once, then adapt it in each search node by changing bounds, re-solve it and use the objective value as a heuristic guiding the overall search. This means that our models are comparatively small, but we have to re-solve them thousands to millions of times per hour. There are exceptions, though, where the model is larger and takes longer to solve. In those cases your tips could come in handy. We already do not set names for things (only for debugging), but I'll have a look at the memory emphasis parameter.

Just for completeness:

I did not yet have time to track down the issue with the help of your assembler dump. But what I can say is that the segfault indeed happens in code related to the offending object file and that it looks like an (indirect) function call is triggering the segfault. So this linker error may indeed be related to the segfault and it is probably indeed the same issue.

Like Laszlo said, I don't see an easy (if any) way around this.

If I understood correctly, then the increased memory footprint for 64bit does not come from CPLEX but from the application build around CPLEX? In that case trying to save memory in CPLEX is not going to help a lot.

But here is another small tip that came to my mind while reading the description of your application: If you only change bounds etc. then CPLEX will keep the solutions of the previous solve as MIP start for the next solve. Depending on how much the problem changes between two invocations of CPLEX, this may turn out to be a complete waste of time because the MIP start from the previous solve is not feasible and not even close to feasible for the next solve. So it may be worth while to disable MIP starts and set CPX_PARAM_ADVIND to 0.

Originally posted by: FlorianPommerening

Thanks. We mostly use LPs, but I'll give it a try next time we use MIPs.