Originally posted by: FCW

We are using CPLEX 12.1.0 with JAVA 1.6.0_17 on Red Hat Linux 5.5 64bit. The optimization has been scheduled as a cron job to run every night since May 2010. Except in only a few occasions, the optimization usually completes without failure. These two days the optimization crashes after the first 5-6 mins of solving a large QP using the barrier algorithm. As part of the cronjob we are setting the environment variable
MKL_DISABLE_FAST_MM=1. If anyone can offer some help, it will be appreciated.
Here is the CPLEX output before it crashed in one instance:

nRows 2004471
nCols 17954262
Presolve has eliminated 44705 rows and 12027075 columns...
Presolve has eliminated 398263 rows and 12397357 columns...
Presolve has eliminated 406598 rows and 12443425 columns...
Presolve has eliminated 406605 rows and 12443877 columns...
Presolve has eliminated 406605 rows and 12443889 columns...
Aggregator has done 412 substitutions...
Presolve has eliminated 406863 rows and 12444138 columns...
Aggregator has done 412 substitutions...
Tried aggregator 1 time.
QP Presolve eliminated 406863 rows and 12444138 columns.
Aggregator did 412 substitutions.
Reduced QP has 1597196 rows, 5640261 columns, and 31967709 nonzeros.
Reduced QP objective Q matrix has 941248 nonzeros.
Presolve time = 39.24 sec.
Parallel mode: using up to 16 threads for barrier.
Number of nonzeros in lower triangle of A*A' = 49408870
Elapsed ordering time = 32.08 sec.
Elapsed ordering time = 52.11 sec.
Elapsed ordering time = 73.18 sec.
Elapsed ordering time = 94.52 sec.
Using Nested Dissection ordering
Total time for automatic ordering = 96.24 sec.
Summary statistics for Cholesky factor:
Threads = 16
Rows in Factor = 1597196
Integer space required = 18041073
Total non-zeros in factor = 375571395
Total FP ops to factor = 326698514909
Here is the JVM output at the time of failure:

*** glibc detected *** /project/tools/jre1.6.0_17/bin/java: free(): invalid pointer: 0x00002abae4016d80 ***
======= Backtrace: =========
/lib64/libc.so.60x3fab67230f
/lib64/libc.so.6(cfree+0x4b)0x3fab67276b
/project/tools/libcplex121.so(mkl_serv_deallocate+0xb2)0x2abade2a9280
/project/tools/libcplex121.so(mkl_blas_mc3_dgemm_freebufs+0x6)0x2abade2da50c
/project/tools/libcplex121.so(mkl_blas_mc3_xdtrsm+0x3e0)0x2abade2a0838
/project/tools/libcplex121.so(mkl_blas_xdtrsm+0xb6)0x2abade28aebe
/project/tools/libcplex121.so(mkl_blas_dtrsm+0x3c)0x2abade1e4ab8
/project/tools/libcplex121.so(dtrsm+0xe8)0x2abade1e4a14
/project/tools/libcplex121.so0x2abade17aad9
/project/tools/libcplex121.so0x2abade176cc1
/project/tools/libcplex121.so0x2abade175b74
/project/tools/libcplex121.so0x2abaddfd3b76
/lib64/libpthread.so.00x3fac20673d
/lib64/libc.so.6(clone+0x6d)0x3fab6d3d1d

An error file was generated. Here is the content at the beginning of the log file:

#

1. A fatal error has been detected by the Java Runtime Environment:

#

1. SIGSEGV (0xb) at pc=0x0000003fab671cd6, pid=21674, tid=1138063680

#

1. JRE version: 6.0_17-b04
2. Java VM: Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode linux-amd64 )
3. Problematic frame:
4. C http://libc.so.6+0x71cd6

#

1. If you would like to submit a bug report, please visit:
2. http://java.sun.com/webapps/bugreport/crash.jsp
3. The crash happened outside the Java Virtual Machine in native code.
4. See problematic frame for where to report the bug.

#

_ In another instance, the JVM output is slightly different but the problematic frame is the same. _

*** glibc detected *** /project/tools/jre1.6.0_17/bin/java: double free or corruption (out): 0x00000000566d4180 ***
======= Backtrace: ========= /lib64/libc.so.60x3fab67230f /lib64/libc.so.6(cfree+0x4b)0x3fab67276b
/project/tools/libcplex121.so(mkl_serv_deallocate+0xb2)0x2abac14db280
/project/tools/libcplex121.so(mkl_blas_mc3_dgemm_freebufs+0x6)0x2abac150c50c
/project/tools/libcplex121.so(mkl_blas_mc3_xdtrsm+0x3e0)0x2abac14d2838
/project/tools/libcplex121.so(mkl_blas_xdtrsm+0xb6)0x2abac14bcebe
/project/tools/libcplex121.so(mkl_blas_dtrsm+0x3c)0x2abac1416ab8
/project/tools/libcplex121.so(dtrsm+0xe8)0x2abac1416a14
/project/tools/libcplex121.so0x2abac13acad9
/project/tools/libcplex121.so0x2abac13a8cc1
/project/tools/libcplex121.so0x2abac13a7b74
/project/tools/libcplex121.so0x2abac1205b76
/lib64/libpthread.so.00x3fac20673d
/lib64/libc.so.6(clone+0x6d)0x3fab6d3d1d
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Is it possible for you to upgrade to the newest CPLEX version? This would currently be CPLEX 12.2.0.2. Maybe, this includes a bug fix that applies to your situation. In particular, you would no longer need the MKL workaround.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: FCW

I looked through the issues that 12.1.1 fixed but they do not look like my problem. Where can I find information on the bug fixes in 12.2.0.2?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

There is a README file that you can download in the same way you would download the 12.2.0.2 release, see
http://www.ibm.com/developerworks/forums/thread.jspa?threadID=357454&tstart=0
#CPLEXOptimizers
#DecisionOptimization