Originally posted by: OlivierCailloux

Hello,

I thought there would be no difference between solving a given problem through the use of the cplex interactive solver or by using the Cplex API (through a Java program), in both cases using the default parameter values. Apparently I'm wrong, or I am doing something wrong. Any documentation on that subject (is it supposed to be identical?) or on what is going on here would be appreciated. More details about what I tested follow.

I have one given problem (a big MIP) which I tried to solve using Cplex interactive and Java. Apparently the path taken to solve the problem is completely different in these two cases, although I use the deterministic mode and (AFAI can tell) the same parameters values, as I did not change any.

• Solve from Cplex interactive (all default parameters). Does not solve: memory error. (The following is according to cplex output.) After 1219.01 sec., tree size is 225.63 MB, solutions = 0. After 2600.84 sec., tree size is 2199.24 MB, solutions = 8, Nodefile size is 2039.74 MB (1266.19 MB after compression). After 3102.55 sec., tree size is 2989.59 MB, solutions = 8, Nodefile size is 2836.30 MB (1759.30 MB after compression) and there it stops (CPLEX Error 1001: Out of memory).
• Solve from Java calling the Cplex library (all default parameters). Uses much less memory and finally solves the problem to optimality. (The following is according to cplex output.) After 2998.24 sec., the tree size is still only 85.36 MB, solutions = 11. The tree size grows up to 800 MiB (after 31k seconds) then decreases to zero. Problem solved after about 47500 seconds.

Same behavior each time I tested (more than three times each), including after a reboot.
In both cases, the program runs on two threads (my system is a dual core). Cplex interactive says when starting “Parallel mode: deterministic, using up to 2 threads.”
Other quick tries on smaller problems show a systematic important difference in solve time, although this is the same computer, whether solving in deterministic (Cplex interactive is faster in my tries) or in opportunistic multi-threading mode (in which case interactive is not always faster than Java-based).

Here is the Java code. Below is the beginning of the Cplex interactive session.

package fr.ecp.lgi.cplex;
 
import ilog.cplex.IloCplex;
import ilog.cplex.IloCplex.CplexStatus;
import ilog.cplex.IloCplex.DoubleParam;
import ilog.cplex.IloCplex.IntParam;
import ilog.cplex.IloCplex.ParallelMode;
 
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
public class CplexTries {
    private static final Logger s_logger = LoggerFactory.getLogger(CplexTries.class);
 
    public static void main(String[] args) {
        final CplexTries tries = new CplexTries();
        tries.testMemoryError();
    }
 
    private static final int CPLEX_CLOCK_TYPE_CPU = 1;
    public static final int CPLEX_CLOCK_TYPE_WALL = 2;
    private static final int CPLEX_MULTI_THREAD = 0;
    public static final int CPLEX_SINGLE_THREAD = 1;
 
    public void testMemoryError() {
        final String modelName = "mem-error.lp";
        s_logger.info("Begin test memory error.");
        IloCplex solver = null;
        try {
            solver = new IloCplex();
            solver.importModel(modelName);
            long begin = System.currentTimeMillis();
            solver.solve();
            long end = System.currentTimeMillis();
            final long systTime = end - begin;
            s_logger.info("Status: " + solver.getCplexStatus() + ", " + systTime + " ms.");
        } catch (Exception exc) {
            s_logger.error("Error caught.", exc);
        } finally {
            if (solver != null) {
                solver.end();
            }
        }
    }

IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
 
IBM ILOG CPLEX Optimization Studio Academic Research Edition
Welcome to IBM(R) ILOG(R) CPLEX(R) Interactive Optimizer 12.2.0.0
  with Simplex, Mixed Integer & Barrier Optimizers
5725-A06 5725-A29 5724-Y48 5724-Y49 5724-Y54 5724-Y55
Copyright IBM Corp. 1988, 2010.  All Rights Reserved.
 
Type 'help' for a list of available commands.
Type 'help' followed by a command name for more
information on commands.
 
CPLEX> read mem-error.lp 
Problem 'mem-error.lp' read. 
Read time =    0.03 sec. 
CPLEX> optimize 
Tried aggregator 1 time. 
MIP Presolve eliminated 132 rows and 42 columns. 
MIP Presolve modified 5424 coefficients. 
Reduced MIP has 10165 rows, 3781 columns, and 24670 nonzeros. 
Reduced MIP has 1246 binaries, 0 generals, 0 SOSs, and 0 indicators. 
Probing time =    0.20 sec. 
Tried aggregator 1 time. 
MIP Presolve eliminated 640 rows and 320 columns. 
MIP Presolve modified 2492 coefficients. 
Reduced MIP has 9525 rows, 3461 columns, and 23390 nonzeros. 
Reduced MIP has 926 binaries, 0 generals, 0 SOSs, and 0 indicators. 
Presolve time =    0.29 sec. 
Probing time =    0.01 sec. 
Clique table members: 16237. 
MIP emphasis: balance optimality and feasibility. 
MIP search method: dynamic search. 
Parallel mode: deterministic, using up to 2 threads. 
Root relaxation solution time =    0.22 sec.

Thank you for any help.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

We reproduced the issue here and are looking into it.
I will get back to you as soon as we have figured out what is going on.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Just to make sure: Is it possible that you have different versions of CPLEX installed and your Java is using a different version of CPLEX than the interactive? Can you please double-check if the CPLEX shared library used by your Java code is of the release as the interactive you are using?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: OlivierCailloux

Thanks for investigating.

Your first answer gave me good hope, but your second one seems to contradict the first information... ?

Anyway here is what I tried to make sure that both versions are indeed the same. I indeed have several cplex versions on my computer but I tried to move the link file in jni to make sure that java crashes (proving that it is the one it uses). See below.

Do you know how I can make sure that the versions are identical in a cleaner way? Some way to ask for an exact version number, maybe?

olivier@Camus:~/Desktop/SharedProfiles-0.5-SNAPSHOT$ sudo mv /usr/lib/jni/libcplex122.so .
olivier@Camus:~/Desktop/SharedProfiles-0.5-SNAPSHOT$ java -classpath "lib/*:conf" fr.ecp.lgi.cplex.CplexTries
java.lang.UnsatisfiedLinkError: no cplex122 in java.library.path
java.library.path must point to the directory containing the CPLEX shared library
try invoking java with java -Djava.library.path=...
Exception in thread "main" java.lang.UnsatisfiedLinkError: ilog.cplex.Cplex.CPXopenCPLEX([I)J
at ilog.cplex.Cplex.CPXopenCPLEX(Native Method)
at ilog.cplex.CplexI.init(CplexI.java:5391)
at ilog.cplex.CplexI.<init>(CplexI.java:525)
at ilog.cplex.IloCplex.<init>(IloCplex.java:9418)
at fr.ecp.lgi.cplex.CplexTries.testMemoryError(CplexTries.java:36)
at fr.ecp.lgi.cplex.CplexTries.main(CplexTries.java:18)

olivier@Camus:~/Desktop/SharedProfiles-0.5-SNAPSHOT$ sudo mv libcplex122.so /usr/lib/jni/
olivier@Camus:~/Desktop/SharedProfiles-0.5-SNAPSHOT$ java -classpath "lib/*:conf" fr.ecp.lgi.cplex.CplexTries
IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
Tried aggregator 1 time.
(snip)
Elapsed real time = 1117.13 sec. (tree size = 29.78 MB, solutions = 0)
(snip)
Elapsed real time = 1204.97 sec. (tree size = 31.71 MB, solutions = 0)
(snip)

olivier@Camus:~/Desktop/SharedProfiles-0.5-SNAPSHOT$ ls -l /usr/lib/jni/
total 400
lrwxrwxrwx 1 root root 88 sep 2 02:39 libcplex122.so -> /home/olivier/Logiciels/Développement/ILog/12.2/cplex/bin/x86_sles10_4.1/libcplex122.so

olivier@Camus:~/Desktop/SharedProfiles-0.5-SNAPSHOT$ /home/olivier/Logiciels/Développement/ILog/12.2/cplex/bin/x86_sles10_4.1/cplex
IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
(snip)
Elapsed real time = 1365.28 sec. (tree size = 25.99 MB, solutions = 0)
(snip)
Elapsed real time = 1458.86 sec. (tree size = 84.96 MB, solutions = 0)
(snip)
Elapsed real time = 1548.16 sec. (tree size = 232.40 MB, solutions = 0)
Nodefile size = 93.27 MB (59.94 MB after compression)
(snip)

olivier@Camus:~/Desktop/SharedProfiles-0.5-SNAPSHOT$ ls -l "/home/olivier/Logiciels/Développement/ILog/12.2/cplex/bin/x86_sles10_4.1"
total 19996
-rw-r--r-- 1 olivier olivier 289 sep 2 14:45 clone0.log
-rwxrwxr-x 1 olivier olivier 36772 aoû 4 15:24 convert
-rwxrwxr-x 1 olivier olivier 6557640 aoû 4 15:24 cplex
-rwxrwxr-x 1 olivier olivier 6406720 aoû 4 15:24 cplexamp
-rw-r--r-- 1 olivier olivier 1997 sep 2 14:45 cplex.log
-rwxrwxr-x 1 olivier olivier 7422524 aoû 4 15:24 libcplex122.so
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

The problem was that by accident we used opportunistic mode to reproduce the problem. So in fact we have not reproduced it yet.
In general, there is no guarantee that Concert and interactive produce the same solution as some minor problem transformations take place in the Concert layer. However, in your example I cannot see a reason why this should happen.
From your post it really looks like you are using the same versions of CPLEX. Can you post the full log files of the CPLEX and the Concert solve (or at least the logs until the first difference occurs that is not related only to time)?
What happens if you say solver.exportModel("model.sav") right before solver.solve() and solve the produced model.sav with the interactive? Does it produce the same output as Concert?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: OlivierCailloux

Hi again, sorry for the delay. Here attached, related to the first post:

• cplex interactive, read mem-error.lp, optimize => three log files (including clone0 and clone1);
• from java, using concert, read mem-error.lp, optimize => one log file (output of the program).

Then I exported to mem-error.sav as suggested and optimized that with cplex interactive. Also attached (three log files).

A diff shows few differences between cplex interactive mem-error.lp and cplex interactive mem-error.sav (although still a few ones, oddly) but a lot of differences with the concert-java one.
#CPLEXOptimizers
#DecisionOptimization