Originally posted by: P1et

I'm using the JAVA library to solve a CPLEX problem but I've noticed that the optimizer keeps exceeding my memory limits. I used the following commands in Java to limit my memory and turn off file nodes:

cplex.setParam(IloCplex.Param.WorkMem, 2048);
cplex.setParam(IloCplex.Param.MIP.Limits.TreeMemory, 2048);
cplex.setParam(IloCplex.Param.MIP.Strategy.File, 0);

I run my Java jar using: java -Xmx2048M -jar ....

Yet when I look at my used memory I see that over 10GB of my 16GB server RAM is being used by my Cplex optimizer... Here is a log from my optimizer when it starts exceeding my RAM limit:

Found incumbent of value 5000.000000 after 0.20 sec. (97.49 ticks)
Tried aggregator 1 time.
MIP Presolve eliminated 1350000 rows and 1 columns.
MIP Presolve modified 155200 coefficients.
Reduced MIP has 163020 rows, 1520000 columns, and 6770000 nonzeros.
Reduced MIP has 1520000 binaries, 0 generals, 0 SOSs, and 0 indicators.
Presolve time = 17.53 sec. (4938.36 ticks)
Tried aggregator 1 time.
Reduced MIP has 163020 rows, 1520000 columns, and 6770000 nonzeros.
Reduced MIP has 1520000 binaries, 0 generals, 0 SOSs, and 0 indicators.
Presolve time = 13.57 sec. (2914.25 ticks)
Probing time = 6.46 sec. (300.74 ticks)
Clique table members: 497953.
MIP emphasis: balance optimality and feasibility.
MIP search method: dynamic search.
Parallel mode: deterministic, using up to 4 threads.
Root relaxation solution time = 49.45 sec. (10161.56 ticks)

And then it will try to decrease the gap % until it solved the problem or in my case until my kernel decides to kill the process due to heavy memory usage and exhausting the system memory.

Can someone please help me figure out why the optimizer is exceeding the 2GB memory limits?

Thank you.

Can you show us the full log output?

In particular, there should be lines like these in the log:

Elapsed time = 160.14 sec. (42922.58 ticks, tree = 23.30 MB, solutions = 30)

What is the tree size in the last of these lines that you can see? Also, can you show us the relevant parts of your code? It looks odd that you should run out of memory when setting the tree limit to 2 GB.

Originally posted by: P1et

Alright here is a full log for when the problem is very large. I noticed that my RAM already exceeds the allowed 2GB before the optimizer attempts to decrease the gap (see log).

Found incumbent of value 5000.000000 after 0.22 sec. (97.49 ticks)
Tried aggregator 1 time. (NOTE: Java RAM usage already goes over 2GB at this step)
MIP Presolve eliminated 1350000 rows and 1 columns.
MIP Presolve modified 155200 coefficients.
Reduced MIP has 163020 rows, 1520000 columns, and 6770000 nonzeros.
Reduced MIP has 1520000 binaries, 0 generals, 0 SOSs, and 0 indicators.
Presolve time = 15.54 sec. (4938.36 ticks)
Tried aggregator 1 time.
Reduced MIP has 163020 rows, 1520000 columns, and 6770000 nonzeros.
Reduced MIP has 1520000 binaries, 0 generals, 0 SOSs, and 0 indicators.
Presolve time = 9.55 sec. (2914.25 ticks)
Probing time = 16.82 sec. (354.89 ticks)
Clique table members: 2607510.
MIP emphasis: balance optimality and feasibility.
MIP search method: dynamic search.
Parallel mode: deterministic, using up to 4 threads. (NOTE: At this point my Java memory is already going up to 4GB, way above the allowed limit).
Root relaxation solution time = 30.34 sec. (7539.76 ticks)

        Nodes                                         Cuts/
   Node  Left     Objective  IInf  Best Integer    Best Bound    ItCnt     Gap

*     0+    0                         5000.0000        0.0000           100.00%
      0     0 -1.00000e+037     0     5000.0000        0.0000   112025  100.00%

Root node processing (before b&c):
  Real time             =  101.95 sec. (18382.87 ticks)
Parallel b&c, 4 threads:
  Real time             =    0.00 sec. (0.00 ticks)
  Sync time (average)   =    0.00 sec.
  Wait time (average)   =    0.00 sec.
                          ------------
Total (root+branch&cut) =  101.95 sec. (18382.87 ticks)

I had to set a time limit of 100 seconds. Without the time limit it would just continue "Parallel mode: deterministic, using up to 4 threads." until my system shuts down Java due to resource suffication (out of memory message by Kernel).

I did a smaller test where I allowed 1GB workMem and TreeLimit while Java launched with -Xmx2048M but unfortunately I never received the "elapsed time" output that you are looking for, even after 30 min of testing.

Found incumbent of value 1500.000000 after 0.05 sec. (19.40 ticks)
Tried aggregator 1 time.
MIP Presolve eliminated 195000 rows and 1 columns.
MIP Presolve modified 99124 coefficients.
Reduced MIP has 107013 rows, 303750 columns, and 1395750 nonzeros.
Reduced MIP has 303750 binaries, 0 generals, 0 SOSs, and 0 indicators.
Presolve time = 2.40 sec. (880.46 ticks)
Probing time = 0.48 sec. (64.00 ticks)
Tried aggregator 1 time.
Reduced MIP has 107013 rows, 303750 columns, and 1395750 nonzeros.
Reduced MIP has 303750 binaries, 0 generals, 0 SOSs, and 0 indicators.
Presolve time = 1.78 sec. (630.27 ticks)
Probing time = 0.58 sec. (63.53 ticks)
Clique table members: 152274.
MIP emphasis: balance optimality and feasibility.
MIP search method: dynamic search.
Parallel mode: deterministic, using up to 4 threads.
Root relaxation solution time = 45.13 sec. (11105.93 ticks) (NOTE: now at 800MB memory)

        Nodes                                         Cuts/
   Node  Left     Objective  IInf  Best Integer    Best Bound    ItCnt     Gap

*     0+    0                         1500.0000        0.0000           100.00%
      0     0     1232.7019  4214     1500.0000     1232.7019       35   17.82%
      0     0     1240.8771  5857     1500.0000    Cuts: 2763    32010   17.27%
*     0+    0                         1490.0000     1240.8771            16.72%
      0     0     1249.6942  7174     1490.0000    Cuts: 2814    88374   16.13% (NOTE: now at 1.4GB RAM, over the allowed 1024M limit)

After 30 min this is all the output I got. Unfortunately I only get the "elapsed time" output when a solution can be found in a very reasonable time for smaller problems.

Here is all my relevant cplex code:

        IloOplFactory.setDebugMode(false);
        IloOplFactory oplF = new IloOplFactory();
        IloOplErrorHandler errHandler = oplF.createOplErrorHandler(System.out);
        IloOplModelSource modelSource = oplF.createOplModelSource(mod);
        
        Log.printLine("Starting CPLEX creation");
        IloCplex cplex = oplF.createCplex();
        IloOplSettings settings = oplF.createOplSettings(errHandler);
        IloOplModelDefinition def = oplF.createOplModelDefinition(modelSource, settings);
        IloOplModel model = oplF.createOplModel(def, cplex);
        
        // time limit (OK)
        cplex.setParam(IloCplex.Param.TimeLimit, SimManager.timeLimit);
        
        // memory limit (NOT WORKING)
        cplex.setParam(IloCplex.Param.WorkMem, 2048);
        cplex.setParam(IloCplex.Param.MIP.Limits.TreeMemory, 2048);
        cplex.setParam(IloCplex.Param.MIP.Strategy.File, 0);

        // add data
        String inDataFile = data;
        IloOplDataSource dataSource = oplF.createOplDataSource(inDataFile);
        model.addDataSource(dataSource);
        
        model.generate();

        cplex.solve();

How can I limit the memory used in the very beginning of the CPLEX solver log?

What exactly do you mean when you say "Java RAM usage above ..."? Is this only the memory allocated for the JVM? In that case you would have to talk to the JVM vendors. Or is the memory allocated by the whole process? In that case note that the JVM memory limit does not apply to allocations made by CPLEX native code. When solving a model with CPLEX, most of the allocations are made in native code. These allocations do not count towards a memory limit set via -X options for the JVM.

The memory limit specified by IloCplex.Param.MIP.Limits.TreeMemory only applies to the search tree. According to your log, CPLEX did not even start the search tree but is still processing the root node. There is no parameter to explicitly limit the amount of memory used while processing the root node. Things you can try is to disable optional tasks that create copies of the model. This can be done with parameter CPX_PARAM_AUXROOTTHREADS (set it to -1, for example).