My knowledge of Java memory management (and debuggers) is pretty limited. My impression is that manually invoking garbage collection (by calling System.gc(), or presumably by clicking on something in a debugger) is more of a suggestion than a hard and fast command to the JVM. I also don't know whether using ForkJoinPool has any special implications for heap management.
Correct me if I'm wrong, but my understanding is that you create multiple instances of your Model class. Each instance contains an instance of IloCplex. Are you implementing AutoCloseable in the Model class and ending the IloCplex instance (and by implication all variables, constraints etc. owned by that instance) in Model.close()? If so, do you manually invoke Model.close() in your code when you no longer need the Model instance?
Are there any static fields in the Model class? (Static objects stick around in memory.)
When you are done with a Model instance, consider what happens to the reference to it. If someplace you have a statement like
nextModel = new Model(...);
and you are done with the model, do you assign a new value to "nextModel" or assign NULL to "nextModel" (either of which would tell the JVM that the previous instance was no longer referenced and eligible for garbage collection)?
Good luck with this!
Paul
------------------------------
Paul Rubin
Professor Emeritus
Michigan State University
------------------------------
Original Message:
Sent: Thu April 13, 2023 04:29 AM
From: ISSA .
Subject: Best practice to solve a large number of CPLEX instances without exploding the Memory
Hello Paul,
Thank you for your reply.
Yes you are right I clear the mode before end it. True though calling clearModel() is unnecessary.
Yes I have tried to force garbage collection, but unfortunately not much space had freed up. Only about 1 GB as you can see in the pic where GC is performed inside the red circles.
Do you have any ideas how to look for possible leak? since I end the cplex instances after solving the models and they aren't used anywhere else.
Thank you I will look for that.
------------------------------
ISSA .
Original Message:
Sent: Wed April 12, 2023 04:09 PM
From: Paul Rubin
Subject: Best practice to solve a large number of CPLEX instances without exploding the Memory
I'm not sure that there is any point to calling IloCplex.clearModel() if you are calling IloCplex.end(). (In fact, I would expect calling clearModel() after end() to trigger an exception.)
In the memory profiler, when the heap is close to filling up, have you tried clicking the button to perform garbage collection? If that releases a significant amount of memory, then you might want to either have your code call System.gc() to collect garbage after every n models and/or look at some of the Java command line parameters for selecting and tuning the method of garbage collection. If forcing garbage collection does not free up much space, then you need to look for a possible memory leak (CPLEX objects that are not being destroyed as they should be).
------------------------------
Paul Rubin
Professor Emeritus
Michigan State University
Original Message:
Sent: Wed April 12, 2023 11:33 AM
From: ISSA .
Subject: Best practice to solve a large number of CPLEX instances without exploding the Memory
Hello everyone,
I have a MILP decomposable into a large number (can go up to 500k) of smaller MILP models that could be solved easily.
For computational experiments purposes I want to solve all of these smaller MILPs in the msot efficient way without exploding the memory.
I am using CPLEX Java API.
In Java I use the ForkjoinPool framework to parallelize the computation on two processors. Suppose the class to solve the MILP is called Model, I create a list of models that is equal to the number of problems I want to solve. For each model there's a CPLEX instance created and then ended and cleared using cplex.end() and cplex.clearModel().
However, this method didn't even pass with 120G of memory. So I used Java VisualVM to profile it.
First the picture memory_.png shows the memory after 49min of computation and solving around 20k instances (100k to go).
Top 5 objects that take around 75% of the Heap are ilog.cplex.CpxNumVar , byte[], and java.lang.String, ilog.cplex.CplexIndex[], and ilog.cplex.CpxExtractable[].
I am not sure whether this is a CPLEX or a Java question, but does anyone has an experience with solving a large number of instances in the most efficient way using the least possible amount of memory ?
Thank you.
------------------------------
ISSA .
------------------------------