Originally posted by: open_ball

Hi,

I am trying to create a model in Java API, but facing an out of memory issue. After my trials, I get two different errors. The first one was:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

The second was: 

java.lang.OutOfMemoryError: GC overhead limit exceeded

I am sharing a minimal example where I face with the issue. I tried to add all the possible information in the code block which only contains the creation of two variable types and a simple constraint.

IloNumVar[][][] x = new IloNumVar[numNodes][numNodes][time+1]; 
IloIntVar[][][] y = new IloIntVar[numNodes][numNodes][time+1];

IloLinearNumExpr expr = cplex.linearNumExpr();
IloLinearNumExpr myExpr = cplex.linearNumExpr();

for (int i = 0; i < numNodes; i++) {           // numNodes =1800                                 
    for(int j =1 ; j <= time ; j++ ) { // time =50
      int from = i; 
      int t = j;                                    
       if(adjencyList.containsKey(i)) { 
        //adjencyList => TreeMap<Integer,TreeSet<Integer>>
    
        adjencyList.get(i).forEach((to) -> { 
         try {            
          Arc instance = new Arc(from, to);  //get the arc                      
          x[from][to][t] = cplex.numVar(0.0, UpperBound.get(instance)); 
          //upperBound => TreeMap<Arc,Integer>           
            
          cplex.add(x[from][to][t]); 
            if(special.get(instance)){
             //special => TreeMap<Arc,Integer>
              y[from][to][t] = cplex.boolVar("y_"+  from+"."+ to+"."+ t );                    
              cplex.add(y[from][to][t]);
              boolean flag = false;
                if(t <= initialTime.get(instance)-1){ 
                    // initialTime => TreeMap<Arc,Integer>
                        myExpr.addTerm(y[from][to][t], 1);
                        flag =true;             
                 }                                                                                               
                    expr.addTerm(Capacity.get(instance), y[from][to][t]);
                     // Capacity => TreeMap<Arc,Integer>
                    cplex.addLe(x[from][to][t],expr);
                    expr.clear();
              }       
           }catch (IloException e) {
             e.printStackTrace();
          } 
        });
      }
    }     
}
cplex.addEq(myExpr, 0);
cplex.addMaximize(objExpr);
expr.clear();
myExpr.clear();

I was wondering if someone could help me out and give me an advice on how to solve this issue. As CPLEX settings, I have added the following changes.

        cplex.setParam( IloCplex.Param.MIP.Strategy.File, 3);
        cplex.setParam( IloCplex.Param.WorkMem, 15000);
        cplex.setParam(IloCplex.Param.MIP.Limits.TreeMemory, 15000);

As I read this, you are allocating space for over 165 million x variables and over 165 million y variables. It appears from the code that not all possible arcs exist (hopefully the network is sparse) and that possibly that the arcs that exist do not have x and/or y variables at all possible time instants. So you might consider doing the following:

1. create a class (call it AT for now) that encapsulates a combination of an Arc and a time (int) for which x and/or y will exist;
2. create a TreeMap<AT, IloNumVar> and a TreeMap<IloNumVar, AT> for x, and populate them with all relevant arc/time combinations (AT instances) and corresponding variables;
3. do the same for y with two more maps.

That lets you allocate memory only for the things you actually need. Whether it will save enough memory I cannot say, but presumably you can calculate how many ATs and IloNumVars you will end up with before going down this road.

Originally posted by: open_ball

Thank your very much for your answer. I just want to make sure that I understood you correctly. I am sharing the steps that I use to define only variable x. Would you mind letting me know if I am on the right path?

TreeMap<ArcTime,IloNumVar> x = new TreeMap<ArcTime,IloNumVar>();
    
for (int i = 0; i < numNodes; i++) {                                                               
    for(int j =1 ; j <= time ; j++ ) {
        from = i; int t = j;                                                    
        if(outGoingArcs.containsKey(i)) {
                       outGoingArcs.get(i).forEach((to) -> {                                       
                           ArcTime instance = new ArcTime(from,  to, t);                             
                           Arc arcInstance= new Arc(from,to);
            try {
                          IloNumVar varHolder= cplex.numVar(0,capacity.get(arcInstance),"x_"+instance.toString());                            cplex.add(varHolder);
                          x.put(instance, varHolder);                                                                     
            } catch (IloException e) {
                        e.printStackTrace();
                });
        }
    }         
}

For the TreeMap<IloNumVar, AT> part, I am currently receiving a casting error. I believe that I need to implement comparable class for IloNumVar. Maybe I should switch to HashMap. Do you think that my implementation gets considerably slower if I use HashMap instead of TreeMap?

The approach looks correct. I think HashMap would probably be fast enough for the reverse mapping. In fact, it might actually be faster if the hashing works properly (constant time, v. logarithmic time for TreeMap).

In case you still get out of memory issues, you can also crank up the memory available to the JVM. Look at the various -X options for the Java virtual machine.

Originally posted by: open_ball

@DanielJunglas thanks for the suggestion. Yet, Paul's comment actually helped me a lot and solved my issue. After the update, I realized how bad I was allocating the memory.