Originally posted by: Bahman

Hello everybody,

 

On the course of running my model, various errors like C0000005 EXCEPTION_ACCESS_VIOLATION and UNEXPECTED CPLEX ERROR 5002: Q IS NOT POSITIVE SEMI-DEFINITE popped up, two of which are still remaining:

 

1) cannot extract expression Ct ... &

2) not enough memory

 (Image attached)

I have a binary decision variable: X[<i,j,tc>][k][t]  and an integer variable: d[k][g] whose lower-bound is 0 and its upper-bound is float D[k][g].

 

In order to linearize my problem to avoid the error 5002, I defined an expression: dexpr float Z[<i,j,tc> in Arcs][k in K][t in T][g in G]=X[<i,j,tc>][k][t]*d[k][g]; Then added two more constraints as follows:

 forall (k in K,g in G)
      CtBoundsOnZ1:
        d[k][g]<=D[k][g];
    
    forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        CtBoundsOnZ4:
            Z[<i,j,tc>][k][t][g]>=d[k][g]-D[k][g]*(1-X[<i,j,tc>][k][t]);

 

But, after running the model, the following error popped up:

What I did was to do some changes in .ops, configurations settings, file:

 

1) increased the memory: (image attached)

2) switching the memory emphasis to True: (image attached)

 

Yet, I have the same problem! 

 

My system info is attached.

 

I would really appreciate if you can help me tackle the issues!

 

Regards,
Bahman 

#CPLEXOptimizers
#DecisionOptimization

Are the errors you report all for the final version of your model? I am asking because I tried to reproduce your issues with this very tiny .mod which I think is the same as yours:

tuple Tuple {
  int i;
  int j;
  int tc;
};
{Tuple} Arcs = { <1,1,1> };
range K = 1..2;
range T = 1..3;
range G = 1..4;

dvar boolean X[Arcs][K][T];
dvar int d[K][G];
dexpr float Z[<i,j,tc> in Arcs][k in K][t in T][g in G]=X[<i,j,tc>][k][t]*d[k][g];

float D[K][G] = [ [ 5, 6, 7, 8 ], [ 9, 10, 11, 12 ] ];

minimize 0;
subject to {
  forall (k in K,g in G)
    CtBoundsOnZ1:
      d[k][g]<=D[k][g];
    
  forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
    CtBoundsOnZ4:
      Z[<i,j,tc>][k][t][g]>=d[k][g]-D[k][g]*(1-X[<i,j,tc>][k][t]);
}

For this, I immediately get the error "5002: Q is not positive semi-definite". Hence the engine would not even start to solve. So I wonder where this out-of-memory errors comes from. If it happens because the model just does not fit into memory there is nothing you can do but shrink the model size.

Could you do the following

1. In the settings file set Language->Run->Export format to "SAV"
2. Solve the model again. This will create a *.sav file on your disk (in your project directory or in the directory from which you launched the IDE)
3. Try to solve the SAV file with the interactive optimizer or post it here (after compressing it).

If you get the out-of-memory error before the SAV file is created then the model does not fit into memory. How big are the sets involved in the constraints you mentioned?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Bahman

Hi DanielJunglas,

I appreciate your prompt reply! 

You are right! For a smaller model where K=100,  N=25, T=20, I got the following error: 5002: Q is not positive semi-definite

But for a problem size of: K=700, N=25, T=31, the following errors appeared: Exception from IBM ILOG Concert: not enough memory

 

Then, I increased the memory within oplide.ini file as follows:

--launcher.XXMaxPermSize
1024m
-clean
-vmargs
-Xmx2000m
-Xms80m

Then ran the model again, but it failed to resolve the memory problem.

To be honest, I have followed various links on this forum to linearize the product of my binary and integer variables, but apparently 5002: Q is not positive semi-definite pops up again!

 

Regards

Bahman

#CPLEXOptimizers
#DecisionOptimization

Hi,

to get rid of error 5002, have you read https://twitter.com/AlexFleischer1/status/843806521918132225

?

regards

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Bahman

Hi Alex,

I did as you recommended on https://www.ibm.com/developerworks/community/forums/html/threadTopic?id=aa9aa3db-4fbc-4209-a767-5b5e54902cbd&ps=25

to linearized the formulation. 

    forall (k in K,g in G)
      CtBoundsOnd:
        d[k][g]<=D[k][g];
        
    forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        CtBoundsOnZ1:
            Z[<i,j,tc>][k][t][g]<=0*X[<i,j,tc>][k][t];    
   
   forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        CtBoundsOnZ2:
            Z[<i,j,tc>][k][t][g]<=D[k][g]*X[<i,j,tc>][k][t];
        
    forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        CtBoundsOnZ3:
            Z[<i,j,tc>][k][t][g]<=d[k][g]-0*(1-X[<i,j,tc>][k][t]);
    
    forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        CtBoundsOnZ4:
            Z[<i,j,tc>][k][t][g]>=d[k][g]-D[k][g]*(1-X[<i,j,tc>][k][t]);  
}

with that difference that the lower-bound and upper-bound of my integer variable d[K]{G] are zero and D[K][G], respectively, so I could not declare dvar int+ d[K][G] in 0...D[K][G], like your example.

The following error pops up: c0000005 EXCEPTION_ACCESS_VIOLATION

 

I would really appreciate if you advise me how to resolve the problem.

Regards,
Bahman

#CPLEXOptimizers
#DecisionOptimization

Hi,

looks like a label issue:

tuple Tuple {
  int i;
  int j;
  int tc;
};
{Tuple} Arcs = { <1,1,1> };
range K = 1..2;
range T = 1..3;
range G = 1..4;

dvar boolean X[Arcs][K][T];
dvar int d[K][G];
dvar float Z[<i,j,tc> in Arcs][k in K][t in T][g in G]; //=X[<i,j,tc>][k][t]*d[k][g];

float D[K][G] = [ [ 5, 6, 7, 8 ], [ 9, 10, 11, 12 ] ];

minimize 0;
subject to {
 forall (k in K,g in G)
      CtBoundsOnd:
        d[k][g]<=D[k][g];
        
    forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        CtBoundsOnZ1:
            Z[<i,j,tc>][k][t][g]<=0*X[<i,j,tc>][k][t];    
   
   forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        CtBoundsOnZ2:
            Z[<i,j,tc>][k][t][g]<=D[k][g]*X[<i,j,tc>][k][t];
        
    forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        //CtBoundsOnZ3:
            Z[<i,j,tc>][k][t][g]<=d[k][g]-0*(1-X[<i,j,tc>][k][t]);
    
    forall (<i,j,tc> in Arcs,k in K,t in T,g in G)  
        CtBoundsOnZ4:
            Z[<i,j,tc>][k][t][g]>=d[k][g]-D[k][g]*(1-X[<i,j,tc>][k][t]);  
}

works fine

regards

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Bahman

Hi Alex,

 

Thanks for your prompt reply!

I just copied and pasted the codes on your previous post, and the same error appears: c0000005 EXCEPTION_ACCESS_VIOLATION

 

 

I went through logical constraints of CPLEX as follows:

 

(b==1) => (bx==x);
(b==0) => (bx==0)

 

and the  c0000005 EXCEPTION_ACCESS_VIOLATION error has been resolved. Yet, I have the problem:CPLEX Error 1001: out of memory!

 

 

Regards,
Bahman
 

#CPLEXOptimizers
#DecisionOptimization

OK, so you have to problems to solve:

1. Fix the "Q not positive semi-definite" error.
2. Solve the issue with the problem size

To solve the "Q not positive semi-definite" error you could try to reformulate constraint CtBoundsOnZ4 using logical constraints. Given that X can only take two values, you can distinguish the two cases X=0 and X=1 and rewrite the constraint as two implication constraints:

  forall (<i,j,tc> in Arcs,k in K,t in T,g in G) {  
    //CtBoundsOnZ4:
    //  Z[<i,j,tc>][k][t][g]>=d[k][g]-D[k][g]*(1-X[<i,j,tc>][k][t]);
    CtBoundsOnZ4a:
      X[<i,j,tc>][k][t] == 0 => 0>=d[k][g]-D[k][g];
    CtBoundsOnZ4b:
      X[<i,j,tc>][k][t] == 1 => d[k][g]>=d[k][g];
    }

I am not sure whether this is enough or you get the "Q not positive-semidefinite" error then in other places as well. The b case clearly looks redundant, so your constraints actually means "if X=0 then there is an upper bound on d, otherwise d is not bounded from above". Is that correct?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Bahman

Hi DanielJunglas,

I appreciate your prompt reply! 

Actually, the logical constraints could resolve the non positive semi-definite Error. However, even for smaller size of the problem where K=30, N=25, T=20, this error: CPLEX Error 1001: out of memory pops up and terminates the run!

I increased the memory within oplide.ini file as follows:

--launcher.XXMaxPermSize
1024m
-clean
-vmargs
-Xmx2000m
-Xms80m

As well, in run settings, I increased the "Memory available for working storage" to 20480 (it's default value was 2018) and checked the "Memory reduction switch" to True. 

Yet, the problem exists!

( I don't know if it helps that I say that for a model like this which was a Mixed Integer Problem and I had to deal with X[A]{K][T] and a N[T], CPLEX solved a problem with N=25, K=700, T=31 within 9 hours without error on memory.)

 

I wonder if there exists some other way to increase the memory or any other resolution for the error on memory.

Regards
Bahman

#CPLEXOptimizers
#DecisionOptimization

Does out of memory occur before the solve actually starts (do you see anything in the "Engine log" tab)?

If the engine starts to solve, can you show the engine log that leads up to the out of memory?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Bahman

Hi DanielJunglas,

I appreciate your prompt reply! 

It occurs before the solve actually starts and nothing appears in Engine Log. 

My windows is 64-bit version, but as I see the CPLEX files are in Program file 86x !

 

Could it be the source of the problem? I mean I think the CPLEX running on my windows may be 32-bit version, so that I should migrate to a 64-bit version of CPLEX?!

Regards
Bahman Borna

#CPLEXOptimizers
#DecisionOptimization

Using 64bit CPLEX on 64bit Windows is actually a good idea. The amount of memory a 32bit application can use is limited.

Another thing you could try is to run your via oplrun. This is a command line tool and thus does not need all the memory required for the GUI.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Bahman

Hi DanielJunglas,

I have installed 64-bit CPLEX and and began running after:

1) increasing the memory: (image attached)

2) switching the memory emphasis to True: (image attached)

3) Modified / increased memory in oplide.ini file

--launcher.XXMaxPermSize
1024m
-clean
-vmargs
-Xmx2048m
-Xms80m

 

This time, at least the model ran for 6:11:46:77. I was watching the engine log. Eventually, memory error popped up and stopped the run unfortunately! 

Please find attached the images of the engine log and statistics if it might help.

I am stuck at this point!

I would really appreciate your suggestion or workarounds!

Regards
Bahman

 

#CPLEXOptimizers
#DecisionOptimization

OK, now we are getting somewhere. You are running out of memory since the search tree grows too large. CPLEX has an option to swap out parts of the tree to disk. This is described here. You set WorkMem to something smaller than your available RAM (80-90% or so) and NodeFileInd to 2 or 3.

Additionally, from your log it seems CPLEX has a hard time finding a feasible solution for your model. If you know your model is feasible and you know a feasible solution (no matter how bad it is), you can provide that solution as a warm start, see opl/examples/opl/warmstart/warmstart.mod in the distribution for an example. You may also want to play with the MIP emphasis switch.

#CPLEXOptimizers
#DecisionOptimization