Originally posted by: larosma

I need to add and remove many columns repeatedly from a model. What is the best way to do this? I use Concert C++.
Here is what I do:
IloNumColumn col(env);
col += (...);

IloNumVar var(col);

// Disable "var"
var.end();

// Enable "var"
IloNumVar var2(col);
var = var2;

// Then I can use "var" again.
Mathieu
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

If you intend to add back the same variables you previously removed in some cases, I would recommend looking at IloModel::add and IloModel::remove.

Paul

Mathematicians are like Frenchmen: whenever you say something to them, they translate it into their own language, and at once it is something entirely different. (Goethe)
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: larosma

IloModel::add(var) "adds" the variable to the model, but not the constraints. IloModel::remove(var) does the opposite. In my opinion, both are useless when they are called with IloNumVar.

I found IloNumVar::removeFromAll which is similar to IloNumVar::end, except that the handle is still valid after the call. But I didn't find how to add the variable back to the model/constraints.

Right now I'm trying to set the lower and upper bound to 0 to disable a variable.
Mathieu
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

> larosma wrote:
> IloModel::add(var) "adds" the variable to the model, but not the constraints. IloModel::remove(var) does the opposite. In my opinion, both are useless when they are called with IloNumVar.

Yes. I thought perhaps there was an exception for a variable that was generated by a column, but no such luck -- nor is the column itself considered "extractable", apparently.
>
> I found IloNumVar::removeFromAll which is similar to IloNumVar::end, except that the handle is still valid after the call.

Interesting -- doesn't seem to be documented.

> But I didn't find how to add the variable back to the model/constraints.
>
> Right now I'm trying to set the lower and upper bound to 0 to disable a variable.

Actually, that's what I normally do to hide a variable.

Paul

Mathematicians are like Frenchmen: whenever you say something to them, they translate it into their own language, and at once it is something entirely different. (Goethe)
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: revolt

I'am using Java-Concert and I want to remove a variable/column from a quadratic problem. Is there a way to remove the variable and the associated column without using the IloCplex.end(IloAddable arg0) function? I tried the following, but the two generated files are identical.

IloCplex cp = new IloCplex();
IloNumVar x1 = cp.numVar(0, 1);
IloNumVar x2 = cp.numVar(0, 1);
cp.addLe( cp.sum(x1, x2), .5 );
cp.addMaximize( cp.prod(x1, x1) );
cp.exportModel("lp1.lp");
cp.remove( x2 );
cp.exportModel("lp2.lp");
cp.end();

PS:
Unfortunately the Java-Doc annotation for the function IloCplex.end(IloAddable arg0) is missing in Cplex 12.1 Java-Concert.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

You could do

x2.setLB(0);
x2.setUB(0);

This fixes x2 to 0, effectively removing it from the model (the presolver will remove the variable before the solve.
Alternatively, you could set the coefficient of the variable to 0 in each constraint:

for (java.util.Iterator it = cp.rangeIterator(); it.hasNext(); /* nothing */) {
   cp.setLinearCoef((IloRange)it.next(), 0.0, x2);
}

This is a little dangerous as depending on how you build your model there may be other objects in your model, for example instances IloLPMatrix. You would have to use IloCplex.iterator() to iterate over all objects in the model and set the coefficients to 0 in all of them.

Why don't you want to end() the variable? Is it shared between multiple models?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: revolt

Thanks for the quick answer!
Actually I was just wondering why the Cplex.remove(var) function has no effect.

The end() function works fine, but in my case it is much slower than simply fixing the variable to 0. Unfortunately this generates a degenerated problem.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: revolt

PS: I can not simply fix the variable to 0 because then my model becomes degenerated. That's why I want to remove/delete it from the model. Deleting the variable works well. I'm just wondering whether there exists a function to remove the variable from the model without destroying the reference to it.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

It is a little more cumbersome but you could create your model using an instance of IloLPMatrix. This interface offers a function to remove a variable from all rows in one shot.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Still, the failure of remove() remove the variable is, at minimum, contrary to the documentation (which says that IloNumVar is a subinterface of IloAddable and remove() removes IloAddables). Not to mention it's rather counterintuitive.

Paul
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Yes, it is at least surprising. Even more so as you cannot remove() a variable from a constraint. This part of Concert predates me so I am not yet sure what is going on. That removing a variable from a model does not remove it recursively from all constraints in the model kind of makes sense because it would be very slow. But as you said, it is counterintuitive. I will check and maybe we can improve our code here ...
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: revolt

It seems like there is a small bug in the end function of the java api:

IloNumVar x1 = cp.numVar(0, 1, "x1");
IloNumVar x2 = cp.numVar(0, 1, "x2");
cp.addLe( cp.sum(x1, x2), .5 );
cp.addMaximize( cp.sum( cp.prod(x1, x1), cp.prod(x2, x2)) );
System.out.println( cp.getObjective() );
cp.end( x2 );
System.out.println( cp.getObjective() );

this code returns:

IloMaximize  :  +1.0*x1*x1 +1.0*x2*x2
IloMaximize  :  +1.0*x1*x1 +1.0*[0.0..1.0]*[0.0..1.0]

the second term of the second objective is confusing me.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

This indeed looks like a small bug, thanks for catching that!

As far as I can see the problem is limited to printing out the objective. When I export the model to a file then the variable does not appear in the exported model. Similarly, when I solve the problem I don't see any artifacts of x2 in the solution process.

Did you observe any other problems with the deleted variable?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: revolt

I did not find other problems with the end function.
The only "problem" that arises is, that I found no possibility to retrieve the current objective function directly from cplex. I also stored the objective in an (quadratic) IloNumExpr, but I found no way to remove the ended variable from my stored objective, so that my stored objective gets useless. But actually this is not a real problem, as I do not really need the IloNumExpr of the objective.

Unfortunately I found another problem related to this topic:

I solve an QP (maximize a quadratic objective, s.t. linear constraints).
Some 0,1-variables are fractional, so I fix them to 0 and start again. (the feasible region gets smaller)
In some cases the *objective will increase*! I think it has something to do with degeneracy.
To fix this issue, I simply delete the previously fixed variables as soon as I think, that the result returned by cplex is unusable. After ending the variables everything gets back to normality. I get the right objective function and the right variable values.

Unfortunately the solving process is a lot slower when I delete the variable instead of fixing it to zero. That's why I would prefer if it would be possible only to fix the variables to 0 instead of deleting them completely.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Thanks a lot for your feedback. Let me try to answer your questions:

• In order to get the current objective function either use IloCplex.getObjective() or iterate over the model using IloCplex.iterator() and look out for an instance of IloObjective.
• You are right, there the API is lacking functionality to iterate over expressions that are not linear. That is why I usually do not store non-linear expressions (like quadratic objective functions) in instances of IloNumExpr. Instead I define my own class that can store such expressions in a compact and easy-to-access format and can generate instances of IloNumExpr on demand. This allows me to efficiently manipulate the expression and convert it to an IloNumExpr as soon as I want to submit it to IloCplex.
• In fact, you can iterate over a non-linear expression. However, this requires use of some undocumented features/functions/classes. You could figure this out all by yourself using reflection. It is dirty. If you are really interested in doing that and don't see a better way then I can give you some hints. Just drop me an email at daniel(dot)junglas(at)de(dot)ibm(dot)com.

Concerning the other problem you mentioned:

• How do you fix the variables? Do you do this via IloNumVar.setBounds()? Or do you do it by adding a constraint that fixes the variable? If you tried only one of these approaches, can you try the other one and see if it helps?
• How big is the increase in the objective function value?
• Do you end() multiple variables? If so then it might be faster to use IloCplex.end(IloAddable[]) and end() all of them in one shot instead of calling IloCplex.end(IloAddable) for each variable individually.
• Is it the call to cplex.solve() that is slower with the deleted variables or is it just the time required to delete the variables that bothers you?
• What you describe sounds like a bug. I would like to investigate that. Could so send me a model M together with a list of variables V so that after fixing the variables in V CPLEX returns an objective function value for the modified model that exceeds the objective function value for M?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: revolt

Thanks for your quick answers.

• I'm also using an own class for quadratic expressions. This seems to be the best solution.
• Undocumented features and reflection sounds quite interesting, but I think I prefer the own quad expression.

About the second problem:

• I fixed the variables by using IloNumVar.setUb(0). (The lower bound is already 0 for all vars.) I will later try the variant of adding an additional constraint.
• First objective 8.67211889492087E9, second: 8.680272066068947E9. The second is obviously a lot to big. I'm using RelativeGap = 1e-12; AbsoluteGap = 1e-5;
• Yes I end() multiple variables. I should do it by using IloCplex.end(IloAddable[]), this would be better.
• actually its the call to cplex.solve() that is slower than in the setUb() case. cplex restarts its presolving if I end a variable. if I use setUb(0) cplex makes a warm start.
• I will try to send you a model, but its a bit difficult. As soon as I create the LP file the model gets perturbed slightly, so that the degeneracy gets destroyed. As I can not send you the whole application i'll try to write a small app that generates such a model.

PS: it would be great if IloCplex would be serializable, but unfortunately the IloCopyManager is not serializable so that I can not serialize IloCplex...
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Two quick hints:

• Try to export the model as .sav file. This will avoid as many perturbation in the model as possible.
• IloCplex can not be serialized. But you can serialize a model. Look at the CplexServer.java example in the CPLEX distribution. It illustrates how to serialize a model.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: revolt

Thanks for the hints. With the .sav file I was able to extract an instance where this problem occurs. I sent you an email with the model and a the list of variables and a java-class.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: revolt

Now I replaced var.setUb(0) by cp.addLe(var, 0). This seems to resolve the issue in all the instances that I checked so far!
#CPLEXOptimizers
#DecisionOptimization