Originally posted by: konrad86

Hello,

I am using concert to solve a maximization problem as a MIP with cplex 12.5. My program uses different types of usercut callbacks and I encounter the following behavior: When preprocessing is switched on (PreInd = 1) I obtain a worse (lower) solution than the known optimal solution for some settings. When I turn off preprocessing it works fine. My suspicion is there is some compatibility problem of usercut callbacks and preprocessing, e.g. some mapping of indices that doesn't work any more - are you aware of anything like that?

I am using a class derived from UserCutCallbackI and I add the cuts by the syntax add( IloConstraint ) to add the user cuts. The model is passed by reference to the callback and I use getValue(s) during the separation process.

I have already studied potential sources of error extensively:

• The errors occur for different instances and in Debug as well as Release mode. I also tried cplex 12.6 without any deviation in behavior.
• Their occurrence seems to be tied to one type of cuts. However, there will also be settings where the optimal solution is found using those cuts, so I am pretty sure there is no logical error.
• Whenever I do not use the cut callback the correct solution is returned (with and without preprocessing)
• The cuts are indeed user cuts, i.e. they do not cut off any feasible integer solutions. Indeed, I exported all the added cuts and manually verified they are fulfilled by the optimal solution. I also added them to the model manually and received the correct solution afterwards.
• I studied which preprocessing settings need to be switched if I want to keep PreInd = 1
  • If Reduce is set to 0, 1 or 2 the error didn't occur, however the speed was equal to PreInd = 0
  • If PrePass is set to 0 the error didn't occur, however the solution process was not accelerated either
  • All the other settings (AggInd, BndStrenInd, CoeRedInd, RelaxPreInd and RepeatPresolve) did not affect the behavior
• I exported the model before and after preprocessing and the .lp-files are identical
• I added the cuts to the model in the callback (by using model.add) and the status "Unknown" occured after a while of branching with the optimal solution already being found.

I am grateful for any hints on what might be going on here.

Thank you and kind regards

Konrad

I wonder if the one type of cut that triggers the problem either introduces numerical instability or is particularly vulnerable to existing numerical instability. Have you tried collecting kappa statistics (with and without the cuts)?

As another experiment, you might try printing out the cuts as your callback introduces them. After that, add them to the initial model (and turn off the callback, for that particular type of cut). Does that still encounter the same problem? If so, it would seem to be more related to the cuts than to the use of a callback. (Even if the problem goes away, there may still be an issue with the cuts, since the presolver behavior will change in their presence.)

Originally posted by: konrad86

Hi Paul

Thank you very much for taking some time for this issue and for your advice!

I had a deeper look into numerical instability, unfortunately without any positive outcomes:

• set the parameter Numerical Emphasis to 1 (and a few other parameters as suggested here: http://www-01.ibm.com/support/docview.wss?uid=swg27038712&aid=1)
• changed all floats into doubles
• included an epsilon in the user cuts I added
• rescaled some of my parameters to avoid multiplications of different orders of magnitudes
• performed the kappa tests. In a larger instance including the "suspicious" cuts changed the stable/suspicious percentages from 91%/9% to 82%/18% (no unstable or worse matrices)

Altogether, I think we can safely exclude numerical instability as a cause for the behavior.

I also exported the cuts again and added them to the model manually. Afterwards it could still provide the correct solution so it's sure they are actually user cuts.

I am still grateful for any further ideas on potential causes or debugging strategies. Meanwhile, I will have to switch off preprocessing or add a few simple redundant constraints that also seem to stop the anomalies consistently.

Best regards

Konrad

Konrad,

I once ran into a problem where CPLEX's preprocessor screwed up my model while trying to reduce symmetry (or what it thought was symmetry). I believe that turned out to be a bug fixed in a subsequent version, but I cannot remember which version was the culprit. You might try turning off symmetry breaking (Symmetry parameter) and see if that helps.

If possible, I would suggest trying a more recent version of CPLEX, in case you have bumped into a known (and fixed) bug.

One other experiment that comes to mind is to attach a LazyConstraintCallback to the model. The main method does not have to do anything; it's enough that CPLEX sees the presence of a LCC. If you get the correct answer, that would suggest that some of your cuts actually may cut off an integer feasible solution.

Paul

Originally posted by: konrad86

Paul,

thanks so much again for your valuable advice!!

Your hint around lazy constraints actually put me on the right track to solve the issue and everything makes sense now: Indeed, the alleged user cuts cut off some integer feasible solution. I had actually thought about those solutions before and decided not to worry about them as I could prove that for each of those there would always be a solution fulfilling all the "user cuts" with at least the same objective value. Still, they are of course no user cuts in the strict sense and therefore caused the issue in combination with the dual reductions while preprocessing.

Thank you again and all the best

Konrad