Originally posted by: VKV7_Anulark_Naber

I now applied an IncumbentCallback to use the information (objective function value) of a new incumbent to update the parameters that will be used in UserCutCallback, so the cuts will be tighter. In the documentation, it says "The user-written callback will be called each time a new incumbent solution has been found, including when solutions are provided by the user's heuristic callback routine." However, it does not seem that the incumbent callback is called everytime a new solution is found. As you can see in the log output of running 1 thread shown below, there is some delay until it is invoked. How can I force cplex to invoke incumbentcallback every time, or shouls I use any other callback to do the job? This model does have lazy constraints (in a table), and the new incumbent should satisfy the lazy constraints.

I also need to mention that when I interrupted solve() when a new incumbent is found, the incumbent callback was not invoked when the last solution is found either. I do not prefer this interruption, because when restart, cplex thows out all cuts, and therefore I changed to implement the above incumbentcallback.

#CPLEXOptimizers
#DecisionOptimization

This looks very odd. The incumbent callback is invoked before a new incumbent is printed to the node log (the callback may reject the incumbent in which case nothing will be printed to the log). Could this be an issue with output buffering? How are you printing the lines from your callback? Are you printing to cplex.out() or to some other stream?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: VKV7_Anulark_Naber

These lines --------------Update... are printed in IncumbentCallback to std::ofstream along with the node log. 

These are the settings for cplex output:

I am using cplex 12.6.0.1.

#CPLEXOptimizers
#DecisionOptimization

I still feel like something must be wrong on your side. I just tried this simple code:

#include <ilcplex/ilocplex.h>

ILOINCUMBENTCALLBACK1(inccb, std::ostream &, output) {
   output << "#### Incumbent callback: " << getObjValue() << std::endl;
}

int
main(int argc, char **argv)
{
   std::ostream &out = std::cout;

   for (int i = 1; i < argc; ++i) {
      IloEnv env;
      IloModel model(env);
      IloCplex cplex(model);
      cplex.setOut(out);
      cplex.setWarning(out);
      cplex.use(inccb(env, out));
      cplex.importModel(model, argv[i]);
      cplex.solve();
      env.end();
   }
   return 0;
}

It produces the expected output:

...
      0     0     1098.3075    55                    Cuts: 16      944         
#### Incumbent callback: 1435
*     0+    0                         1435.0000     1098.3075      944   23.46%
#### Incumbent callback: 1423
*     0+    0                         1423.0000     1098.3075      944   22.82%
#### Incumbent callback: 1169
*     0+    0                         1169.0000     1098.3075      944    6.05%
#### Incumbent callback: 1168
*     0+    0                         1168.0000     1098.3075      944    5.97%
      0     2     1098.3075    55     1168.0000     1098.4376      944    5.96%                        0             0
Elapsed time = 1.30 sec. (1063.51 ticks, tree = 0.00 MB, solutions = 4)
     25    23     1143.0156    22     1168.0000     1103.9525     1998    5.48%            x365 D     25     24     20
     70    50     1125.7917    35     1168.0000     1107.3700     4139    5.19%            x268 D     70     69      7
    178   109     1146.9736    29     1168.0000     1109.6815     6793    4.99%            x153 D    178    177     15
    335   224     1154.3310    31     1168.0000     1111.3530    10452    4.85%            x270 U    335    334     25
    501   336     1148.0454    27     1168.0000     1113.6307    14164    4.65%            x199 U    501     40      6
    637   428     1160.4792    17     1168.0000     1116.1585    18197    4.44%            x419 U    637    635     39
    782   518     1155.0654    24     1168.0000     1119.3741    22236    4.16%            x397 U    782    781     11
    928   606     1157.8902    23     1168.0000     1121.0777    26517    4.02%            x147 U    928    926     12
   1092   711     1153.1257    17     1168.0000     1122.8651    30435    3.86%             x88 D   1092   1091     64
#### Incumbent callback: 1162
*  1450   793      integral     0     1162.0000     1126.1360    39911    3.09%            x352 D   1450   1449     41
Elapsed time = 3.54 sec. (3792.65 ticks, tree = 0.53 MB, solutions = 5)
#### Incumbent callback: 1160
*  1550+  783                         1160.0000     1126.9947    43092    2.85%
   1966   932     1155.7112    30     1160.0000     1132.0412    57356    2.41%              x3 D   1966   1965     29
   2468  1095     1146.4107    30     1160.0000     1136.8281    74076    2.00%            x298 D   2468   2467     32
   3062  1069        cutoff           1160.0000     1142.8966    90662    1.47%            x411 N   3062   1610     22
   3760   858     1155.8244     8     1160.0000     1149.2136   107294    0.93%            x351 D   3760   3759     36
#### Incumbent callback: 1159
*  3830+  759                         1159.0000     1149.8031   108633    0.79%
#### Incumbent callback: 1158
*  4347   275      integral     0     1158.0000     1154.0122   115604    0.34%            x108 D   4347   3018     33
...

As you can see, the callback is always invoked before anything is printed to the node log. Could you try my simple code on your side?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: VKV7_Anulark_Naber

Thank you so much, Daniel. 

The problem is that I use getIncumbentObjValue() instead of getObjValue(). Now it's working as we expected.

I misunderstood what it says in the documentation:

getIncumbentObjValue

 

#CPLEXOptimizers
#DecisionOptimization

Ah, this is a little subtle in the incumbent callback, I agree. Since the current incumbent has not yet been accepted (the incumbent callback could still reject it), getIncumbentObjValue() will return the objective value for the last incumbent that was accepted and not the incumbent for which the callback is invoked.

The interesting thing is that you did not get an exception for the very first invocation of the incumbent callback. In that call there is no previous incumbent and getIncumbentObjValue() should throw an exception. Did you just ignore that exception or did you really not get any?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: VKV7_Anulark_Naber

No, I did not get any exceptions from getIncumbentObjValue().

What are the reasons why incumbent callback would reject the current incumbent when it is better anyway? Why not just accept all better ones within tolerances?

 

#CPLEXOptimizers
#DecisionOptimization

 

The user may want reject a proposed incumbent without generating a cut. For instance, one way to collect a bunch of "good" (but not necessarily optimal) solutions is to store proposed incumbent and then reject them (so that they don't move the bound and cut off other good solutions). The solution pool feature might be a better way to do this, but it's possible a user would have criteria for letting the bound change that would be too funky to accomplish with the solution pool.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: VKV7_Anulark_Naber

Do you mean that "other good solutions" are those with the same obj value? In my case, I want any optimal solution, because my problem is there are too many solution and therefore slow convergnce to optimality. What I gathered from the difference or delay between getIncumbentObjValue() and getObjValue() is that even if a new solution is found, cplex does not automatically update the bound immediately and so some nodes which could have been pruned with the updated bound are not pruned. That answers the situation i got, when I terminate solve() each time a better solution is found in a solution pool , update bounds manually, and then solve() again., I got a better performance than waiting for IncumbentCallback to update an incumbent for me. The disadvantage of this manual approach is that cplex throws all cuts away and restarts from scratch.

How can I force IncumbentCallback to do otherwise (as manually)? I would prefer that cplex accepts each BETTER solution and updates the bound immediately.

#CPLEXOptimizers
#DecisionOptimization

 

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: VKV7_Anulark_Naber

Thank you. Only one point:

Do you mean that the incumbent callback is invoked before checking the lazy constraints? Does that also mean that getObjValue() is not really of an incument because it may not pass the lazy constraints? 

As shown earlier in my log output (in the first post), getIncumbentObjValue() is not updated until later, shown with ---update, but getObjValue() is updated immediately as later corrected.

#CPLEXOptimizers
#DecisionOptimization

I think the lazy constraint callback is invoked before the incumbent callback (but I might be mistaken). Regardless of the order, getObjective() in both callbacks should give the objective value of the proposed incumbent. If whichever callback is first called rejects the proposed incumbent, the second callback will not be called (for that solution -- it could be called at the same node for a different solution).

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: VKV7_Anulark_Naber

Thank you.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: VKV7_Anulark_Naber

I have come across for some instances that the incumbentcallback is not invoked for every incument found. For example, this is the first incumbent found and the log file looks like:

Then, the second incumbent is found and the log file looks like:

 --------Internally Update Parameters with Makespan 32.2571

The comment line also shows that the incumbent callback is invoked.

If the incumbent did not pass the lazyconstraints, the incumbent was then rejected, then the log file should not report the "Found incumbent.."

What can be the reason why it is not doing so for the first incumbent? Thanks for any help.

#CPLEXOptimizers
#DecisionOptimization

That should not happen and smells like a bug :-(

Do you have an easy way or minimal code that you could provide us so that we can try to figure out what is going wrong?

#CPLEXOptimizers
#DecisionOptimization