Originally posted by: Falcon_G

Dear all

I have done this before in earlier versions without any problem but with 12.5 seems to have an issue.

I have several different callbacks including a UserCutcallback where I generate cuts.

To make use of threads, I declare an instance of IloFastMutex and pass it to every callback.

Of course I lock it when I enter and unlock it when I return.

Every thing is fine, there is no deadlock, I think.

Visual studio 10 C++ compiler reports like:

{code}

The thread 'Win32 Thread' (0xea88) has exited with code 0 (0x0).
The thread 'CLazyConstraintCallback::main' (0xc1cc) has exited with code 0 (0x0).
The thread 'CUserCutCallback::main' (0xc0c4) has exited with code 0 (0x0).
First-chance exception at 0x77d745ce in mphlppt-benders-bc-multi-thread.exe: 0xC0000005: Access violation reading location 0xfeeefeee.
Unhandled exception at 0x77d615de in mphlppt-benders-bc-multi-thread.exe: 0xC0000005: Access violation reading location 0xfeeefeee.
First-chance exception at 0x77d5016e in mphlppt-benders-bc-multi-thread.exe: 0x00000000: The operation completed successfully.
Unhandled exception at 0x77d615de in mphlppt-benders-bc-multi-thread.exe: 0x00000000: The operation completed successfully.
The program '[49468] mphlppt-benders-bc-multi-thread.exe: Native' has exited with code 0 (0x0).

{code}

but  in the end the just before end I am re-directed to this function with an error message:

 

{code}

IloBool solve() { return IloAlgorithm::solve(); }

{code}

 

Apparently something went wrong which is not clear what it was and there is no message clarifying it.

The few last lines of cplex report are the following:

 

{code}

     29     2   132691.7570    25   139302.9383   131215.3395    23116    5.81%      y[1][0][9] N     65     33      5
*    30     1      integral     0   136576.4953   131215.3395    23397    3.93%      y[1][1][9] U     69     65      6

GUB cover cuts applied:  0
Clique cuts applied:  0
Cover cuts applied:  5
Implied bound cuts applied:  7
Flow cuts applied:  0
Mixed integer rounding cuts applied:  9
Flow path cuts applied:  0
Zero-half cuts applied:  0
Gomory fractional cuts applied:  0
Disjunctive cuts applied:  0
User cuts applied:  1089

Root node processing (before b&c):
  Real time             =   58.09 sec. (34873.18 ticks)
Parallel b&c, 4 threads:
  Real time             =   22.93 sec. (8373.74 ticks)
  Sync time (average)   =    0.00 sec.
  Wait time (average)   =    0.00 sec.
                          ------------
Total (root+branch&cut) =   81.03 sec. (43246.92 ticks)

{code}

I appreciate constructive comments.

 

 

#CPLEXOptimizers
#DecisionOptimization

The error message looks like someone is reading memory through an invalid pointer.

I take it, when you say you pass the IloFastMutex instance to the callbacks then you pass it by reference/pointer and not by value? Do things work better if you make this IloFastMutex instance just a global variable?

Is it possible that in one of your callbacks you get an exception and due to the exception the callback is exited without unlocking the mutex?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Falcon_G

Many thanks Daniel for your reply.T

I remember once upon a time (ages back) one of your colleagues suggested to use this global instance of IloFastMutex to enable making use of more threads without any problem when playing with callbacks. In older codes with the earlier versions it was working fine.

 

I had this habit until now,.

 

yes, I noticed that the error was due to the IloFastMutex which was sent by reference.

Now, I made an instance of IloFastMutex global to the CB class and nothing is sent by value/by reference. The code works fine. I lock and unlock when I enter and exit, respectively.  It was also fine and worked.

Afterwards, now I totally removed the IloFastMutex and seems like that error is not appearing anymore and my code compiles fine.

Moreover, it runs fine and

when I set

 

nCores =  omp_get_num_procs();
    cplex.setParam(IloCplex::Threads, nCores);

the logs indicate that I am using all the 8 cores!!!  but is very slow. 

 

 

#CPLEXOptimizers
#DecisionOptimization

OK, my guess is that in the offending code you somehow used the copy constructor or assignment operator of class IloFastMutex and that results in undefined behavior (which may be a crash).

As for the slow running time: Are nodes processed slower than before or does CPLEX just need much more nodes to solve the problem? Do you spend a lot of time in your callbacks? Do you use deterministic or opportunistic multi-threading?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Falcon_G

Thanks for the comment.

The nodes are more.

I do spend a lot of time in two callbacks (lazy and usercut)  and I think it uses deterministic multi-threading

 

log:

 

Warning:  Output names have been modified to conform to LP format.
Lazy constraint(s) or lazy constraint callback is present.
    Disabling dual reductions (CPX_PARAM_REDUCE) in presolve.
    Disabling non-linear reductions (CPX_PARAM_PRELINEAR) in presolve.
Clique table members: 477.
MIP emphasis: balance optimality and feasibility.
MIP search method: traditional branch-and-cut.
Parallel mode: deterministic, using up to 4 threads.
Root relaxation solution time = 0.03 sec. (9.92 ticks)

 

General question:

Would you kindly describe the steps and the settings needed for the multi-thread running of branch and cuts?

By default, CPLEX uses one thread if we use callbacks. So what steps/elements are required to run multi-thread?

Using IloFastMutex is a must? how to synchronize them? does it need o be one global shared among all CBs?

What should be done for an optima implementation of user-written BC in multi-thread?

 

 

#CPLEXOptimizers
#DecisionOptimization

If CPLEX just uses more nodes then this may be bad luck/performance variability: using multiple threads CPLEX will just take a different path through the search tree and that may be beneficial or not.

Here are some statements about multi-threading and callbacks that I hope answer your questions:

• When control callbacks are used then the default value 0 (zero) for parameter IloCplex::Threads means "use only a single thread"
• If you want to have control callbacks and multiple threads then you explicitly need to set IloCplex::Threads to a non-zero value, for example:
      cplex.setParam(IloCplex::Threads, cplex.getNumCores())
• Whether you need to use IloFastMutex or not depends on your code. If the different clones of your callback class share some data and concurrently write that data then you will need some sort of mutex to prevent race conditions. This is nothing that is special to CPLEX, it is just standard multi-threaded programming.
  If the callback clones do not share any data or share only read-only data then no mutex is needed.
• Like in any multi-threading code, a mutex serializes code. That is, a code path that is protected by a mutex cannot be run in parallel. To get best performance those critical sections should be kept as short as possible.
• About IloFastMutex: Instances of this class must not be copied (neither by copy constructor nor by assignment operator) as this may result in undefined behavior. One simple way to make sure this does not happen is to use IloFastMutex only through pointers in callback classes and pass the pointer to the mutex to the constructor. Something like this:
     class MyCallback : public ... {
        IloFastMutex *const mtx;
     public:
        MyCallback(IloEnv env, IloFastMutex *m) : ...(env), mtx(m) {}
        IloCplex::CallbackI *duplicateCallback() const { return new (getEnv()) MyCallback(getEnv(), mtx); }
        void main() {
           mtx->lock();
           ...
           mtx->unlock();
        }
     };
     ...
     int main(void) {
        IloFastMutex mtx;
        ...
        cplex.use(new (env) MyCallback(env, &mtx));
        cplex.solve();
     }
  This makes sure that all callbacks use the same mutex but does not ever copy the mutex (instead of pointers one could of course use references as well).

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Falcon_G

Thank you for your comments.

 

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: Falcon_G

Daniel,  how does deterministic or opportunistic multi-threading play role here?

#CPLEXOptimizers
#DecisionOptimization

To make your life a little simpler, CPLEX automatically serializes callback invocations in deterministic parallel. That is, in deterministic parallel, callbacks will never be invoked in parallel. This is a technical restriction that may be removed any time (soon), so you should not rely on that.

Deterministic and opportunistic multi-threading should not make any difference in your implementation: just always protect write-access to shared resources by an instance of IloFastMutex in case you use multiple threads.

#CPLEXOptimizers
#DecisionOptimization