Originally posted by: SystemAdmin

Hi everybody

(as mentioned in two other threads) I am currently implementing a branch-and-price algorithm.

For this, I create a branch-and-bound search tree and solve the node by column generation, so that I'm having in every node on master problem and various pricing problems.

When the tree becomes quite big, like 1400 nodes, the solution process stops and I am getting this message:

CPLEX Error 1234: Failed to create parallel thread.

What can I do to prevent this?
Does it mean, that I have to many problems "opened"?
Can I close a problem in some way and earn memory by this?
Can I "move" the problem cache from the memory to the hard disk?

All the problems are not very big, but there are many of them..

Thanks for your answers!
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

An additional information:

In the meantime I tried to work with the parameter "threads" and set it to 1 (no parallelism). But this didn't change anything.

Why does cplex try to create a parallel thread? And why does it not work?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

This error message occurs when the operating system cannot create more threads. What operating system do you use? What API do you use?
If you use C or C++: If you say you have many problems, does each problem have its own CPXENV/IloEnv or do they all belong to the same environment? Do you solve problems in parallel or one after the other?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

I am using Windows 7 (32 bit) with Cplex 12.1 (32 bit) and the Python API. But I also tried it on Windows Server (64 bit) with Cplex 12.1 (32bit, because of numpy) and it did not work either.

In my opinion I am solving the problems one after the other, each in a new iteration of a loop. Each node is an own clas object linking various MIP.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

This is really weird.
My best guess at the moment is that you are running out of memory and the OS is therefore unable to allocate resources for a new thread. But I have never seen this happen before so I am not quite sure whether this is the real problem.
You said you tried setting the thread count to 1. Where did you do this? In order to make sure that you never use more than 1 thread you should set the thread parameter to 1 right before each call to solve(). Did you do that?
When you say it did not help: Did it change anything at all. Could you solve more nodes than without this setting?
Are there other components of your program that spawn threads? If there are, do they release the threads properly?

Concerning saving memory: Do you make sure that you do not keep references to problems that you do not need any longer? What memory consumption does the task manager report when you observe this error? Are you close to the physical memory limit?
In general, to avoid storing the complete models in memory I see two options:
1. Write out the model to a SAV file and delete the in-memory model afterwards. When you actually want to solve the model then you need to read in the SAV file.
2. Don't store a copy of the whole model. Instead store only the delta between the model and the original model (and potentially serialize this delta to disk). When you want to solve the model then create it on the fly from the original model and the delta you stored.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Right now, I tested everything you told me on the Win7 32 bit machine with 3GB memory and made the following observations:

• When I run the program without setting the parallel thread parameter to 1, after like 1450 nodes the the memory usage is about 95% and the program stops with the error:

cplex.exceptions.CplexSolverError: CPLEX Error 1234: Failed to create parallel thread

The thread and handle counter stays quite constant (840 resp. 19600) during the operation.

• When I run the program WITH setting the parallel thread parameter to 1 right in front of every solve()-call, the program stops after like 1500 nodes (so very slightly more) and prints a different error:

cplex.exceptions.CplexSolverError: CPLEX Error 1001: Out of memory.

Thread and handle counter also stay almost the same during the operation.
So, its obviously a memory problem. But why do I get different errors?

I am currently deleting problems I don't need anymore by just deleting the reference to them (del prob; remember I am using Python), coz as far as I know there is no method similiar to CPXfreeprob for deleting a problem, is it?

For examining this problem I made run with this program:

a=[]
gc.enable()
for i in range(3000):
print i
a.append(cplex.Cplex())
del a[0]
gc.collect()
print a

Although I am deleting the reference to the object and using the garbage collector, the memory usage is growing constantly.

Do you think this is the same problem as reported in this thread, I just found?

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14423587
So, to sum up my questions:

• Why do I get different error messages, when I turn on / off parallel mode?
• How can I delete a cplex in Python to free space?
• Is the not working delete procedure possibly a bug in version 12.1 I have to fix?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Right now, I tested everything you told me on the Win7 32 bit machine with 3GB memory and made the following observations:

• When I run the program without setting the parallel thread parameter to 1, after like 1450 nodes the the memory usage is about 95% and the program stops with the error:

cplex.exceptions.CplexSolverError: CPLEX Error 1234: Failed to create parallel thread

The thread and handle counter stays quite constant (840 resp. 19600) during the operation.

• When I run the program WITH setting the parallel thread parameter to 1 right in front of every solve()-call, the program stops after like 1500 nodes (so very slightly more) and prints a different error:

cplex.exceptions.CplexSolverError: CPLEX Error 1001: Out of memory.

Thread and handle counter also stay almost the same during the operation.
So, its obviously a memory problem. But why do I get different errors?

I am currently deleting problems I don't need anymore by just deleting the reference to them (del prob; remember I am using Python), coz as far as I know there is no method similiar to CPXfreeprob for deleting a problem, is it?

For examining this problem I made run with this program:

a=[]
gc.enable()
for i in range(3000):
print i
a.append(cplex.Cplex())
del a[0]
gc.collect()
print a

Although I am deleting the reference to the object and using the garbage collector, the memory usage is growing constantly.

Do you think this is the same problem as reported in this thread, I just found?

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14423587
So, to sum up my questions:

• Why do I get different error messages, when I turn on / off parallel mode?
• How can I delete a cplex in Python to free space?
• Is the not working delete procedure possibly a bug in version 12.1 I have to fix?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

> So, to sum up my questions:
> - Why do I get different error messages, when I turn on / off parallel mode?
>
This is because you run out of memory in different places. If you get Error 1001 then CPLEX has tried to allocate some memory from the system but failed because there is no more memory available. If you get Error 1234 then CPLEX has asked the OS to create a new thread but the OS failed to do so because it could not find any more memory for the new thread. Actually, CPLEX cannot tell why creation of the thread failed and so it simply reports to you that thread creation failed.
It is not surprising that toggling parallel mode changes the execution path and therefore changes the memory usage pattern. This then moves the place at which you run out of memory.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Hello,

Yes, it looks like you have run into the same bug as reported here http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14423587 . As described in that thread, your options are either to upgrade to a newer version (CPLEX 12.1.1 or later) or to change your code to reuse instances of Cplex() objects wherever possible.

If you would prefer not to upgrade and you need some help modifying your code, please post your source code as an attachment on this forum thread or send it to me at pstarhill (at) us (dot) ibm (dot) com.

Philip Starhill
CPLEX Research Engineer
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Thanks for this useful answers! I am probably doing the upgrade as soon as possible. I cannot do it myself because of license reasons.

One last question:

When I am modifying an exisiting problem and reusing it - do I have to "reset" in some way the problem, so that the exisiting solutions are not influencing the new solutions?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

If you don't want CPLEX to use information from the previous solve then set CPX_PARAM_ADVIND to 0. Maybe also delete all MIP starts using cplex.MIP_starts.delete to save some more space.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

One other thing: In the task manager, can you please enable columns "Thread Count" and "Handle Count" (this is done via View->Select Columns ...). What is the handle and thread count when you observe the error?
#CPLEXOptimizers
#DecisionOptimization