Originally posted by: giuliapedrielliPolimi

Hi,

I am solving an LP model developed using the C++ libraries for CPLEX 12.1 (using the IloConcert technology).
I am basically trying to repeatedly solve the LP just perturbating the right hand side at each iteration.

At the end of every iteration I call the end() method for all the decision variables and in general all the objects I have declared in the model, however after some iterations (not always at the same point) the application crashes and I receive the following information.

Fault Module Name: cplex121.dll
Exception Code: C0000005
Exception Offset: 00000000005ad355

I can't understand the reason why I get this exception, I also tried on other computers with larger RAM memory but I always get the same exception.

Can anybody help me on this?

Thank you,

Best,

Giulia
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Any chance you can provide a stacktrace for this error? For example from a debugger?
If not, is it possible to identify the statement in your code that causes the crash?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: giuliapedrielliPolimi

Thank you for your answer,

Unfortunately just a warning from windows appears even in debug state and I cannot identify the statement generating the error.
In addition sometimes the error is thrown during the deletion of the decision variables (while the method end() is being performed) other times during the solution process.

In addition, again sometimes I get the error I posted before other times I just receive the exception "Not enough memory" (always form windows interface). But the memory profile does not make me think of any memory problems (the memory usage is pretty constant).

Anything I can use to improve the debugging and have more insights?

Best,

Giulia
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

One option would be to try application verifier or purify, in case you have any of these tools available.
I also found this thread where people were able to extract a backtrace from a dump that was created by the exception. Maybe you can do the same (I cannot assist you with that since I never tried it myself)?

When you end() your objects, are you sure you never end the same object twice? A simple way to check that would be

#include <set>
...
std::set<void *> ended;
for each object o to be deleted {
   void *impl = reinterpret_cast<void *>(o.getImpl());
   if (ended.find(impl) != ended.end()) {
      std::cerr << impl << " was already deleted" << std::endl;
      ::abort();
   }
   set.insert(impl);
   o.end();
}

Are you sure you only end() objects that have been properly constructed/initialized? Are you end()ing objects in an exception handler or after an exception has been caught?
These were just wild guesses about what may go wrong.

Would it be an option to change your code so that you can just do env.end() instead of end()ing all objects individually?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: giuliapedrielliPolimi

Hi,

Thanks for the link, I actually got into it before writing in the forum and couldn't solve the problem looking at the log (I am not an expert though).
I will go on checking that way.

By the way, I also changed the code and now I just delete the environment. However the application crashes (after a longer time, though) throuwing this exception:

"the instruction in 0x805e9729 referenced to memory in 0x00000000, the memory couldn't be written".
I get that something referenced a null point in the memory, but since we are talking about replicates of the same optimization code I still don't get how this can happen and whether is a cplex problem or a problem of the OS.

Any guesses?

Thank you again,

Giulia
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

> Thanks for the link, I actually got into it before writing in the forum and couldn't solve the problem looking at the log (I am not an expert though).
> I will go on checking that way.
>
If you manage to extract the backtrace from the dump then please post it here. It may help to understand what is going wrong and where the application actually fails.

> By the way, I also changed the code and now I just delete the environment. However the application crashes (after a longer time, though) throuwing this exception:
>
> "the instruction in 0x805e9729 referenced to memory in 0x00000000, the memory couldn't be written".
> I get that something referenced a null point in the memory, but since we are talking about replicates of the same optimization code I still don't get how this can happen and whether is a cplex problem or a problem of the OS.
>
> Any guesses?
>
Since it happens after a long time: May this be an out-of-memory problem? Do you see large memory consumption in the task manager right before the crash occurs?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: giuliapedrielliPolimi

I don't see anything strange on the memory consumption. However running the memory profile I can see several:
1) null request errors
2) cache errors (having spikes when the .solve() is run

If the problem is within the solution process how can I solve it?

Attached you can find the dump file I generated debugging the program.

Thank you
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: giuliapedrielliPolimi

I don't know if this helps, but even replicating the SAME problem, i.e. solving exactly the same LP many times I get the error.
I have used different memory profilers, but still can't see any memory issue.

Thank you

Giulia
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Could you try the following with this single problem:
1. Export it to a SAV file using cplex.exportModel("model.sav")
2. Run the following very small program in a loop:

while (true) {
   IloEnv env;
   IloModel model(env);
   IloCplex cplex(model);
   cplex.importModel(model, "model.sav");
   cplex.solve();
   env.end();
}

Does this trigger an exception eventually? If so, does the problem persist if you add 'cplex.setParam(IloCplex::Threads, 1);' right before 'cplex.solve()'?
Also could you post an outline of your code here that illustrates exactly what you are doing?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: giuliapedrielliPolimi

Hi,

thank you, this seems to work, but only if I add the instruction related to the single threading. Can you explain me why?
Actually I tried also without exporting and importing the SAV format, so I guess I was not forgetting to delete objects.

Hence, it is not a memory problem and you suggest an error incurs synchronizing the threads at some point?

Best,

Giulia
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

I suspect this to be a known memory leak that occurs only when using multiple threads. Due to the memory leak the program runs out of memory eventually and a failing memory allocation is then handled improperly.

If my guess is correct then there are other ways to fix the problem (that allow you work with multiple threads again):

• either switch to a more recent version of CPLEX,
• or set environment variable MKL_DISABLE_FAST_MM to a non-empty value before you start your application (so that your application runs with this variable set to a non-empty value).

Either thing should fix the memory leak.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: giuliapedrielliPolimi

Hi,

I tried to modify the environment variable, unfortunately I get this exception: "Unhandled exception at 0x000007fefd8ca88d in OPT_mod.exe: Microsoft C++ exception: IloWrongUsage at memory location 0x001be3a0"
And now even if I deleted the change this error remains, and I can't debug/run my program!

Can you help me fix this?

Thank you

Giulia
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

OK, this exception is completely unrelated to changing environment variables. Something else must be wrong with your program.
Can you isolate the statement that triggers the exception?
Can you try running the program through a debugger and attempt to get a stack trace for the exception?
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: giuliapedrielliPolimi

Hi,

sorry it was my mistake, I tracked the exceptions and found out my licence was expired. Sorry about that, btw I fixed the problem and tried again to add the environment variable you suggested me.
In particular:
I removed the instruction of the single thread
Added the environment variable assigning to it a random positive value (2)

I started debugging the application, but, unfortunately the exception comes again with the same features it had at the beginning, so you suggest to upgrade my CPLEX version?

Best,

Giulia
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

If you set the environment variable correctly and it did not help then the problem is probably different from the one I had in mind.
However, if it is not too much trouble for you then I would suggest to upgrade to a more recent version of CPLEX and see if the problem persists.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: giuliapedrielliPolimi

Hi,

actually in any case even with the single thread, the exception comes in after 324 replications. I will try to run the same with a newer version if I can have it.

Giulia
#CPLEXOptimizers
#DecisionOptimization