Dear members of the Decision Optimization Community,

I am solving a linear optimization problem (with all variables being binary) using Cplex 12.10 and encountered the "CPLEX Error 3019: Failure to solve MIP subproblem." error for some instances (see "log_12_10.txt"). I am aware that the question has been discussed already in some other posts, however, the suggested fixes (increasing numerical emphasis, updating to recent Cplex version etc.) did not worked out for me.

I attached the model ("test.sav") and a small Python script ("test.py") reading and solving "test.sav". In addition, I put the parameter "preprocessing reduce" to one (to apply only primal reductions, as I am going to use lazy constraints later on which are incompatible with dual reductions). Based on numerical experiments, I observed the following:

• the error does not occur if c.parameters.preprocessing.reduce.set(1) is removed
• the error does not occur when using the prior version Cplex 12.9 (see "log_12_9.txt" attachted) with c.parameters.preprocessing.reduce.set(1)
• I think the model is uncritical from a numerical point of view, at least all variable coefficients are either -1,0,1/3 or 1, so there is only a small spread in the coefficients.

Assuming that there is in fact a problematic subproblem, I understand that applying only primal reductions or using a prior Cplex version might results in a different Branch and Bound Tree, where the problematic subproblem is maybe pruned and never touched. However, I observed this behavior for several instances, so this explanation seems quite unlikely for me in general (note: i attached an instance with the error occurring early during optimization/close to the root, sometimes it happens after processing several nodes).

In summary, I don't understand why this error occurs for the model under consideration. Especially, I don't understand if and how the error is related with the preprocessing reduce parameter.
I am very grateful for any explanation of this behavior and for any advice to overcome the problem. Thank you for your help.

Best regards



------------------------------
Jan E. J.
------------------------------

#DecisionOptimization

With the preprocessing reduce parameter set to 1, CPLEX 20.1 solves your problem to optimality in under a minute on my PC, getting a solution with objective value 21. Without the preprocessing setting, CPLEX finds an incumbent of 22 and a best bound of 21 fairly quickly but after 2+ minutes has still not gotten the incumbent down to 21.

I also tried CPLEX 12.10 (the only other version I have installed). With the preprocessing setting, it gets a proven optimum of 21 in under two minutes. Turning on the data check parameter did not change that. (I did not expect it to, but tried just for completeness.)

I'm running on Linux Mint, and CPLEX uses four threads (deterministic parallel) as opposed to eight on your machine according to your log. So perhaps the problem has something to do with either computer architecture or number of parallel threads. You might try 12.10 with deterministic parallel limited to four threads and see if that helps.

------------------------------
Paul Rubin
Professor Emeritus
Michigan State University
------------------------------

Dear Prof. Rubin,

thank you very much for your swift response. I am going to run 12.10 with deterministic parallel limited to four threads, as suggested.

Unfortunately, I missed to mention that I used a different system (but also with 8 threads) when running 12.9. I am going to install 12.10 on that second system as well to have a more meaningful comparison.

I will update the post with the new results and system properties once all adjustments have been made.

Best regards



------------------------------
Jan E. J.
------------------------------

Original Message:
Sent: Tue March 02, 2021 03:38 PM
From: Paul Rubin
Subject: CPLEX Error 3019: Failure to solve MIP subproblem.

With the preprocessing reduce parameter set to 1, CPLEX 20.1 solves your problem to optimality in under a minute on my PC, getting a solution with objective value 21. Without the preprocessing setting, CPLEX finds an incumbent of 22 and a best bound of 21 fairly quickly but after 2+ minutes has still not gotten the incumbent down to 21.

I also tried CPLEX 12.10 (the only other version I have installed). With the preprocessing setting, it gets a proven optimum of 21 in under two minutes. Turning on the data check parameter did not change that. (I did not expect it to, but tried just for completeness.)

I'm running on Linux Mint, and CPLEX uses four threads (deterministic parallel) as opposed to eight on your machine according to your log. So perhaps the problem has something to do with either computer architecture or number of parallel threads. You might try 12.10 with deterministic parallel limited to four threads and see if that helps.

------------------------------
Paul Rubin
Professor Emeritus
Michigan State University
------------------------------

Dear Prof. Rubin,

I have reconfigured the numerical experiments as follows:

• CPLEX Versions 12.9, 12.10, 20.1
• With or without deterministic parallel limited to four threads
• With or without c.parameters.preprocessing.reduce.set(1)
• Notebook 1: Windows 10 Pro 64-bit, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz (8 CPUs), ~2.1GHz, 32GB RAM, Dell Latitude 7400
• Notebook 2: Windows 10 Pro 64-bit, Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz (8 CPUs), ~2.0GHz, 16GB RAM, HP ProBook 470 G5

This results in 24 different settings. I run each setting with ten instances (among them test.sav) known to trigger error 3019 when using the setting defined by 12.10, Notebook 1, with c.parameters.preprocessing.reduce.set(1) and without deterministic parallel limited to four threads (called setting S₁ in the following).

Only setting S₂ defined by 12.10, Notebook 2, with c.parameters.preprocessing.reduce.set(1) and without deterministic parallel limited to four threads showed the same problematic behavior as setting S₁ (i.e., for all ten instances error 3019 occurred [and very close to the root/before processing nodes with depth > 0]). Actually, I run this setting on a third notebook of a colleague, with the same results.

The error has not occurred for any instance when applying the remaining settings.

I was not aware that there is a newer CPLEX version than 12.10 (a lot of numbers have been skipped). Updating to 20.1 seems to fix the problem for me (or alternatively activating deterministic parallel with four threads), however, I still feel a bit uncomfortable with the situation. To be honest, I still can't explain where the problem was, why it occurred, why it doesn't occur under the new version or changed parameters, and what I would do in the future if the error occurs again. Is there any way to detect more precisely and comprehensible what the problem was?

Thank you very much for your help, it allowed me to continue my work.

Best regards

------------------------------
Jan E. J.
------------------------------

Original Message:
Sent: Tue March 02, 2021 03:38 PM
From: Paul Rubin
Subject: CPLEX Error 3019: Failure to solve MIP subproblem.

With the preprocessing reduce parameter set to 1, CPLEX 20.1 solves your problem to optimality in under a minute on my PC, getting a solution with objective value 21. Without the preprocessing setting, CPLEX finds an incumbent of 22 and a best bound of 21 fairly quickly but after 2+ minutes has still not gotten the incumbent down to 21.

I also tried CPLEX 12.10 (the only other version I have installed). With the preprocessing setting, it gets a proven optimum of 21 in under two minutes. Turning on the data check parameter did not change that. (I did not expect it to, but tried just for completeness.)

I'm running on Linux Mint, and CPLEX uses four threads (deterministic parallel) as opposed to eight on your machine according to your log. So perhaps the problem has something to do with either computer architecture or number of parallel threads. You might try 12.10 with deterministic parallel limited to four threads and see if that helps.

------------------------------
Paul Rubin
Professor Emeritus
Michigan State University
------------------------------

You are quite welcome. I'm glad you are able to get past the problem. As far as tracking down the source of the problem, I suspect that would require action by IBM programmers (who could run CPLEX 12.10 in a debugger and see exactly what is going on). My guess is that it is a subtle bug that appeared in 12.10 and was fixed (hopefully) in 20.1. The fact that you experience it when using eight threads but not when using four strongly suggests that parallelism is involved, which in my experience makes for tricky debugging. It might have something to do with faulty synchronization of some operations, but that is just speculation.

Hopefully the problem will not arise again, but if it does, I would suggest that you try running the model with a single thread, and if that prevents the error then try increasing the number of threads to the most that you can have without the error manifesting. Also, if it happens with 20.1, I would suggest filing a bug report with IBM.

Cheers,
Paul

------------------------------
Paul Rubin
Professor Emeritus
Michigan State University
------------------------------

Original Message:
Sent: Wed March 03, 2021 07:16 PM
From: Jan E. J.
Subject: CPLEX Error 3019: Failure to solve MIP subproblem.

Dear Prof. Rubin,

I have reconfigured the numerical experiments as follows:

• CPLEX Versions 12.9, 12.10, 20.1
• With or without deterministic parallel limited to four threads
• With or without c.parameters.preprocessing.reduce.set(1)
• Notebook 1: Windows 10 Pro 64-bit, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz (8 CPUs), ~2.1GHz, 32GB RAM, Dell Latitude 7400
• Notebook 2: Windows 10 Pro 64-bit, Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz (8 CPUs), ~2.0GHz, 16GB RAM, HP ProBook 470 G5

This results in 24 different settings. I run each setting with ten instances (among them test.sav) known to trigger error 3019 when using the setting defined by 12.10, Notebook 1, with c.parameters.preprocessing.reduce.set(1) and without deterministic parallel limited to four threads (called setting S₁ in the following).

Only setting S₂ defined by 12.10, Notebook 2, with c.parameters.preprocessing.reduce.set(1) and without deterministic parallel limited to four threads showed the same problematic behavior as setting S₁ (i.e., for all ten instances error 3019 occurred [and very close to the root/before processing nodes with depth > 0]). Actually, I run this setting on a third notebook of a colleague, with the same results.

The error has not occurred for any instance when applying the remaining settings.

I was not aware that there is a newer CPLEX version than 12.10 (a lot of numbers have been skipped). Updating to 20.1 seems to fix the problem for me (or alternatively activating deterministic parallel with four threads), however, I still feel a bit uncomfortable with the situation. To be honest, I still can't explain where the problem was, why it occurred, why it doesn't occur under the new version or changed parameters, and what I would do in the future if the error occurs again. Is there any way to detect more precisely and comprehensible what the problem was?

Thank you very much for your help, it allowed me to continue my work.

Best regards

------------------------------
Jan E. J.
------------------------------


Original Message:
Sent: Tue March 02, 2021 03:38 PM
From: Paul Rubin
Subject: CPLEX Error 3019: Failure to solve MIP subproblem.

With the preprocessing reduce parameter set to 1, CPLEX 20.1 solves your problem to optimality in under a minute on my PC, getting a solution with objective value 21. Without the preprocessing setting, CPLEX finds an incumbent of 22 and a best bound of 21 fairly quickly but after 2+ minutes has still not gotten the incumbent down to 21.

I also tried CPLEX 12.10 (the only other version I have installed). With the preprocessing setting, it gets a proven optimum of 21 in under two minutes. Turning on the data check parameter did not change that. (I did not expect it to, but tried just for completeness.)

I'm running on Linux Mint, and CPLEX uses four threads (deterministic parallel) as opposed to eight on your machine according to your log. So perhaps the problem has something to do with either computer architecture or number of parallel threads. You might try 12.10 with deterministic parallel limited to four threads and see if that helps.

------------------------------
Paul Rubin
Professor Emeritus
Michigan State University

Thanks again for the advice and further explanation, I will consider everything mentioned next time.

Best regards

------------------------------
Jan E. J.
------------------------------

Original Message:
Sent: Wed March 03, 2021 11:14 PM
From: Paul Rubin
Subject: CPLEX Error 3019: Failure to solve MIP subproblem.

You are quite welcome. I'm glad you are able to get past the problem. As far as tracking down the source of the problem, I suspect that would require action by IBM programmers (who could run CPLEX 12.10 in a debugger and see exactly what is going on). My guess is that it is a subtle bug that appeared in 12.10 and was fixed (hopefully) in 20.1. The fact that you experience it when using eight threads but not when using four strongly suggests that parallelism is involved, which in my experience makes for tricky debugging. It might have something to do with faulty synchronization of some operations, but that is just speculation.

Hopefully the problem will not arise again, but if it does, I would suggest that you try running the model with a single thread, and if that prevents the error then try increasing the number of threads to the most that you can have without the error manifesting. Also, if it happens with 20.1, I would suggest filing a bug report with IBM.

Cheers,
Paul

------------------------------
Paul Rubin
Professor Emeritus
Michigan State University
------------------------------


Original Message:
Sent: Wed March 03, 2021 07:16 PM
From: Jan E. J.
Subject: CPLEX Error 3019: Failure to solve MIP subproblem.

Dear Prof. Rubin,

I have reconfigured the numerical experiments as follows:

• CPLEX Versions 12.9, 12.10, 20.1
• With or without deterministic parallel limited to four threads
• With or without c.parameters.preprocessing.reduce.set(1)
• Notebook 1: Windows 10 Pro 64-bit, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz (8 CPUs), ~2.1GHz, 32GB RAM, Dell Latitude 7400
• Notebook 2: Windows 10 Pro 64-bit, Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz (8 CPUs), ~2.0GHz, 16GB RAM, HP ProBook 470 G5

This results in 24 different settings. I run each setting with ten instances (among them test.sav) known to trigger error 3019 when using the setting defined by 12.10, Notebook 1, with c.parameters.preprocessing.reduce.set(1) and without deterministic parallel limited to four threads (called setting S₁ in the following).

Only setting S₂ defined by 12.10, Notebook 2, with c.parameters.preprocessing.reduce.set(1) and without deterministic parallel limited to four threads showed the same problematic behavior as setting S₁ (i.e., for all ten instances error 3019 occurred [and very close to the root/before processing nodes with depth > 0]). Actually, I run this setting on a third notebook of a colleague, with the same results.

The error has not occurred for any instance when applying the remaining settings.

I was not aware that there is a newer CPLEX version than 12.10 (a lot of numbers have been skipped). Updating to 20.1 seems to fix the problem for me (or alternatively activating deterministic parallel with four threads), however, I still feel a bit uncomfortable with the situation. To be honest, I still can't explain where the problem was, why it occurred, why it doesn't occur under the new version or changed parameters, and what I would do in the future if the error occurs again. Is there any way to detect more precisely and comprehensible what the problem was?

Thank you very much for your help, it allowed me to continue my work.

Best regards

------------------------------
Jan E. J.


Original Message:
Sent: Tue March 02, 2021 03:38 PM
From: Paul Rubin
Subject: CPLEX Error 3019: Failure to solve MIP subproblem.

With the preprocessing reduce parameter set to 1, CPLEX 20.1 solves your problem to optimality in under a minute on my PC, getting a solution with objective value 21. Without the preprocessing setting, CPLEX finds an incumbent of 22 and a best bound of 21 fairly quickly but after 2+ minutes has still not gotten the incumbent down to 21.

I also tried CPLEX 12.10 (the only other version I have installed). With the preprocessing setting, it gets a proven optimum of 21 in under two minutes. Turning on the data check parameter did not change that. (I did not expect it to, but tried just for completeness.)

I'm running on Linux Mint, and CPLEX uses four threads (deterministic parallel) as opposed to eight on your machine according to your log. So perhaps the problem has something to do with either computer architecture or number of parallel threads. You might try 12.10 with deterministic parallel limited to four threads and see if that helps.

------------------------------
Paul Rubin
Professor Emeritus
Michigan State University