Hi all:

I'm using Cplex (12.10) python API, namely Docplex. I have a question about the process of constructing Cplex model. I would like to use multi-processing method to construct model, e.g. add variables and constraints, since the big problem I dealt with, which have 20000k+ constraints and 30000k variables. Therefore, the construction of Cplex model is time-consuming, about 30 minutes, and I need to iterate this process many times.
I notice that only one-cpu is used during construction. Is there any primitive support from cplex package to speed up this process? Or any idea about how to optimize this problem?
(BTW, I tried it with multiprocessing module in python, but it seems that object(model) cannot share between different processes.)

Thanks!

------------------------------
Jianan Zhou
------------------------------

#DecisionOptimization

Hi,

do you know that once you built a model you may change this model incrementally ? And not rebuild from scratch ?

In https://www.linkedin.com/pulse/making-optimization-simple-python-alex-fleischer/

see https://github.com/AlexFleischerParis/zoodocplex/blob/master/zooincremental.py

regards

------------------------------
ALEX FLEISCHER
------------------------------

Hi,

Thank you for giving advice.
Yes, I know it. And sorry, I forgot to mention that my model is different in different iterations. If I operate on the model constructed in the last iteration, I need to remove lots of constraints and add new constraints, which almost take the same time, or even more time than constructing from scratch. So I construct model from scratch in each iteration.

Best,

------------------------------
Jianan Zhou
------------------------------

Original Message:
Sent: Thu June 11, 2020 08:47 AM
From: ALEX FLEISCHER
Subject: Constructing model with multi-processing

Hi,

do you know that once you built a model you may change this model incrementally ? And not rebuild from scratch ?

In https://www.linkedin.com/pulse/making-optimization-simple-python-alex-fleischer/

see https://github.com/AlexFleischerParis/zoodocplex/blob/master/zooincremental.py

regards

------------------------------
ALEX FLEISCHER
------------------------------

Hello,

 Building the model cannot be parallelized, as the model object itself would have to be shared between processed.
Your model size is quite large, you could benefit by trying out the techniques described in this notebook:
https://github.com/IBMDecisionOptimization/docplex-examples/blob/master/examples/mp/jupyter/efficient.ipynb

The first ideas to try are:
- make sure variables are created in batches (Model.xxx_var_list or xxx_var_dict, not one by one)
- make sure your constraints are added to the model in batches, not one by one.
- disable the checker by adding the keyword argument checker='off' to the Model constructor
- avoid manipulating names  by adding the keyword argument ignore_names=True to the model constructor.

The notebook gives general ideas to optimize model build time, to go further would require additional information about your model,
for example,  the number of non-zeros, as printed by Cplex in the output log.

As regards running several models, with variants, in parallel, I wrote a sample code here:
https://github.com/PhilippeCouronne/docplex_contribs/blob/master/docplex_contribs/src/process_pool.py

together with an example showing a very simple monte-carlo-style sampling with N models running in parallel,
with slight variants.
Let me know if that helps your configuration.

Regards.
Philippe.

------------------------------
Philippe Couronne
------------------------------

Original Message:
Sent: Thu June 11, 2020 09:05 AM
From: Jianan Zhou
Subject: Constructing model with multi-processing

Hi,

Thank you for giving advice.
Yes, I know it. And sorry, I forgot to mention that my model is different in different iterations. If I operate on the model constructed in the last iteration, I need to remove lots of constraints and add new constraints, which almost take the same time, or even more time than constructing from scratch. So I construct model from scratch in each iteration.

Best,

------------------------------
Jianan Zhou
------------------------------


Original Message:
Sent: Thu June 11, 2020 08:47 AM
From: ALEX FLEISCHER
Subject: Constructing model with multi-processing

Hi,

do you know that once you built a model you may change this model incrementally ? And not rebuild from scratch ?

In https://www.linkedin.com/pulse/making-optimization-simple-python-alex-fleischer/

see https://github.com/AlexFleischerParis/zoodocplex/blob/master/zooincremental.py

regards

------------------------------
ALEX FLEISCHER

Hi Philippe:

Thank you so much! That's very helpful. I will have a try.

Best, 
Jianan

------------------------------
Jianan Zhou
------------------------------

Original Message:
Sent: Mon June 22, 2020 05:03 AM
From: Philippe Couronne
Subject: Constructing model with multi-processing

Hello,

 Building the model cannot be parallelized, as the model object itself would have to be shared between processed.
Your model size is quite large, you could benefit by trying out the techniques described in this notebook:
https://github.com/IBMDecisionOptimization/docplex-examples/blob/master/examples/mp/jupyter/efficient.ipynb

The first ideas to try are:
- make sure variables are created in batches (Model.xxx_var_list or xxx_var_dict, not one by one)
- make sure your constraints are added to the model in batches, not one by one.
- disable the checker by adding the keyword argument checker='off' to the Model constructor
- avoid manipulating names  by adding the keyword argument ignore_names=True to the model constructor.

The notebook gives general ideas to optimize model build time, to go further would require additional information about your model,
for example,  the number of non-zeros, as printed by Cplex in the output log.

As regards running several models, with variants, in parallel, I wrote a sample code here:
https://github.com/PhilippeCouronne/docplex_contribs/blob/master/docplex_contribs/src/process_pool.py

together with an example showing a very simple monte-carlo-style sampling with N models running in parallel,
with slight variants.
Let me know if that helps your configuration.

Regards.
Philippe.

------------------------------
Philippe Couronne
------------------------------


Original Message:
Sent: Thu June 11, 2020 09:05 AM
From: Jianan Zhou
Subject: Constructing model with multi-processing

Hi,

Thank you for giving advice.
Yes, I know it. And sorry, I forgot to mention that my model is different in different iterations. If I operate on the model constructed in the last iteration, I need to remove lots of constraints and add new constraints, which almost take the same time, or even more time than constructing from scratch. So I construct model from scratch in each iteration.

Best,

------------------------------
Jianan Zhou


Original Message:
Sent: Thu June 11, 2020 08:47 AM
From: ALEX FLEISCHER
Subject: Constructing model with multi-processing

Hi,

do you know that once you built a model you may change this model incrementally ? And not rebuild from scratch ?

In https://www.linkedin.com/pulse/making-optimization-simple-python-alex-fleischer/

see https://github.com/AlexFleischerParis/zoodocplex/blob/master/zooincremental.py

regards

------------------------------
ALEX FLEISCHER

Hi Jianan, 
I'am facing the same situation as you one vCPU is used for the construction and even for the solving phase, did you found any solutions?
at least for the solving phase, i have a model who takes hours.
Thnx, Best 

------------------------------
Ryad Cherifi
------------------------------

Original Message:
Sent: Mon June 22, 2020 09:55 AM
From: Jianan Zhou
Subject: Constructing model with multi-processing

Hi Philippe:

Thank you so much! That's very helpful. I will have a try.

Best, 
Jianan

------------------------------
Jianan Zhou
------------------------------


Original Message:
Sent: Mon June 22, 2020 05:03 AM
From: Philippe Couronne
Subject: Constructing model with multi-processing

Hello,

 Building the model cannot be parallelized, as the model object itself would have to be shared between processed.
Your model size is quite large, you could benefit by trying out the techniques described in this notebook:
https://github.com/IBMDecisionOptimization/docplex-examples/blob/master/examples/mp/jupyter/efficient.ipynb

The first ideas to try are:
- make sure variables are created in batches (Model.xxx_var_list or xxx_var_dict, not one by one)
- make sure your constraints are added to the model in batches, not one by one.
- disable the checker by adding the keyword argument checker='off' to the Model constructor
- avoid manipulating names  by adding the keyword argument ignore_names=True to the model constructor.

The notebook gives general ideas to optimize model build time, to go further would require additional information about your model,
for example,  the number of non-zeros, as printed by Cplex in the output log.

As regards running several models, with variants, in parallel, I wrote a sample code here:
https://github.com/PhilippeCouronne/docplex_contribs/blob/master/docplex_contribs/src/process_pool.py

together with an example showing a very simple monte-carlo-style sampling with N models running in parallel,
with slight variants.
Let me know if that helps your configuration.

Regards.
Philippe.

------------------------------
Philippe Couronne


Original Message:
Sent: Thu June 11, 2020 09:05 AM
From: Jianan Zhou
Subject: Constructing model with multi-processing

Hi,

Thank you for giving advice.
Yes, I know it. And sorry, I forgot to mention that my model is different in different iterations. If I operate on the model constructed in the last iteration, I need to remove lots of constraints and add new constraints, which almost take the same time, or even more time than constructing from scratch. So I construct model from scratch in each iteration.

Best,

------------------------------
Jianan Zhou


Original Message:
Sent: Thu June 11, 2020 08:47 AM
From: ALEX FLEISCHER
Subject: Constructing model with multi-processing

Hi,

do you know that once you built a model you may change this model incrementally ? And not rebuild from scratch ?

In https://www.linkedin.com/pulse/making-optimization-simple-python-alex-fleischer/

see https://github.com/AlexFleischerParis/zoodocplex/blob/master/zooincremental.py

regards

------------------------------
ALEX FLEISCHER