Hello,
Building the model cannot be parallelized, as the model object itself would have to be shared between processed.
Your model size is quite large, you could benefit by trying out the techniques described in this notebook:
https://github.com/IBMDecisionOptimization/docplex-examples/blob/master/examples/mp/jupyter/efficient.ipynbThe first ideas to try are:
- make sure variables are created in batches (Model.xxx_var_list or xxx_var_dict, not one by one)
- make sure your constraints are added to the model in batches, not one by one.
- disable the checker by adding the keyword argument checker='off' to the Model constructor
- avoid manipulating names by adding the keyword argument ignore_names=True to the model constructor.
The notebook gives general ideas to optimize model build time, to go further would require additional information about your model,
for example, the number of non-zeros, as printed by Cplex in the output log.
As regards running several models, with variants, in parallel, I wrote a sample code here:
https://github.com/PhilippeCouronne/docplex_contribs/blob/master/docplex_contribs/src/process_pool.pytogether with an example showing a very simple monte-carlo-style sampling with N models running in parallel,
with slight variants.
Let me know if that helps your configuration.
Regards.
Philippe.
------------------------------
Philippe Couronne
------------------------------
Original Message:
Sent: Thu June 11, 2020 09:05 AM
From: Jianan Zhou
Subject: Constructing model with multi-processing
Hi,
Thank you for giving advice.
Yes, I know it. And sorry, I forgot to mention that my model is different in different iterations. If I operate on the model constructed in the last iteration, I need to remove lots of constraints and add new constraints, which almost take the same time, or even more time than constructing from scratch. So I construct model from scratch in each iteration.
Best,
------------------------------
Jianan Zhou
Original Message:
Sent: Thu June 11, 2020 08:47 AM
From: ALEX FLEISCHER
Subject: Constructing model with multi-processing
Hi,
do you know that once you built a model you may change this model incrementally ? And not rebuild from scratch ?
In https://www.linkedin.com/pulse/making-optimization-simple-python-alex-fleischer/
see https://github.com/AlexFleischerParis/zoodocplex/blob/master/zooincremental.py
regards
------------------------------
ALEX FLEISCHER
Original Message:
Sent: Thu June 11, 2020 08:34 AM
From: Jianan Zhou
Subject: Constructing model with multi-processing
Hi all:
I'm using Cplex (12.10) python API, namely Docplex. I have a question about the process of constructing Cplex model. I would like to use multi-processing method to construct model, e.g. add variables and constraints, since the big problem I dealt with, which have 20000k+ constraints and 30000k variables. Therefore, the construction of Cplex model is time-consuming, about 30 minutes, and I need to iterate this process many times.
I notice that only one-cpu is used during construction. Is there any primitive support from cplex package to speed up this process? Or any idea about how to optimize this problem?
(BTW, I tried it with multiprocessing module in python, but it seems that object(model) cannot share between different processes.)
Thanks!
------------------------------
Jianan Zhou
------------------------------
#DecisionOptimization