Decision Optimization

 View Only

  • 1.  Reading docplex.mp model from a SAV file is slower than recreating it from scratch

    Posted Fri November 06, 2020 03:40 PM
    Edited by System Test Fri January 20, 2023 04:47 PM

    Hi,

    I have a large docplex.mp model with about 10 million binary variables. I would like to generate and save the model in advance, and then later load it, make some small adjustments, and solve it. I found that although it takes less only 10 seconds to write the model using Model.export_as_sav(), it is taking 7 minutes to read it back in using ModelReader.read(). Reading the model back in is by far the dominant cost, because solving it only takes 45 seconds!

    Here's a very minimal example with a toy model that just contains 10M binary variables, and shows that ModelReader.read() is slower than building the model from scratch.

    Are there any tricks to speed up reading in a saved model?

    Thanks!

    In [1]: from docplex.mp.model import Model
In [2]: from docplex.mp.model_reader import ModelReader
In [3]: m = Model()
In [4]: %time x = m.binary_var_list(10_000_000)
CPU times: user 17.1 s, sys: 468 ms, total: 17.5 s Wall time: 17.5 s
In [5]: %time m.export_as_sav('model.sav')
CPU times: user 691 ms, sys: 278 ms, total: 969 ms Wall time: 970 ms
Out[5]: 'model.sav' In [6]: %time m2 = ModelReader.read('model.sav')
CPU times: user 43.3 s, sys: 2.27 s, total: 45.6 s Wall time: 46.2 s
    ------------------------------
    Leo Singer
    ------------------------------
    #DecisionOptimization


  • 2.  RE: Reading docplex.mp model from a SAV file is slower than recreating it from scratch

    IBM Champion
    Posted Fri November 06, 2020 06:45 PM
    This may be specific to docplex (or to Python). I did the same experiment in Java with CPLEX 12.10 (add 10M binary variables to an empty model, export it and then import it back in). Times were random, so I ran it a few times, but the pattern was consistent. Adding the variables to the initial model took around eight seconds. Exporting the model took under two seconds. Importing the SAV file into a new model took a bit over five seconds, making it faster than creating the original model.

    ------------------------------
    Paul Rubin
    Professor Emeritus
    Michigan State University
    ------------------------------

    Original Message


  • 3.  RE: Reading docplex.mp model from a SAV file is slower than recreating it from scratch

    Posted Sat November 07, 2020 04:06 AM
    Edited by System Test Fri January 20, 2023 04:26 PM
    Hi,

    when you import a big .sav with docplex this can be long since docplex will parse and rebuild all constraints and variables.

    If you want that to be fast then instead of docplex I recommend the matrix cplex api.

    And then you would write

    import cplex
    c = cplex.Cplex()
    out = c.set_results_stream(None)
    out = c.set_log_stream(None)
c.read("import.sav")

    regards

    ------------------------------
    [Alex] [Fleischer]
    [EMEA CPLEX Optimization Technical Sales]
    [IBM]
    ------------------------------

    Original Message


  • 4.  RE: Reading docplex.mp model from a SAV file is slower than recreating it from scratch

    Posted Sat November 07, 2020 08:51 AM
    Hi Alex,

    Thanks, I'll try that! I would like to use the docplex API for modifying some selected constraint right hand sides, retrieving solution results, etc., though. If I load a SAV file using the cplex API, is there a way to load it into docplex and only reconstruct the variables and constraints that I will care to have access to?

    Leo

    ------------------------------
    Leo Singer
    ------------------------------

    Original Message


  • 5.  RE: Reading docplex.mp model from a SAV file is slower than recreating it from scratch

    Posted Sat November 07, 2020 10:30 AM
    Edited by System Test Fri January 20, 2023 04:14 PM
    Hi,

    you can do the slight change in cplex matrix python api.

    Let me use the zoo example.

    Suppose you have zoo.lp

    \ENCODING=ISO-8859-1
    \Problem name: Configuration1

    Minimize
    obj1: 500 nbBus40 + 400 nbBus30
    Subject To
    c1: 40 nbBus40 + 30 nbBus30 >= 300
    Bounds
    nbBus40 >= 0
    nbBus30 >= 0
    Generals
    nbBus40 nbBus30
    End

    and then you write in python

    import cplex

c = cplex.Cplex()
out = c.set_results_stream(None)
out = c.set_log_stream(None)
c.read("zoo.lp")
c.linear_constraints.set_coefficients(0,0,45)
c.write("zoo2.lp")​




    then this will write zoo2.lp

    \ENCODING=ISO-8859-1
    \Problem name: zoo.lp

    Minimize
obj1: 500 nbBus40 + 400 nbBus30
Subject To
c1: 45 nbBus40 + 30 nbBus30 >= 300
Bounds
nbBus40 >= 0
nbBus30 >= 0
Generals
nbBus40 nbBus30
End​





    where you may notice the change from 40 to 45








    ------------------------------
    [Alex] [Fleischer]
    [EMEA CPLEX Optimization Technical Sales]
    [IBM]
    ------------------------------

    Original Message


  • 6.  RE: Reading docplex.mp model from a SAV file is slower than recreating it from scratch

    Posted Mon November 09, 2020 04:34 AM
    Edited by System Test Fri January 20, 2023 04:40 PM
    Thanks Leo for isolating this sample code, I'll study it in detail to see if I can improve this.
    When you read a SAV into DOcplex, it delegates the read to the native-code CPLEX DLL, so the basic read is "as fast" as CPLEX C code.
    The next step is to rebuild a full DOcplex model from the in-memory matrix, and this is what takes time in Python:
    in your case, creating 10 million Python "var" objects takes time.
    There's not much we can do there; more precisely,
    it is not possible to build a partial docplex view of a huge model.
    That's an interesting idea, and we'll see what we can do in that direction.

    For now, the only working alternative for very large models is using the Python matrix api (or C/C++)



    ------------------------------
    Philippe Couronne
    ------------------------------

    Original Message