Decision Optimization

Decision Optimization

Delivers prescriptive analytics capabilities and decision intelligence to improve decision-making.

 View Only

  • 1.  Docplex mp cost too many memory before solve

    Posted Wed February 16, 2022 11:04 AM
    Hi All,

    We run a python model on a 48GB memory VM(redhat 8.5), and it got killed when trying to add only 320 constraints, the process got killed when added 175 constraints only before we can solve the problem.

    [Env]
    OS: redhat 8.5
    CPU:2 core
    Mem:48GB
    python: 3.8
    cplex studio: 20.10

    after adding 5 constraints, the tracemalloc is like:
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:364: size=400 MiB, count=10, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:410: size=256 MiB, count=2792647, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:502: size=255 MiB, count=2783914, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:123: size=234 MiB, count=2792640, average=88 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/frame.py:3580: size=192 MiB, count=8377921, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:446: size=149 MiB, count=2792640, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:82: size=149 MiB, count=2783913, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:428: size=80.0 MiB, count=2, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:868: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1255: size=80.0 MiB, count=1, average=80.0 MiB
                /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1197: size=80.0 MiB, count=1, average=80.0 MiB

    When started to add 320 constraints by loop, the memory increased very quickly:
    [1st]
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:364: size=401 MiB, count=17466, average=23.5 KiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:410: size=256 MiB, count=2792647, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:123: size=234 MiB, count=2792640, average=88 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/frame.py:3580: size=192 MiB, count=8377921, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:446: size=149 MiB, count=2792640, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:428: size=80.0 MiB, count=2, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:868: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1255: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1197: size=80.0 MiB, count=1, average=80.0 MiB


    [30th]
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:364: size=587 MiB, count=34978, average=17.2 KiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:410: size=256 MiB, count=2792647, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:123: size=234 MiB, count=2792640, average=88 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/frame.py:3580: size=192 MiB, count=8377921, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:911: size=186 MiB, count=8116110, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/xcounter.py:253: size=167 MiB, count=30, average=5703 KiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:446: size=149 MiB, count=2792640, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:428: size=80.0 MiB, count=2, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:868: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1255: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1197: size=80.0 MiB, count=1, average=80.0 MiB

    [50th]
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:364: size=905 MiB, count=35018, average=26.5 KiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:911: size=509 MiB, count=22253850, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/xcounter.py:253: size=467 MiB, count=50, average=9566 KiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:410: size=256 MiB, count=2792647, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:123: size=234 MiB, count=2792640, average=88 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/frame.py:3580: size=192 MiB, count=8377921, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:446: size=149 MiB, count=2792640, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:428: size=80.0 MiB, count=2, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:868: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1255: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1197: size=80.0 MiB, count=1, average=80.0 MiB

    [100th]
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:364: size=2345 MiB, count=35118, average=68.4 KiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:911: size=2017 MiB, count=88142700, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/xcounter.py:253: size=1867 MiB, count=100, average=18.7 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:410: size=256 MiB, count=2792647, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:123: size=234 MiB, count=2792640, average=88 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/frame.py:3580: size=192 MiB, count=8377921, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:446: size=149 MiB, count=2792640, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:428: size=80.0 MiB, count=2, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:868: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1255: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1197: size=80.0 MiB, count=1, average=80.0 MiB

    [150th]
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:911: size=4524 MiB, count=197666550, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:364: size=4345 MiB, count=35218, average=126 KiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/xcounter.py:253: size=3867 MiB, count=150, average=25.8 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:410: size=256 MiB, count=2792647, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:123: size=234 MiB, count=2792640, average=88 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/frame.py:3580: size=192 MiB, count=8377921, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:446: size=149 MiB, count=2792640, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:428: size=80.0 MiB, count=2, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:868: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1255: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1197: size=80.0 MiB, count=1, average=80.0 MiB

    Got killed by OS at 174th:
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:911: size=6082 MiB, count=265737150, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:364: size=5942 MiB, count=35266, average=173 KiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/xcounter.py:253: size=5387 MiB, count=174, average=31.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:410: size=256 MiB, count=2792647, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:123: size=234 MiB, count=2792640, average=88 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/frame.py:3580: size=192 MiB, count=8377921, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:446: size=149 MiB, count=2792640, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:428: size=80.0 MiB, count=2, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:868: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1255: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1197: size=80.0 MiB, count=1, average=80.0 MiB

    for the above model, we defined a binary matrix decision var:  9017 * 320
    and the constraints definition is like:
    for intervalInx in range(self.numberIntervals):
        total_risk = [
            df_D.loc[asset_ID, intervalInx] * (1 - self.model.sum(
                [self.assets_assignment_vars[asset_ID, index] for index in range(0, intervalInx+1)])) for
            asset_ID in self.setOfAssets.keys()]

        total_count = [(1 - self.model.sum(
            [self.assets_assignment_vars[asset_ID, index] for index in range(0, intervalInx + 1)])) for asset_ID in
                       self.setOfAssets.keys()]
        self.model.add_constraint(self.model.sum(total_risk) <= value * self.model.sum(total_count),
                                  'risk' + str(intervalInx))
        logger.info('Added risk constraints for interval:' + str(intervalInx))
        snapshot = tracemalloc.take_snapshot()
        top_stats = snapshot.statistics('lineno')
        print("[ Top 20 ]")
        for stat in top_stats[:20]:
            print(stat)
    self.numberIntervals = 320
    self.assets_assignment_vars : decision var Matrix with size 9017(assets number) * 320(intervals)
    df_D is a constant number Matrix with the same size like decision var :assets_assignment_vars
    self.setofAssets.keys : id of the assets (totol: 9017 assets)

    Can anyone know why it occupied so many memory(48 GB) to add only 100+ constraits, is it caued by decision var too much. If so, is there any estimation for the memory consumption calcualtion to let user know how many memory will need considering the decision var and constraints complex(?).




    ------------------------------
    Lan Cao
    ------------------------------

    #DecisionOptimization


  • 2.  RE: Docplex mp cost too many memory before solve

    Posted Wed February 16, 2022 12:20 PM
    You are playing with a very big model (3 millions variables), so using any user friendly CPLEX API (docplex, C++, OPL...) will require extra care to avoid copy of data as those apis may copy some of it when you use mathematical operators.
    Going to a matrix oriented CPLEX api (cplex python package or C) may be easier to fit in your machine (but harder to code and maintain).

    From your snapshot, I see issues with scaling to big models due to the mathematical operators. And to work this around, you will have to use method calls instead of them and rewrite most of your block code.

    • First issue
    value * self.model.sum(total_count)​

    docplex will first create an expression to represent the sum, then will look to scale it by value. It will clone the sum and mutiply the coefs to protect the previous sum.
    Indeed, if you write something like

    e1 = self.model.sum(total_count)
e2 = value * e1

    you will still want e1 unchanged.

    To avoid this, you have to use the multiply method (Doc says: This method does not create a new expression but modifies the `self` instance.).

    So you should write something like

    self.model.sum(total_count).multiply(value)

    • 2nd similar issue 
      Then same approach for
    1 - self.model.sum

    that you should write something like

    self.model.sum(...).multiply(-1).add(1)
    • 3 rd issue
      total_risk and total_count are arrays of big sum expressions: total_risk = [sum(...), sum(...),...] with N1, N2,... terms
    ​So when you call sum(total_risk), you will create an expression from all of them with N1+N2+... terms.
    So you end up with dictionaries of (var -> coef) with millions and millions of coefs.

    ------------------------------
    Vincent Beraudier
    ------------------------------

    Original Message


  • 3.  RE: Docplex mp cost too many memory before solve

    Posted Mon February 21, 2022 09:42 AM
    Hi Vincent,

    Thanks for your suggestions, I tried the 1 & 2, it did not enhance too much, the process still got killed after adding only 120 constraints.
    About the 3rd issue, I did not quite understand your suggestions, I think the expression is actually very big as we have million of decision vars, is there any way to reduce the size of this expression. Should I use something like Matrix oriented API, will this API: https://ibmdecisionoptimization.github.io/docplex-doc/mp/docplex.mp.model.html#docplex.mp.model.Model.scal_prod_f work.

    [Top 20 ]
    /usr/local/lib/python3.8/site-packages/docplex/mp/xcounter.py:253: size=5334 MiB, count=240, average=22.2 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:911: size=2880 MiB, count=125844840, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/linear.py:364: size=400 MiB, count=10, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:410: size=190 MiB, count=2080087, average=96 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:123: size=174 MiB, count=2080080, average=88 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/frame.py:3580: size=143 MiB, count=6240241, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:446: size=111 MiB, count=2080080, average=56 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/mfactory.py:428: size=80.0 MiB, count=2, average=40.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:868: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1255: size=80.0 MiB, count=1, average=80.0 MiB
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1197: size=80.0 MiB, count=1, average=80.0 MiB
    /root/aio/aiolib/aiolib.py:120: size=78.8 MiB, count=3440799, average=24 B
    /root/aio/aiolib/Parse_AIO_Asset_info.py:70: size=78.8 MiB, count=3440799, average=24 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/aggregator.py:264: size=76.6 MiB, count=17334, average=4632 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/utils.py:867: size=55.5 MiB, count=2079823, average=28 B
    /usr/local/lib/python3.8/site-packages/docplex/mp/model.py:1190: size=55.5 MiB, count=2079823, average=28 B
    /root/aio/aiolib/aiolib.py:119: size=26.8 MiB, count=8667, average=3240 B
    /root/aio/aiolib/Parse_AIO_Asset_info.py:150: size=26.7 MiB, count=17334, average=1616 B
    /root/aio/aiolib/single_asset_schedule.py:189: size=18.3 MiB, count=17335, average=1104 B
    /usr/local/lib64/python3.8/site-packages/pandas/core/internals/managers.py:1957: size=16.1 MiB, count=6, average=2754 KiB
    ESC[32m[2022-02-21 05:53:19,031][INFO] Added risk constraints for interval:120

    ------------------------------
    Lan Cao
    ------------------------------

    Original Message