Decision Optimization

Originally posted by: JeremyBloom

Python is playing an increasingly prominent role in Data Science, and it is desirable to enhance optimization's position in that market. At the same time, there are other languages that are vying for market share, notably R and Julia. I think that we could engineer our connection with Python through OPL and also connect to these other languages, and any that subsequently emerge, with minimal effort. My proposal is simply to simplify the existing OPL API and embed it in each of the target languages, perhaps through a REST API.

I believe that math programming modeling languages, and OPL in particular, play an important role in easing the way for OR personae to build applications. Their declarative syntax mirrors the mathematical formulation of optimization problems, providing clarity and compactness that is hard to achieve with procedural APIs, such as we have in Concert. So I think we should bring these advantages to the new languages we want to support.

We already have an API for OPL which essentially enables a Java program (or C++ or C#) to use an OPL .mod file to specify an optimization model, to exchange data with it, and to call the CPLEX solver. I have used this API to build a decomposition algorithm in Java as an alternative to using an OPLScript flow control script. I have talked to a number of customers in the past year or so who build applications, or whose IT departments recode applications, in Concert instead of OPL because of a preference for Java, without realizing that the OPL API could be used in this fashion. This feature of COS perhaps deserves to be more widely known. Also, from my personal experience, the OPL API is not as transparent as it might be, which could also inhibit wider use.

Of course, we don't want to translate this API into every language we want to support. We already maintain it in three languages, and we would just add more maintenance effort every time create a new API. But if we create a REST API, we only have to maintain that one. Each of these languages, I believe, already has classes that support REST, so that accessing OPL REST could simply rely on these.

In making this proposal, I have reviewed (at a high level) a couple of competitors. Sandia National Laboratories in the US offers an open-source Python modeling language called Pyomo (https://software.sandia.gov/trac/coopr/wiki/Pyomo). It is a curious mix of declarative and procedural syntax. I find it much clearer than Concert but more intimidating than OPL. See the example in pyomo-jnl.pdf  on p. 10-12 which translates an AMPL model into Python. Pyomo supports index sets, but it was not clear to me how to implement features like slicing; perhaps that requires a deeper understanding than I have of how Python handles set objects.

The new language Julia also has an optimization modeling facility called Jump (http://jump.readthedocs.org/en/latest/quickstart.html). Although it has a different syntax than Pyomo, structurally it is similar, and it certainly is not as concise and clear as OPL. I did not see any support for index sets.

In summary, I think simplifying our existing OPL API and porting our into REST gives us an easy access point to Python and other languages and leverages the many attractive features of OPL.

#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: CplexUser1453

I would like to say just that I think Jeremy's proposal is an EXCELLENT idea. A couple of other thoughts, from my personal experience:

1. I really like OPL - if you need to work with sparse data and have to deal with really complicated models you are really going to appreciate the wonderful things you can do with OPL. I used OPL in a couple of optimization MSc. thesis on very complicated industrial MILP problems, my students pushed it pretty far and were able to do pretty cool things in terms of building very efficient models.

2. I think that OPL is a bit under appreciated right now because people are simply not aware of the nice features. I myself really started to appreciate OPL only after I went though the IBM OPL training. I was using CPLEX pretty heavily for a number of years before attending the training and I had access to the OPL examples but I never really bothered to go through them to understand how slicing works for example.

3. For me, the major pain point with OPL, is integrating the OPL models into the codes I am writing. If you just model a MILP problem by itself, then OPL is great. However, if  you are solving a big number of MILPs in some code,  like a lagrangian relaxation for example (and let us say you do simple lagrangian, followed by an augmented lagrangian) then using OPLScript is not ideal. Being able to use Python or R for data processing/flow control would be a LOT more preferable.

4. Right now, I work as a consultant at a commodity trading company where most quant models are written in R or Python. There are definitely problems here where CPLEX could be used. However, given the extremely short development cycles here, CPLEX is not going to be used, ever, without a good Python or R connector. I have access to a CPLEX license but the last (very simple) model I wrote used GLPK - because of the rglpk connector made my life easy.  Of course CPLEX could solve the problem MUCH faster, but in this particular case, I prefer easy interfacing to computation speed...

#DecisionOptimization
#OPLusingCPLEXOptimizer

Hi,

more connectors between Python and OPL would help but we can already call OPL from Python:

subprocess.check_call(["oplrun", "c:/volsay.mod"])

will call opl and solve volsay.mod

This is only a building block but then through file communication, we can do much more.

regards

 

Alex Fleischer

PS:

Many how to with OPL at https://www.linkedin.com/pulse/how-opl-alex-fleischer/

Many examples from a very good book : https://www.linkedin.com/pulse/model-building-oplcplex-alex-fleischer/

Making optimization simple : https://www.linkedin.com/pulse/making-decision-optimization-simple-alex-fleischer/

#DecisionOptimization
#OPLusingCPLEXOptimizer

Let me give you a more complete example.

From Python in order to use CPLEX you may use DOCPLEX as can be seen in the example

https://rawgit.com/IBMDecisionOptimization/docplex-doc/master/docs/mp/diet.html

And then you use DOCPLEX which is the best way to do the linear model in Python.

See attached diet.py

You may also call OPL from Python and for that you may want to generate the .dat that will be used by OPL

See attached diet2.py that does the same as diet.py but in 2 steps : python and then OPL

Let me explain:

diet2.py

    # The goal of the diet problem is to select a set of foods that satisfies
    # a set of daily nutritional requirements at minimal cost.
    # Source of data: http://www.neos-guide.org/content/diet-problem-solver

    import subprocess

    FOODS = [
        ("Roasted Chicken", 0.84, 0, 10),
        ("Spaghetti W/ Sauce", 0.78, 0, 10),
        ("Tomato,Red,Ripe,Raw", 0.27, 0, 10),
        ("Apple,Raw,W/Skin", .24, 0, 10),
        ("Grapes", 0.32, 0, 10),
        ("Chocolate Chip Cookies", 0.03, 0, 10),
        ("Lowfat Milk", 0.23, 0, 10),
        ("Raisin Brn", 0.34, 0, 10),
        ("Hotdog", 0.31, 0, 10)
    ]

    NUTRIENTS = [
        ("Calories", 2000, 2500),
        ("Calcium", 800, 1600),
        ("Iron", 10, 30),
        ("Vit_A", 5000, 50000),
        ("Dietary_Fiber", 25, 100),
        ("Carbohydrates", 0, 300),
        ("Protein", 50, 100)
    ]

    FOOD_NUTRIENTS = [
        ("Roasted Chicken", 277.4, 21.9, 1.8, 77.4, 0, 0, 42.2),
        ("Spaghetti W/ Sauce", 358.2, 80.2, 2.3, 3055.2, 11.6, 58.3, 8.2),
        ("Tomato,Red,Ripe,Raw", 25.8, 6.2, 0.6, 766.3, 1.4, 5.7, 1),
        ("Apple,Raw,W/Skin", 81.4, 9.7, 0.2, 73.1, 3.7, 21, 0.3),
        ("Grapes", 15.1, 3.4, 0.1, 24, 0.2, 4.1, 0.2),
        ("Chocolate Chip Cookies", 78.1, 6.2, 0.4, 101.8, 0, 9.3, 0.9),
        ("Lowfat Milk", 121.2, 296.7, 0.1, 500.2, 0, 11.7, 8.1),
        ("Raisin Brn", 115.1, 12.9, 16.8, 1250.2, 4, 27.9, 4),
        ("Hotdog", 242.1, 23.5, 2.3, 0, 0, 18, 10.4)
    ]

    def writedat(dat,tupleName,tupleInstance):
        dat.write(tupleName)
        dat.write('={\r')
        for f in tupleInstance:
          dat.write('<')
          for e in f:
              
              if not isinstance(e, str):
                  dat.write(str(e))
              else:
                  dat.write("\"");
                  dat.write(e);
                  dat.write("\"");
              dat.write(',');
          dat.write('>');     
          dat.write('\r');    
        dat.write('};\r');

        

    def build_diet_model():

        dat = open('diet.dat','w')

        writedat(dat,'FOODS',FOODS);
        writedat(dat,'NUTRIENTS',NUTRIENTS);
        writedat(dat,'FOOD_NUTRIENTS',FOOD_NUTRIENTS);
         
        dat.close()

        subprocess.check_call(["C:/ILOG/CPLEXStudio1263/opl/bin/x64_win64/oplrun", "diet.mod", "diet.dat"])

    build_diet_model()

and the OPL model:

    tuple Food
    {
        key string name;
        float unit_cost;
        float qmin;
        float qmax;
    };

    {Food} FOODS=...;

    tuple Nutrient
    {
        key string name;
        float qmin;
        float qmax;
    }

    {Nutrient} NUTRIENTS=...;

    tuple food_nutrients
    {
        key string name;
        float q1;
        float q2;
        float q3;
        float q4;
        float q5;
        float q6;
        float q7;
    }

    {food_nutrients} FOOD_NUTRIENTS=...;

    float array_FOOD_NUTRIENTS[f in FOODS][n in NUTRIENTS];

    // turn tuple set into an array
    execute
    {
    for(var fn in FOOD_NUTRIENTS)
        for(var n in NUTRIENTS)
            array_FOOD_NUTRIENTS[FOODS.find(fn.name)][n]=fn[fn.getFieldName(1+Opl.ord(NUTRIENTS,n))];
    }

    // Decision variables
    dvar float qty[f in FOODS] in f.qmin .. f.qmax;

    // cost
    dexpr float cost=sum (f in FOODS) qty[f]*f.unit_cost;

    // KPI
    dexpr float amount[n in NUTRIENTS] = sum(f in FOODS)
    qty[f] * array_FOOD_NUTRIENTS[f,n];

    minimize cost;
    subject to
    {
    forall(n in NUTRIENTS) n.qmin<=amount[n]<=n.qmax;
    }

    execute
    {
    var f=new IloOplOutputFile("dietoutput.txt");
    f.writeln("quantity = ",qty);
    f.writeln("cost = ",cost);
    f.writeln("amount = ",amount);
    f.close();
    }

The OPL model that is called by Python will use the .dat that is built by Python :

    FOODS={
    <"Roasted Chicken",0.84,0,10,>
    <"Spaghetti W/ Sauce",0.78,0,10,>
    <"Tomato,Red,Ripe,Raw",0.27,0,10,>
    <"Apple,Raw,W/Skin",0.24,0,10,>
    <"Grapes",0.32,0,10,>
    <"Chocolate Chip Cookies",0.03,0,10,>
    <"Lowfat Milk",0.23,0,10,>
    <"Raisin Brn",0.34,0,10,>
    <"Hotdog",0.31,0,10,>
    };
    NUTRIENTS={
    <"Calories",2000,2500,>
    <"Calcium",800,1600,>
    <"Iron",10,30,>
    <"Vit_A",5000,50000,>
    <"Dietary_Fiber",25,100,>
    <"Carbohydrates",0,300,>
    <"Protein",50,100,>
    };
    FOOD_NUTRIENTS={
    <"Roasted Chicken",277.4,21.9,1.8,77.4,0,0,42.2,>
    <"Spaghetti W/ Sauce",358.2,80.2,2.3,3055.2,11.6,58.3,8.2,>
    <"Tomato,Red,Ripe,Raw",25.8,6.2,0.6,766.3,1.4,5.7,1,>
    <"Apple,Raw,W/Skin",81.4,9.7,0.2,73.1,3.7,21,0.3,>
    <"Grapes",15.1,3.4,0.1,24,0.2,4.1,0.2,>
    <"Chocolate Chip Cookies",78.1,6.2,0.4,101.8,0,9.3,0.9,>
    <"Lowfat Milk",121.2,296.7,0.1,500.2,0,11.7,8.1,>
    <"Raisin Brn",115.1,12.9,16.8,1250.2,4,27.9,4,>
    <"Hotdog",242.1,23.5,2.3,0,0,18,10.4,>
    };

and then write the result in the file dietoutput.txt

    quantity =  [0 2.1552 0 0 0 10 1.8312 0 0.9297]
    cost = 2.69040917
    amount =  [2000 800 11.278 8518.4 25 256.81 51.174]

BOTH docplex and (python + OPL) are possible

 

#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: ShaCplex

Hi sir,

Could you please tell me which was the IBM training you attended ?

 

 

 

 

Regards,

shahul

#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: pcacioppi

Your diet.mod assumes there are exactly 7 kinds of nutrients. What would be the way to write this file if there were an arbitrary number of nutrients (and foods). To be truly robust, you should also assume the food_nutrients table isn't fully populated. That is to say, the input data might not specify every food/nutrient combination, but rather only specify those pairs for which there is a positive amount of nutrient n in food f.

Thanks for any help.

#DecisionOptimization
#OPLusingCPLEXOptimizer

Hi,

in my example, I mainly aimed at showing how to call OPL from Python.

Of course, the OPL Model can be improved.

Let me address you concerns:

/*********************************************
 * OPL 12.7.0.0 Model
 * Author: AlexFleischer
 * Creation Date: 3 janv. 2017 at 16:32:12
 *********************************************/
tuple Food
{
    key string name;
    float unit_cost;
    float qmin;
    float qmax;
};

{Food} FOODS=...;

tuple Nutrient
{
    key string name;
    float qmin;
    float qmax;
}

{Nutrient} NUTRIENTS=...;

tuple food_nutrient_value
{
  string food;
  string nutrient;
  float value;
}

{food_nutrient_value} FOOD_NUTRIENTS=...;;

 

// Decision variables
dvar float qty[f in FOODS] in f.qmin .. f.qmax;

// cost
dexpr float cost=sum (f in FOODS) qty[f]*f.unit_cost;

// KPI
dexpr float amount[n in NUTRIENTS] = sum(f in FOODS)
qty[f] * sum(fn in FOOD_NUTRIENTS:fn.food==f.name && fn.nutrient==n.name) fn.value;

minimize cost;
subject to
{
forall(n in NUTRIENTS) n.qmin<=amount[n]<=n.qmax;
}

execute
{
var f=new IloOplOutputFile("dietoutput3.txt");
f.writeln("quantity = ",qty);
f.writeln("cost = ",cost);
f.writeln("amount = ",amount);
f.close();
}

regards

#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: pcacioppi

Looks good! Thanks.

#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: Avi_ash

Hi,

I am using subprocess.Popen instead of check_call because for me it was giving error - "raise CalledProcessError(retcode, cmd)".

Popen is not giving any error. But the problem is that there is no output file generated.

It is model is not actually running I guess and therefore not generating the output also.

Can someone please help?

 

#DecisionOptimization
#OPLusingCPLEXOptimizer

Hi,

and do you get an error if you call oplrun in the command line ?

regards

#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: Avi_ash

I am able to run it now using : call(["oplrun", "diet3.mod","diet3.dat"]).

This is basically running the opl model file from python shell through windows cmd by using "call" (add on top: from subprocess import call)

Because windows cmd was detecting "oplrun" command and was running the model, but python shell itself is not detecting "oplrun".

Thanks Alex.

 

#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: pjcpjcpjc

Just as a FUP for anyone who lands here from Google searching - the Opalytics team generalized Alex's suggestion and published it as part of the ticdat package.

 

https://t.co/8x44aojYLm

#DecisionOptimization
#OPLusingCPLEXOptimizer

And then later on came doopl

https://pypi.org/project/doopl/



------------------------------
[Alex] [Fleischer]
[EMEA CPLEX Optimization Technical Sales]
[IBM]
------------------------------

Original Message:
Sent: Tue February 16, 2021 12:30 PM
From: Archive User
Subject: Re: OPL in Python and other languages

Originally posted by: pjcpjcpjc

Just as a FUP for anyone who lands here from Google searching - the Opalytics team generalized Alex's suggestion and published it as part of the ticdat package.

 

https://t.co/8x44aojYLm

#DecisionOptimization