Decision Optimization

 View Only
Expand all | Collapse all

WML Decision Optimization - Tuple Format

  • 1.  WML Decision Optimization - Tuple Format

    Posted Wed March 10, 2021 01:31 PM
    Edited by System Test Fri January 20, 2023 04:18 PM
    Hi IBM,
    I am trying to setup WML for decision optimization. As a test, I borrowed "Bidding" example. The data in tuple formats were not correctly recognized by WML. WML seems to take only "CSV" file format. Please help me to learn how to make WML recognize tuple formats.
    #DecisionOptimization


  • 2.  RE: WML Decision Optimization - Tuple Format

    Posted Wed March 10, 2021 03:13 PM
    Edited by System Test Fri January 20, 2023 04:11 PM
    Hi,

    in Aha there is a wish to allow .dat in Watson Studio: https://ibm-data-and-ai.ideas.aha.io/ideas/WS-I-149

    In the meantime, I encourage you to rely on "How to export into a csv file ?"

    https://www.linkedin.com/pulse/how-opl-alex-fleischer/

    NB:

    With WML, .dat are allowed as can be seen in the oplrunwml tool : https://alexfleischer-84755.medium.com/command-line-call-opl-cplex-model-in-watson-machine-learning-518aba1be0b4


    ------------------------------
    [Alex] [Fleischer]
    [EMEA CPLEX Optimization Technical Sales]
    [IBM]
    ------------------------------



  • 3.  RE: WML Decision Optimization - Tuple Format

    Posted Wed March 10, 2021 03:45 PM
    I have just casted my vote on the wish (WS-I-149).
    In the mean time, I will follow your suggestion - exporting tuples into CSV file.
    Thanks for the sample opl script for that!

    ------------------------------
    Andy Ham
    ------------------------------



  • 4.  RE: WML Decision Optimization - Tuple Format

    Posted Wed March 10, 2021 06:26 PM

    Using your script, I have successfully exported most of tuple data into csv files, but I see two errors. I imagined a smooth transitioning from OPL environment to WML, but it is not. Sad..

    1. I am using array-tuple in OPL. I manually exported (since your script does support this format) and fed WML, but I got an following error.

    [2021-03-10T23:03:38Z, INFO] Element 'DataK' provided in CSV data source cannot be mapped to OPL type: 'ARRAY' as this type is not supported for input data on the Cloud.

    tuple t_Travel {
    key int b1; key int b2; int d;
    }; {t_Travel} DataK[Mchs];

    2. I am also using a following tuple format that is not supported by your script.

    tuple t_Mx {
    key int job;
    {int} mch;
    }; {t_Mx}

    [2021-03-10T23:03:38Z, SEVERE] expecting a collection and found an integer/float

    Can you please suggest any solution?
    Thanks,
    Andy



    ------------------------------
    Andy Ham
    ------------------------------



  • 5.  RE: WML Decision Optimization - Tuple Format

    Posted Thu March 11, 2021 11:59 AM
    Hi,

    indeed my script works with flat tuple sets.

    You may use scripting to turn your tuple sets with arrays and sets into a flat tuple set:

        range r=1..2;
        
        int a1[r]=[1,2];
        int a2[r]=[1,4];
        {int} set1={5,6};
        {int} set2={9,10};
        tuple t
        {
        string firstname;
        int number[r];
        {int} intset;
        }
        {t} s={<"Nicolas",a1,set1>,<"Alexander",a2,set2>};
        
        tuple t2
        {
          string firstname;
          int keynumber;
          int number;
          int intset;
        }
        
        {t2} s2;
        
        execute
        {
          for(var i in s)
            for(var j in r) 
              for(var k in i.intset) 
                s2.add(i.firstname,j,i.number[j],k)
          for(var l in s2 )writeln(l);
        }
      


    which gives

     <"Nicolas" 1 1 5>
     <"Nicolas" 1 1 6>
     <"Nicolas" 2 2 5>
     <"Nicolas" 2 2 6>
     <"Alexander" 1 1 9>
     <"Alexander" 1 1 10>
     <"Alexander" 2 4 9>
     <"Alexander" 2 4 10>


    More scripting tricks in https://www.linkedin.com/pulse/javascript-within-opl-cplex-alex-fleischer/



    ------------------------------
    [Alex] [Fleischer]
    [EMEA CPLEX Optimization Technical Sales]
    [IBM]
    ------------------------------



  • 6.  RE: WML Decision Optimization - Tuple Format

    Posted Thu March 11, 2021 01:13 PM

    I will follow your suggestion. In fact, that was my last choice. Looks like there is no other way other than flattening data.

    I sincerely appreciate your help. You have a great day!



    ------------------------------
    Andy Ham
    ------------------------------



  • 7.  RE: WML Decision Optimization - Tuple Format

    Posted Fri March 12, 2021 11:42 AM

    I flatten the array tuple by adding machine ID into b1(1000*MchID+b1) and b2 (1000*MchID+b2) fields and assign "types" of (mch.id*1000+jid) to a sequence. It works ok with a small problem, but it got really slow when the size gets bigger. How to avoid this performance issue coming from flattening tuples?

    tuple t_Travel {
    key int b1; key int b2; int d;
    }; {t_Travel} DataK[Mchs];

    tuple t_Travel {
    key int b1; key int b2; int d;
    }; {t_Travel} DataFlatten; //



    ------------------------------
    Andy Ham
    ------------------------------



  • 8.  RE: WML Decision Optimization - Tuple Format
    Best Answer

    Posted Fri March 12, 2021 12:13 PM
    Hi,

    what you did with javascript can also be done with OPL modeling.

    See https://github.com/AlexFleischerParis/oplscripting/blob/main/turntuplesetintoflattupleset.mod

    within https://www.linkedin.com/pulse/javascript-within-opl-cplex-alex-fleischer/

    /*
    
    here we ll see how to turn a non flat tuple set with arrays, sub tuple and sets inside
    into a flat tuple set that can be tuen into a CSV that can be used within
    Cloudpak for Data Watson Studio
    
    */
    
        execute
        {
          
          function isTupleSetFlat(tupleSet)
          {
            var nbFields=tupleSet.getNFields();
            var fistLine=Opl.first(tupleSet);
            var res=1; // True unless proved false
            
            for(var j=0;j<nbFields;j++)
            {
              var value=fistLine[tupleSet.getFieldName(j)];
              var typ=typeof(value);
              writeln(tupleSet.getFieldName(j), " is ",typ);
              if ((typ!="number") && (typ!="string"))  res=0;
            }   
            if (res==1) writeln("This tuple set is flat");
            else  writeln("This tuple set is not flat"); 
            return res;
        }
      }    
        
        
        range r=1..2;
        
        int a1[r]=[1,2];
        int a2[r]=[1,4];
        {int} set1={5,6};
        {int} set2={9,10};
        
        tuple subt
        {
          int x;
          int y;
        }
        
        tuple t
        {
        string firstname;
        int number[r];
        {int} intset;
        subt sub;
        }
        
        // not flat tuple set
        {t} s={<"Nicolas",a1,set1,<1,2>>,<"Alexander",a2,set2,<6,7>>};
        
        tuple t2
        {
          string firstname;
          int keynumber;
          int number;
          int intset;
          int sub1;
          int sub2;
        }
        
        {t2} s2;
        
        // turn s that is not flat into s2 that is a flat tuple set
        execute
        {
          writeln();
          writeln("is tuple set s flat ? ");
          writeln(isTupleSetFlat(s));
          writeln();
          
          for(var i in s)
            for(var j in r) 
              for(var k in i.intset) 
                s2.add(i.firstname,j,i.number[j],k,i.sub.x,i.sub.y)
                
          
          
          writeln();
          writeln("is tuple set s2 flat ? ");
          writeln(isTupleSetFlat(s2));
          writeln();
        }
        
        // the same but through OPL modeling and not scripting
        {t2} s2bis=union (i in s) union(j in r) union(k in i.intset) {<i.firstname,j,i.number[j],k,i.sub.x,i.sub.y>};
        
        execute display_s2
        {
          writeln("s=");
          writeln(s);
          writeln();
          writeln("s2=");
          for(var l in s2 )writeln(l);
          writeln();
          writeln("s2bis=");
          for(var l in s2bis )writeln(l);
        }
      
      
      /*
      
      gives
      
      is tuple set s flat ? 
    firstname is string
    number is IloMap
    intset is IloDiscreteDataCollection
    sub is IloTuple
    This tuple set is not flat
    0
    
    
    is tuple set s2 flat ? 
    firstname is string
    keynumber is number
    number is number
    intset is number
    sub1 is number
    sub2 is number
    This tuple set is flat
    1
    
    s=
     {<"Nicolas" [1 2] {5 6} <1 2>>
         <"Alexander" [1 4] {9 10} <6 7>>}
    
    s2=
     <"Nicolas" 1 1 5 1 2>
     <"Nicolas" 1 1 6 1 2>
     <"Nicolas" 2 2 5 1 2>
     <"Nicolas" 2 2 6 1 2>
     <"Alexander" 1 1 9 6 7>
     <"Alexander" 1 1 10 6 7>
     <"Alexander" 2 4 9 6 7>
     <"Alexander" 2 4 10 6 7>
     
     s2bis=
     <"Nicolas" 1 1 5 1 2>
     <"Nicolas" 1 1 6 1 2>
     <"Nicolas" 2 2 5 1 2>
     <"Nicolas" 2 2 6 1 2>
     <"Alexander" 1 1 9 6 7>
     <"Alexander" 1 1 10 6 7>
     <"Alexander" 2 4 9 6 7>
     <"Alexander" 2 4 10 6 7>
     
     */


    ------------------------------
    [Alex] [Fleischer]
    [EMEA CPLEX Optimization Technical Sales]
    [IBM]
    ------------------------------



  • 9.  RE: WML Decision Optimization - Tuple Format

    Posted Fri March 12, 2021 01:57 PM

    "the same but through OPL modeling and not scripting"

    Thanks! That was it! I found an way to change some of scripts into OPL modeling. It has been a lot of works to go around "csv" limitation on Watson ^^



    ------------------------------
    Andy Ham
    ------------------------------



  • 10.  RE: WML Decision Optimization - Tuple Format

    Posted Fri March 12, 2021 03:08 PM

    I have successfully tested a same code on both WML (w/ basic configuration) and local OPL. I don't believe this result. WML cloud delivered a slower computation compared to my own laptop. I will try WML with full configuration in coming weeks, but I cannot understand.

    Maximization problem staticLex (A, B)

    WML Cloud
    [2021-03-12T19:53:51Z, INFO] ! Time = 600.02s, Average fail depth = 245, Memory usage = 729.2 MB
    [2021-03-12T19:53:51Z, INFO] ! Current objective is 594; 17


    Laptop with i9 and 32 G memory on OPL 20.1
    New objective is 604; 11
    ! Time = 595.67s, Average fail depth = 254, Memory usage = 7.8 GB



    ------------------------------
    Andy Ham
    ------------------------------



  • 11.  RE: WML Decision Optimization - Tuple Format

    Posted Fri March 12, 2021 03:14 PM
    Hi,

    with WML have you used a small, medium or large machine ?

    regards

    ------------------------------
    [Alex] [Fleischer]
    [EMEA CPLEX Optimization Technical Sales]
    [IBM]
    ------------------------------



  • 12.  RE: WML Decision Optimization - Tuple Format

    Posted Fri March 12, 2021 03:20 PM
    I don't know where I can select "small, medium or large machine".

    I realized that WML is running with old CP version. 
    [2021-03-12T19:45:42Z, INFO] ! -------------------------------------------------- CP Optimizer 12.10.0.0 --

    ------------------------------
    Andy Ham
    ------------------------------



  • 13.  RE: WML Decision Optimization - Tuple Format

    Posted Mon March 15, 2021 10:08 AM
    In experiments, the configuration is small which means 2 cores
    regards

    ------------------------------
    [Alex] [Fleischer]
    [EMEA CPLEX Optimization Technical Sales]
    [IBM]
    ------------------------------



  • 14.  RE: WML Decision Optimization - Tuple Format

    Posted Mon March 15, 2021 10:30 AM
    I could not locate the experiment you mentioned, but I am using free "Lite" license. I guess "Lite" has 2 cores. I will get a full license in coming weeks. Let me update here after testing under full license. I hope WML has the latest CPO version ^^




    ------------------------------
    Andy Ham
    ------------------------------