Decision Optimization

 View Only
  • 1.  CPLEX and Spark

    Posted Sat January 14, 2023 03:58 PM
    I'm building an application that uses Spark to manage optimization models to be solved with CPLEX. I'm creating and storing a bunch of CPLEX artifacts (IloNumVar,  IloRange, etc) in Spark RDDs. I'm using IloCplexModeler to create the artifacts.

    I've worked out most of the bugs and I've run the application successfully using Java Streams in place of Spark RDDs.

    However, when I run with Spark, I get the error ilog.cplex.IloCplex.

    I note that IloCplex implements Serializable, so I'm puzzled about why I'm getting this error. It's a show stopper.

    I've done a bunch of research on the web, including checking Stack Overflow, but only a few posts address the integration of Spark with CPLEX, and I can't find any useful code.

    Can anyone explain how to work through this issue? Sample code would be greatly appreciated.

    BTW, I'm using Java, CPLEX Optimization Studio 12.9, and Spark 3.2.1.



  • 2.  RE: CPLEX and Spark

    IBM Champion
    Posted Sat January 14, 2023 05:54 PM
    This question was answered two years ago on Stack Overflow: Curiously, the OP was "J. Bloom" -- you?

    At any rate, if you have an instance "lp" of IloCplex, you can serialize lp.getModel(). When you deserialize it, you deserialize to an instance (call it "x") of IloCplexModeler, then use lp.setModel(x) to turn that back into an IloCplex instance.

    Paul Rubin
    Professor Emeritus
    Michigan State University

  • 3.  RE: CPLEX and Spark

    Posted Sun January 15, 2023 07:26 AM
    Paul reminded me that I asked a similar question a couple of years ago. I've just come back to this project, so I'll look at the answer again.

    What I've concluded is that you can't serialize the individual CPLEX modeling artifacts, like IloNumVar, only a whole model. That's not what I need, since I want to use the artifacts in Spark to create a model. I've decided to use Spark to create my own artifacts (which would be Serializable classes) and then generate the model from these artifacts, avoiding serializing the CPLEX artifacts.

    Jeremy Bloom

  • 4.  RE: CPLEX and Spark

    Posted Thu January 19, 2023 03:40 PM
    After a bunch of research and coding, I'm abandoning my effort to use Spark to create CPLEX models. I'm happy to discuss my findings in detail if you want to contact me directly.

    Jeremy Bloom