Decision Optimization

Decision Optimization

Delivers prescriptive analytics capabilities and decision intelligence to improve decision-making.

 View Only
Expand all | Collapse all

Ill Conditioned Problem

  • 1.  Ill Conditioned Problem

    Posted Wed April 11, 2012 08:48 PM

    Originally posted by: Eumpfenbach


    Pretty sure I have a linear programming that would fall under the category of "ill-conditioned". I want to solve it using column generation (so, with presolve off).

    If I just solve with the standard cplex algorithm, there is a giant difference in solve time with presolve on/off. With Presolve on, I see this information printed (using Matlab API).

    Tried aggregator 1 time.
    MIP Presolve eliminated 394 rows and 101 columns.
    MIP Presolve modified 59535 coefficients.
    Reduced MIP has 67028 rows, 55287 columns, and 389241 nonzeros.
    Reduced MIP has 255 binaries, 0 generals, 0 SOSs, and 0 indicators.
    Probing time =    0.05 sec.
    Tried aggregator 1 time.
    Reduced MIP has 67028 rows, 55287 columns, and 389241 nonzeros.
    Reduced MIP has 255 binaries, 0 generals, 0 SOSs, and 0 indicators.
    Presolve time =    0.55 sec.
    Found feasible solution after 0.64 sec.  Objective = -1.8148e+010
    


    My question: Is there anything in this information that points to the likely cause of the performance gap? Is it scaling? Something else that presolve might be doing (I am assuming that removing 400 rows and 100 columns has little impact on the solution time, given the size of the problem)? Is there any way that I can make cplex presolve agree with column generation? (I tried setting presolve.reduce = 0. It no longer removed the rows or columns, but also got rid of the message "MIP Presolve modified 59535 coefficients", which I am assuming is the part that is greatly improving performance.).

    Any advice would be greatly appreciated. Hopefully that can save me from having to do trial and error on my own. Thanks.
    #CPLEXOptimizers
    #DecisionOptimization


  • 2.  Re: Ill Conditioned Problem

    Posted Thu April 12, 2012 08:05 AM

    Originally posted by: T_O


    Did you try to solve your model with presolver and then to resolve without presolver?

    Best regards,
    Thomas
    #CPLEXOptimizers
    #DecisionOptimization


  • 3.  Re: Ill Conditioned Problem

    Posted Thu April 12, 2012 09:35 AM

    Originally posted by: Eumpfenbach


    Yes, there is a VERY large performance gap. I clearly need to presolve, but am also doing column generation. Therefore I need to:

    1) Make cplex presolve compatible with CG (I was kinda hoping that maybe as long as I didn't remove any rows or columns it would be. I'm not sure if it says cplex "modified coefficients" whether this means the dual of my solution is not a proper dual of the original problem. Maybe this is wishful thinking).

    or

    2) Create my own presolve routine. In this case, I was hoping for expert opinion as to what is the presolve routine most likely to cause the giant performance gap (I know no one can tell me for sure, but just an educated guess) so that I can try and do it myself.
    #CPLEXOptimizers
    #DecisionOptimization


  • 4.  Re: Ill Conditioned Problem

    Posted Thu April 12, 2012 02:16 PM

    Originally posted by: SystemAdmin


    I do not understand what you are trying to do. Apparently your are solving a MIP. How are you going to use column generation for this? You will not get duals!

    Or is it more like a rolling horizon application where you start with a small problem, and then use the solution of one period as a MIP start for the next solve?
    Tobias
    #CPLEXOptimizers
    #DecisionOptimization


  • 5.  Re: Ill Conditioned Problem

    Posted Thu April 12, 2012 03:05 PM

    Originally posted by: Eumpfenbach


    I have an MIP where the number of continuous variables is much, much greater than the number of binaries. Thus I am trying to use column generation to solve the subproblems efficiently. I have a separate branch-and-bound code and call cplex to solve the subproblems. I have had test trials where this approach was 5 times faster than cplex (due to greatly accelerated solving of the relaxations).
    Now I am using a different data set and believe it to be ill-conditioned. I need to do something to correct the problem.
    #CPLEXOptimizers
    #DecisionOptimization


  • 6.  Re: Ill Conditioned Problem

    Posted Thu April 12, 2012 08:16 PM

    Originally posted by: SystemAdmin


    And where do those presolve logs come from? Those are for MIPs, but as far as I understand you are using CPLEX only to solve the LP relaxations inside your own branch-and-bound framework...
    #CPLEXOptimizers
    #DecisionOptimization


  • 7.  Re: Ill Conditioned Problem

    Posted Thu April 12, 2012 08:47 PM

    Originally posted by: Eumpfenbach


    Correct. It is the same problem. I just posted logs from when I solve the full MIP as a standard cplex problem, not one of the relaxations. If it makes a difference, here is a log from the initial LP relaxation of the same problem:

    Selected objective sense:  MINIMIZE
    Selected objective  name:  R67423
    Selected RHS        name:  B
    Selected bound      name:  BOUND
    Parallel mode: deterministic, using up to 2 threads for concurrent optimization.
    Tried aggregator 1 time.
    LP Presolve eliminated 190 rows and 101 columns.
    Reduced LP has 67232 rows, 55287 columns, and 426827 nonzeros.
    


    I still suspect it is scaling. Scaling is disabled when presolve is turned off, right? I'm trying to write my own script to rescale the problem.
    #CPLEXOptimizers
    #DecisionOptimization


  • 8.  Re: Ill Conditioned Problem

    Posted Fri April 13, 2012 11:55 AM

    Originally posted by: EdKlotz


    > Eumpfenbach wrote:
    > Correct. It is the same problem. I just posted logs from when I solve the full MIP as a standard cplex problem, not one of the relaxations. If it makes a difference, here is a log from the initial LP relaxation of the same problem:
    >
    >
    
    > Selected objective sense:  MINIMIZE > Selected objective  name:  R67423 > Selected RHS        name:  B > Selected bound      name:  BOUND > Parallel mode: deterministic, using up to 2 threads 
    
    for concurrent optimization. > Tried aggregator 1 time. > LP Presolve eliminated 190 rows and 101 columns. > Reduced LP has 67232 rows, 55287 columns, and 426827 nonzeros. >
    

    >
    > I still suspect it is scaling. Scaling is disabled when presolve is turned off, right? I'm trying to write my own script to rescale the problem.
    Scaling is not disabled when presolve is turned off. If you want to turn off
    scaling, set CPLEX's scale parameter to -1.

    I am speculating now, but when you say that you suspect your model is ill conditioned, and also show that presolve modified a large number of coefficients,
    and find that turning presolve off dramatically degrades performance despite the
    fact that the output you posted indicates presolve removed relatively few constraints and variables, I wonder if you have some constraints with a big M
    formulation, e.g. x continuous, z binary, and

    x - Mz <= 0

    Here, M is a really large number (e.g. 1e+8 or higher). If other constraints
    imply an upper bound an x much smaller than M, presolve will probably detect
    this. It will then modify the large M value to the much smaller upper bound value. In that case, the range of coefficient values for the presolved model may be much smaller than the range for the unpresolved model. Thus, the
    original model may be ill conditioned, while the presolved model might not be. That in turn could explain the big difference in performance. To assess this,
    compare the log files of the two runs. Also, try using CPLEX's MIP kappa feature on both the original and presolved model. This provides info on the
    condition numbers of the optimal node LP bases, and will help you determine if
    the conditioning of the presolved model is significantly better than the original
    model. You can also read in the original model into interactive CPLEX, then write out the presolved model:

    CPLEX> read mymodel.sav
    CPLEX> write mymodel.pre

    Then read in the .pre file and do 'display problem stats' to examine the numerical data.
    If you have additional questions on this, consider posting the log files of the CPLEX runs for the original and presolved model to this thread.
    #CPLEXOptimizers
    #DecisionOptimization


  • 9.  Re: Ill Conditioned Problem

    Posted Mon April 16, 2012 12:07 AM

    Originally posted by: Eumpfenbach


    First of all, thank you very much for the responses.

    I don't really have big-m constraints of the typical form, but I definitely have similar problems. I am trying to solve a linear-fractional program which I transform into an MIP. I have coefficients on the order of e9 with variables and coefficients that are less than one. Without presolving, even the LP relaxation does not even solve (or at least, it takes so long I cancel before it does). I see the "Parallel mode: deterministic" message and then nothing. It just stops there for hours. With presolving, it solves in a couple minutes.

    I appreciate the information about how to test the conditioning of my problem, but I think it is pretty clear that whatever Cplex is doing to fix my numerical problems, I need it. If I can utilize presolve, I don't really need to get to the VERY bottom of the problem. If I couldn't utilize presolve, then I would need to dig. I just need to figure out how to presolve it with cplex, write to a file, and then load that file into Matlab. I am trying to use the Matlab API and have my problem stored in MPS format.

    Now, when I tried to write out the presolved model, this is what I encountered:

    
    CPLEX> read mnl_v2.mps Selected objective sense:  MINIMIZE Selected objective  name:  R116407 Selected RHS        name:  B Selected bound      name:  BOUND Problem 
    'mnl_v2.mps' read. Read time =    0.37 sec. CPLEX> write mymodel.pre Primal unbounded due to dual bounds, variable 
    'C0001'. CPLEX Error  1101: Presolve determines problem is infeasible or unbounded. No file written. CPLEX>
    


    This is the exact same MPS file that I load into Matlab. There, cplex presolves and then solves the model. Here it declares it infeasible. I will leave it to the experts to try and interpret what is going on.

    I have attached my mps file in case anyone wants to test anything with it.

    Thanks...
    #CPLEXOptimizers
    #DecisionOptimization


  • 10.  Re: Ill Conditioned Problem

    Posted Mon April 16, 2012 02:16 AM

    Originally posted by: T_O


    > I have attached my mps file in case anyone wants to test anything with it.

    Where?
    #CPLEXOptimizers
    #DecisionOptimization


  • 11.  Re: Ill Conditioned Problem

    Posted Mon April 16, 2012 09:20 AM

    Originally posted by: Eumpfenbach


    Its a 21 MB file. Something must have gone wrong with the attachment. Trying again...
    #CPLEXOptimizers
    #DecisionOptimization


  • 12.  Re: Ill Conditioned Problem

    Posted Mon April 16, 2012 09:23 AM

    Originally posted by: Eumpfenbach


    Looks like it failed again. Email Eumpfenbach at wayne dot edu and I will send the mps file (if anyone wants to experiment with it).
    #CPLEXOptimizers
    #DecisionOptimization


  • 13.  Re: Ill Conditioned Problem

    Posted Mon April 16, 2012 09:44 AM

    Originally posted by: Eumpfenbach


    This should work:

    http://www.mediafire.com/?zff86wp19ryfekl
    #CPLEXOptimizers
    #DecisionOptimization


  • 14.  Re: Ill Conditioned Problem

    Posted Mon April 16, 2012 04:14 PM

    Originally posted by: SystemAdmin


    The LP relaxation of your problem is indeed unbounded, and this means that the MIP is infeasible or unbounded.

    To analyze the reason, I did the following in the interactive:
    1. Read in the model, optimize it - unbounded error.
    2. Change problem type to LP, optimize again - still unbounded.
    3. Write out the dual formulation ("write mnl.dua"), read it back in, optimize - infeasible.
    4. Run the conflict refiner ("conflict") to find a minimal infeasible subset - result is a single constraint and one bound.
    5. Display the conflict.
    Minimal conflict:    1 linear constraint(s)
                         1 lower bound(s)
                         0 upper bound(s)
    Conflict computation time =    0.02 sec.  Iterations = 1
     
    CPLEX> disp conf all
    Minimize
     obj:
    Subject To
     C0001: - 300000 R37661 >= 100000000
    Bounds
     The variable is >= 0.
    


    This conflict shows the infeasibility of the dualized model, i.e., the existence of an unbounded ray in the primal formulation.
    Indeed, if one looks at the original model, variable C0001 has a negative objective coefficient (in a minimization problem), an infinite upper bound, and increasing it will not make any constraint infeasible (the only coefficient is negative and appears in a smaller-or-equal constraint). Consequently, the unit vector with a 1 at the position of C0001 is an unbounded ray.
    Tobias
    #CPLEXOptimizers
    #DecisionOptimization


  • 15.  Re: Ill Conditioned Problem

    Posted Mon April 16, 2012 04:40 PM

    Originally posted by: Eumpfenbach


    Yikes. I made a very careless error in my posting. My model is a maximization problem. When I read the mps file into Matlab, I then multiply the objective by -1.

    When I correct this error, I am able to save the presolved model, load it back in, and write an mps file (for export to Matlab). I will test and see if this eliminates all of my numerical difficulties.

    Thanks.
    #CPLEXOptimizers
    #DecisionOptimization


  • 16.  Re: Ill Conditioned Problem

    Posted Mon April 16, 2012 05:16 PM

    Originally posted by: Eumpfenbach


    I am still seeing inconsistent numerical behavior. I use interactive cplex to write the presolved model, load it back into interactive cplex, then write an mps file.

    I solve this presolved mps file (both with Cplex/Matlab's presolve on and off, this doesn't make a difference), and it gives me a different objective function value than if I had just taken my original MPS file, loaded into Matlab, presolved, and then solved. The answers differ by a magnitude of 100.

    Unfortunately I do not know the optimal solution to my problem so I cannot determine which way is correct. Any thoughts on this behavior?
    #CPLEXOptimizers
    #DecisionOptimization


  • 17.  Re: Ill Conditioned Problem

    Posted Tue April 17, 2012 03:38 AM

    Originally posted by: SystemAdmin


    What do you mean by "different objective function value"? You are not able to solve this model to optimality, right? So, are you talking about the incumbent objective value or the global dual bound? And at which point do you measure them? After some time limit is exceeded?

    Tobias
    #CPLEXOptimizers
    #DecisionOptimization


  • 18.  Re: Ill Conditioned Problem

    Posted Tue April 17, 2012 09:09 AM

    Originally posted by: Eumpfenbach


    Note: this uses slightly different problem data than the file I posted on Mediafire. It is the same model with the same problems though.

    1) Original MPS file in Matlab, no presolve:

    
    Selected objective sense:  MINIMIZE Selected objective  name:  R116407 Selected RHS        name:  B Selected bound      name:  BOUND Parallel mode: deterministic, using up to 2 threads 
    
    for concurrent optimization.
    


    Then it just freezes. No iterations shown. I don't know if it will eventually solve but not anytime soon.

    2) Original MPS file in Matlab, with presolve:

    
    Selected objective sense:  MINIMIZE Selected objective  name:  R116407 Selected RHS        name:  B Selected bound      name:  BOUND Parallel mode: deterministic, using up to 2 threads 
    
    for concurrent optimization. Tried aggregator 1 time. LP Presolve eliminated 111 rows and 101 columns. Reduced LP has 116295 rows, 55318 columns, and 525009 nonzeros. Using devex.   Iteration log . . . Iteration:     1    Objective     =    2787758391.255035   ----Removing a bunch of intermediate iterations 
    
    for readability ----------   Markowitz threshold set to 0.5 Switched to devex. Iteration:    77    Objective     =    2787758391.254920   obj =   2.7878e+009
    


    3) Original MPS presolved in interactive Cplex, saved as MPS, loaded into Matlab, Matlab/cplex Presolve off

    
    >> miptest Selected objective sense:  MINIMIZE Selected objective  name:  obj Selected RHS        name:  rhs Selected bound      name:  bnd Parallel mode: deterministic, using up to 2 threads 
    
    for concurrent optimization.   obj =   2.5456e+011
    


    4) Original MPS presolved in interactive Cplex, saved as MPS, loaded into Matlab, Matlab/cplex Presolve on

    
    Selected objective sense:  MINIMIZE Selected objective  name:  obj Selected RHS        name:  rhs Selected bound      name:  bnd Parallel mode: deterministic, using up to 2 threads 
    
    for concurrent optimization. Tried aggregator 1 time. LP Presolve eliminated 78647 rows and 88 columns. Reduced LP has 37648 rows, 55230 columns, and 367440 nonzeros.   obj =   2.5456e+011
    

    #CPLEXOptimizers
    #DecisionOptimization


  • 19.  Re: Ill Conditioned Problem

    Posted Tue April 17, 2012 09:55 AM

    Originally posted by: Eumpfenbach


    And anticipating a possible question, I did change my problem type to LP before I presolved in interactive cplex. Everything I am showing results for is the initial LP relaxation of the MIP.
    #CPLEXOptimizers
    #DecisionOptimization


  • 20.  Re: Ill Conditioned Problem

    Posted Wed April 18, 2012 09:31 AM

    Originally posted by: Eumpfenbach


    I am sure someone will get to my other question but in the meantime, is it possible to set "aggressive scaling" in cplex for Matlab? I cannot find the parameter in the guide or by creating an object and looking through cplex.Param.
    #CPLEXOptimizers
    #DecisionOptimization


  • 21.  Re: Ill Conditioned Problem

    Posted Wed April 18, 2012 07:06 PM

    Originally posted by: Eumpfenbach


    Another thought. I did some googling and found this:

    http://groups.google.com/group/ampl/browse_thread/thread/7e24843742d14aa9/66d7db390a5ece1c?lnk=gst&q=mps#66d7db390a5ece1c

    Per Paul's statement, what are the chances that the conversion to .mps file is causing problems? This is maybe why using interactive cplex to presolve is resulting in answers that are drastically superoptimal (ie because I am writing an mps, reading it, writing another mps, and then solving).

    I only use an MPS file because I first learned modeling with AMPL. I am assuming I could convert it into ILOG language relatively easy. I am just looking for confirmation that this would have value before I put in the effort. Please let me know what the experts think...
    #CPLEXOptimizers
    #DecisionOptimization


  • 22.  Re: Ill Conditioned Problem

    Posted Fri April 27, 2012 03:27 AM

    Originally posted by: SystemAdmin


    Is it possible that you use the .sav file format instead .mps? The .sav format is a binary format and preserves every single bit of model data.

    Tobias
    #CPLEXOptimizers
    #DecisionOptimization


  • 23.  Re: Ill Conditioned Problem

    Posted Sat April 28, 2012 06:18 PM

    Originally posted by: Eumpfenbach


    I transferred my code to ILOG and am using a .lp format. I no longer see poor results (meaning that when I presolve using interactive cplex and import into matlab, the problem isn't "corrupted").

    Thanks. I still have some trouble with what I am trying to do but will use an appropriate thread for the other questions.
    #CPLEXOptimizers
    #DecisionOptimization


  • 24.  Re: Ill Conditioned Problem

    Posted Mon April 30, 2012 08:21 PM

    Originally posted by: EdKlotz


    I don't have any answers for the most recent questions in the thread. But, regarding the general question about the ill conditioning in the problem,
    I had a quick look at the model and have a few comments.

    1) Setting CPLEX's scaling parameter significantly improved the basis condition
    numbers during the MIP solve. This can be seen by using the recently added
    MIP kappa feature that accumulates statistics on the basis condition numbers during
    a MIP solve ('set mip strategy kappastats 1' (or 2) when using interactive CPLEX).

    2) Regarding the higher level of ill conditioning when presolve is turned off,
    consider a constraint like the following in the model:

    
    R37660:  103.28 C0102 + 103.28 C0103 + 103.28 C0104 + 102.13 C0105 + 102.13 C0106 + 102.13 C0107 + 114.49 C0108 + 114.49 C0109 + 114.49 C0110 + 113.315 C0111 + 113.315 C0112 + 113.315 C0113   ...   + 128.9702 C36975 + 110.128 C36976 + 128.9702 C36977 + 55.064 C36978 + 64.4851 C36979 + 55.064 C36980 + 64.4851 C36981 + 55.064 C36982 + 64.4851 C36983 <= 1000000000000
    


    This large right hand side value, particularly when coupled with potentially
    high basis condition numbers, could introduce significant round off error
    into various calculations, leading to inconsistent results. CPLEX's
    presolve managed to delete this constraint, which could improve the numerics
    significantly. This does not surprise me; given the modest coefficients of
    the variables in this constraint, if CPLEX can derive any sort of modest finite
    upper bounds on this variable, this constraint will be redundant. Are you sure
    you really need this constraint in the model? If you could remove it yourself,
    that might help.

    3) I also notice a wide range of objective coefficients in the model. Also,
    based on some MIP runs, it really looks like the coefficients of the largest
    order of magnitude dominate until their variables can all be set to 0. In such
    cases, we often recommend using a goal programming approach where you first
    optimize over the really large objective coefficients, then constrain them based
    on that result. This will enable you to avoid using such large coefficients,
    which can create numerical problems that you could otherwise avoid.
    More information on this can be found at
    http://www-01.ibm.com/support/docview.wss?uid=swg21399935
    Hopefully one or more of these will help with your original question.
    #CPLEXOptimizers
    #DecisionOptimization


  • 25.  Re: Ill Conditioned Problem

    Posted Tue May 01, 2012 08:35 AM

    Originally posted by: Eumpfenbach


    First of all, thank you very much for looking at my problem in so much detail!

    1) I want to use Cplex/Matlab. I still haven't found the scaling parameter control in this API.

    2) this constraint is sort of necessary. I purposely set the right hand side so large that the constraint will never be binding. Later, on sensitivity runs, I will reduce it and observe the effect on the objective. I did reduce it by several orders of magnitude without it becoming binding, improving the conditioning.

    3) I have not considered watching which variables cplex attacks first and trying your approach of creating constraints with them. Given that there are 2 problems here (the first was about numerical accuracy with mps files and the other is how to solve efficiently), I can't be sure that the solution you saw is a proper one. I do have a large difference in the magnitude of some of my coefficients (fixed costs for building a manufacturing plant vs. unit costs to produce products). I will have to think about this and experiment.

    Thanks again!
    #CPLEXOptimizers
    #DecisionOptimization


  • 26.  Re: Ill Conditioned Problem

    Posted Tue May 01, 2012 07:15 PM

    Originally posted by: EdKlotz


    > Eumpfenbach wrote:
    > First of all, thank you very much for looking at my problem in so much detail!
    >
    > 1) I want to use Cplex/Matlab. I still haven't found the scaling parameter control in this API.

    MATLAB allows you to set every CPLEX parameter. Specifics depend on whether
    you are using the toolbox functions or the object oriented CPLEX Class API.
    If the latter, look at the description of the Param property in the docs for
    setting arbitrary parameters. Based on that, if you had an instance of the
    Cplex Class called cpx, you would set scaling to 1 with

    cpx.Param.read.scale.Cur = 1;

    More generally, look up the parameter of interest in the parameter reference
    manual, then note that the MATLAB convention for parameter naming uses the
    same identifier as interactive CPLEX, with a '.' between each word.

    If you are using the toolbox functions, the naming convention also resembles
    interactive CPLEX, but within the framework of the cplexoptimset function.
    As the manual describes, you would do something like

    opt = cplexoptimset('cplex');
    opt.read.scale=1;

    < other non default parameter settings >
    then pass opt to the optimization toolkit function you call.
    >
    > 2) this constraint is sort of necessary. I purposely set the right hand side so large that the constraint will never be binding. Later, on sensitivity runs, I will reduce it and observe the effect on the objective. I did reduce it by several orders of magnitude without it becoming binding, improving the conditioning.

    OK, but you can accomplish this in other ways that don't create the numerical
    trouble. So, let's say the maximum right hand side that can make this constraint
    binding is b. You take your original constraint

    ax <= b

    and making it nonbinding by increasing the right hand side;

    ax <= b + M

    Now, b + M is very large, and has the potential to create numerical troubles,
    particularly when presolve is disabled. But, instead, you could make it non
    binding by reformulating by introducing a slack variable and adjusting the
    slack bounds

    ax + s <= b
    -infinity <= s <= +infinity

    When you want to enforce this constraint, change the upper and lower bounds on
    s to 0. This would work better numerically.
    >
    > 3) I have not considered watching which variables cplex attacks first and trying your approach of creating constraints with them. Given that there are 2 problems here (the first was about numerical accuracy with mps files and the other is how to solve efficiently), I can't be sure that the solution you saw is a proper one. I do have a large difference in the magnitude of some of my coefficients (fixed costs for building a manufacturing plant vs. unit costs to produce products). I will have to think about this and experiment.

    Given your model description, in terms of MIP performance you probably want
    to give higher priority to the decision variables associated with paying the
    fixed cost to build a particular plant. After all, until you decide which
    plants to build and know their production capabilities, deciding which products
    to produce doesn't make much sense. On the LP side, it sounds like you can't
    optimize over just the largest coefficients and constrain the resulting objective at its optimal value, but you might be able to constrain it within
    some modest percentage of optimality. That too would enable you to use the goal
    programming approach.

    >
    > Thanks again!
    #CPLEXOptimizers
    #DecisionOptimization