Originally posted by: JPRyan

I'm getting a weird error "CPLEX Error  5012: Q is not symmetric."  adding absolute values to cost functions when building a toy (two variable) problem in DocPlex.  I attached the python file for the full script.

 

My full cost function has three parts c_costs, t_costs, and f_costs and one constraint:

    mdl = AdvModel()

    current = Series([0.6, 0.4], ['a1', 'a2'])

    var = mdl.continuous_var_list(len(current), lb=0, ub=1, name='Target')
    target = Series(var, current.index, name='Target')

    # Cost Code Below

    mdl.minimize(c_costs + t_costs + f_costs)

    mdl.add_constraint(mdl.sum(target) == 1)

 

Where c_costs and f_costs are both quadratic:

    c_costs =  r * mdl.quad_matrix_sum(covar, target)

    f_costs = h * mdl.sumsq(target - optimal)

and r&h are constants greater than zero.

 

Interestingly, minimizing over just these quadratic costs works fine as covar is symmetric so Q must be symmetric

    mdl.minimize(c_costs + f_costs)

    mdl.solve()

 

When I add the t_costs with absolute values (maybe this could be written better but there is no vector version of abs() in CPlex??) it fails with the error "CPLEX Error  5012: Q is not symmetric."

    abs_series = Series([mdl.abs(target[i] - current[i]) for i in current.index], current.index)
    t_costs = linear_costs.dot(abs_series)

    mdl.minimize(c_costs + t_costs + f_costs)

The crazy thing is these costs are _linear_ so they shouldn't affect the Q matrix!

 

Even weirder... both of the following work combined with the absolute value term!  But the combination of all three fails!!

    mdl.minimize( t_costs + f_costs)
    mdl.solve()

    mdl.minimize( c_costs + t_costs )
    mdl.solve()

 

 

 

 

 

 

 

 

 

#CPLEXOptimizers
#DecisionOptimization

What version of CPLEX and docplex do you use? When I run your script here with CPLEX 12.8 and docplex 2.4.61 then it does not raise any exception. It just prints

integer optimal solution
objective: 0.000
  Target_0=0.525
  Target_1=0.475

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: JPRyan

Interesting:

docplex '2.7.113'

CPlex "12.8.0.0"

 

Thanks for giving it a run.  I've updated my Anaconda as well with no luck.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: JPRyan

Is there a way to get to Q directly so I can see the asymmetry?

 

Also, something even crazier.  If I take the t_costs out of the minimize it still fails, but if I take out the t_costs and comment out lines 24-25 with the mdl.abs it works.

#CPLEXOptimizers
#DecisionOptimization

I have managed to reproduce this here but it seems rather random. It smells like a bug in docplex, we are investigating.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: JPRyan

thank you.

#CPLEXOptimizers
#DecisionOptimization

It seems there is a problem in the way how docplex handles quadratic expressions. Can you please try this workaround:

Define function

def clean_expr(expr):
    """Work around a problem in handling of quadratic expressions.
    The function explicitly collects terms with the same variables.
    """
    quadterms = dict()
    idxmap = dict()
    for v1, v2, a in expr.iter_quad_triplets():
        idxmap[v1._index] = v1
        idxmap[v2._index] = v2
        key = (min(v1._index, v2._index), max(v1._index, v2._index))
        quadterms[key] = quadterms.get(key, 0.0) + a
    lin = expr.linear_part # includes the constant
    return lin + sum([idxmap[k[0]] * idxmap[k[1]] * quadterms[k] for k in sorted(quadterms)])

When registering the objective function, use

mdl.minimize(clean_expr(your_objective_function))

Probably also double-check that results are reasonable. One way to check the model looks as expected is to export via mdl.export_as_lp() and then look at the exported LP file. That file should be more or less human readable.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: JPRyan

Thanks.  I'll give this a shot.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: JPRyan

Revisited this problem a few months later for another project. 

 

If it is at all helpful to your programmers the original program's issue can be solved with just reordering the addition:

 

mdl.minimize(c_costs  + t_costs + f_costs) # FAILS!

 

mdl.minimize( t_costs + f_costs + c_costs) # WORKS!

 

So much for addition being commutative :) 

 

It's pretty crazy and I haven't fully tested it but it seems to give reasonable results.  Is there a way I can track the bug fix?

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: HUGUES JUILLE

This bug has been fixed. It concerns an issue in the calculation of coefficients in quadratic expressions.

It will be delivered in the next release of docplex.
No precise date has been set for this release yet but it should be made available in the next couple of months.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: JPRyan

Great.  Many thanks.

#CPLEXOptimizers
#DecisionOptimization

Originally posted by: JPRyan

Checking back on this bug.  I've been using Daniel's "clean_expr" function below but it does slow down the calculations some.  Has the fix been pushed into production?

#CPLEXOptimizers
#DecisionOptimization

Yes, this should be fixed and recent versions of docplex should no longer have this problem.

#CPLEXOptimizers
#DecisionOptimization