Originally posted by: PingLiu

Dear All,

I am working on an optimization problem, where we are trying to minimize some indicators like energy usage, energy cost, CO2 emission. In this problem, we have a bunch of energy conversion technologies for electricity and thermal purpose, such as heat pump, boiler, chiller, etc.. We are trying to model it with a one year time period. the current time step is one hour.

We have preselected the  candidate technologies to exclude those we don't want to study so that the problem size could be reduced with a limited candidate technologies. In the current case study, we only analyze the electric chiller and heat pump to supply the cooling load, while power grid will supply the electricity for all electric loads. There are binary variables regarding installation decisions of technologies and continuous variables related to installation capacity and hourly operational decisions.

For small cases, Python works well. But if we consider longer time period. then it would fail due to the memory usage issues. We have tested several case studies to check the memory use for different time period, including 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours each day, as well as 4) 30 days with 24 hours each day. The first 3 cases are feasible while the last case gives out the memory error.

When we are testing the forth case, the memory error comes out while creating the inequality constraints. The problem size is 1) Aeq: 12 * 26, Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq: 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.

The solver is CPLEX (academic). It turns out that the solver is taking a lot of memory as you can see in the memory test report. for the first three cases, different memory usage is observed, and it grows up dramatically with the increase of the time period. 1) solver memory usage: 25.6 MB, 2) 19.5 MB; 3) solver memory usage: 830.0742 MB.

How can I reduce the memory usage of CPLEX solver in Python? Why would the memory decrease for the solver with the 24 hours in one day case compared with the case 1?

Thanks in advance,
Ping

Sorry, you description is rather confusing. You say that CPLEX goes out of memory when creating Aineq. However, according to the attached PDF files you code looks like this:

cpx = cplex.Cplex()
cpx.read("file.lp")
...
cpx.solve()

So you read the model into CPLEX from a file instead of building it up in memory? In that case I don't see how CPLEX could go out of memory while building Aineq (building Aineq is unrelated to CPLEX).

Also, could it be that you are building Aeq and Aineq as a dense rather than a sparse matrix?

Originally posted by: PingLiu

Hi, Daniel,

in case 4, the memory is used up when creating Aineq, which happens right before we start to solve the problem with CPLEX because at this time we have not created the problem yet. so this memory error has nothing to do with the CPLEX solver.

the following code is commented out, which does not work in our case.

cpx = cplex.Cplex()
cpx.read("file.lp")
...
cpx.solve()

But for the first 3 cases with different time period, the CPLEX shows varying performance. when it is only 2 hr, cplex memory usage is 15.6 MB; when it is 24 hr, then it decreases to 19.5 MB. I am not sure why the memory usage of cplex solver goes down when the problem size goes from  Aeq(12*26) and Aineq(30*26) to Aeq(144*268) and Aineq(316*268).

Another observation is when we tested the case with 10 days, and 24 hr each day, the memory usage is 192 MB. For 20 days, 24 hr each day, the memory usage of cplex is 719MB. So I wonder how I can reduce the memory usage of the CPLEX solver since I need to solve a much longer time period (e.g. 12 months, 30 days each month, 24 hours each day). Can I make any change to the default parameters of CPLEX API for Python so that the memory usage could be reduced?

Is the memory consumption you report for 10 and 20 days only for the model itself or for the whole solution process? What is the number of non-zeros in either case?

Originally posted by: PingLiu

The memory is only for the CPLEX solver, not for the whole problem modeling.

In terms of the number of non-zero elements of Aeq and Aineq, it seems that the sparsity of those matrix is very high.

1) In order to reduce the memory usage in creating the problem, may I use scipy.sparse to create sparse matrix for Aeq and Aineq in the format of csr_matrix for the MILP problem? Does CPLEX accept sparse matrix as input?

p = MILP(f=f, lb=lb, ub=ub, A=Aineq, b=bineq, Aeq=Aeq, beq=beq, intVars=intVars, goal='min')

2) Regarding solver memory usage, what parameters can I change for CPLEX solver when it is called as following?

r = p.solve('lpSolve')

You may want to take a look at this chapter in the user manual to learn what parameters can be changed to modify memory usage.