Decision Optimization

Hello,

I need to solve a large scale MILP with a sparse constraint matrix using CPLEX within C++.
In my initial implementation, I do not exploit the sparsity of the constraint matrix, and feed all  zero and non-zero elements of the constraint matrix to CPLEX. However, the problem is so large that it cannot even be load in memory.
So my question is: If I filter out the zero elements, and only feed the nonzero elements of the constraint matrix to CPLEX, will that help with memory issue?
If this helps, please share any advice/insight that you may have on the implementation side since this is a large problem with different classes of variables. I was running out of English alphabets when I was writing the formulation :)

Thanks

------------------------------
Amin
------------------------------

#DecisionOptimization

It's hard to answer without more details. It will certainly help but I don't think it will remove the out of memory.
When dealing with huge models, you have to be careful with the concert operators, functions you use (do you have some code block which is consuming all the memory?), and maybe normalize the linear expressions yourself before pushing to concert (try top put all variables only once in the linear expressions to avoid to have an enormous linked list of variables, which will be simplified via normalization): duplicates and memory used by normalization can have a huge cost.
Also instanciate variables only if they are really part of the model?

------------------------------
Vincent Beraudier
------------------------------

Original Message:
Sent: Sat December 24, 2022 01:51 AM
From: Amin
Subject: spare constraint matrix

Hello,

I need to solve a large scale MILP with a sparse constraint matrix using CPLEX within C++.
In my initial implementation, I do not exploit the sparsity of the constraint matrix, and feed all  zero and non-zero elements of the constraint matrix to CPLEX. However, the problem is so large that it cannot even be load in memory.
So my question is: If I filter out the zero elements, and only feed the nonzero elements of the constraint matrix to CPLEX, will that help with memory issue?
If this helps, please share any advice/insight that you may have on the implementation side since this is a large problem with different classes of variables. I was running out of English alphabets when I was writing the formulation :)

Thanks

------------------------------
Amin
------------------------------

#DecisionOptimization

Hi Amin, 

just to get a sense of the issue: how many constraints, variables, and non-0 coefficient are in your model ?

------------------------------
Stefano Gliozzi
------------------------------

Original Message:
Sent: Sat December 24, 2022 01:51 AM
From: Amin
Subject: spare constraint matrix

Hello,

I need to solve a large scale MILP with a sparse constraint matrix using CPLEX within C++.
In my initial implementation, I do not exploit the sparsity of the constraint matrix, and feed all  zero and non-zero elements of the constraint matrix to CPLEX. However, the problem is so large that it cannot even be load in memory.
So my question is: If I filter out the zero elements, and only feed the nonzero elements of the constraint matrix to CPLEX, will that help with memory issue?
If this helps, please share any advice/insight that you may have on the implementation side since this is a large problem with different classes of variables. I was running out of English alphabets when I was writing the formulation :)

Thanks

------------------------------
Amin
------------------------------

#DecisionOptimization

Hi Stefano,
around 100 million or more variables. I need to check the number of constraints

------------------------------
Amin
------------------------------

Original Message:
Sent: Tue January 03, 2023 10:42 AM
From: Stefano Gliozzi
Subject: spare constraint matrix

Hi Amin, 

just to get a sense of the issue: how many constraints, variables, and non-0 coefficient are in your model ?

------------------------------
Stefano Gliozzi

Original Message:
Sent: Sat December 24, 2022 01:51 AM
From: Amin
Subject: spare constraint matrix

Hello,

I need to solve a large scale MILP with a sparse constraint matrix using CPLEX within C++.
In my initial implementation, I do not exploit the sparsity of the constraint matrix, and feed all  zero and non-zero elements of the constraint matrix to CPLEX. However, the problem is so large that it cannot even be load in memory.
So my question is: If I filter out the zero elements, and only feed the nonzero elements of the constraint matrix to CPLEX, will that help with memory issue?
If this helps, please share any advice/insight that you may have on the implementation side since this is a large problem with different classes of variables. I was running out of English alphabets when I was writing the formulation :)

Thanks

------------------------------
Amin
------------------------------
#DecisionOptimization

I ment: how many constraint ? How many variables ? How many non zero coefficient (I assume these are in the order of magnitude > 100M). 

Also, looking at the details of the calls to the concert library would be useful to help.

Finally, which arr the characteristics oc the machine you are using ? (OS, cores, memory)

​

------------------------------
Stefano Gliozzi
------------------------------

Original Message:
Sent: Sat January 07, 2023 11:35 PM
From: Amin
Subject: spare constraint matrix

Hi Stefano,
around 100 million or more variables. I need to check the number of constraints

------------------------------
Amin

Original Message:
Sent: Tue January 03, 2023 10:42 AM
From: Stefano Gliozzi
Subject: spare constraint matrix

Hi Amin, 

just to get a sense of the issue: how many constraints, variables, and non-0 coefficient are in your model ?

------------------------------
Stefano Gliozzi

Original Message:
Sent: Sat December 24, 2022 01:51 AM
From: Amin
Subject: spare constraint matrix

Hello,

I need to solve a large scale MILP with a sparse constraint matrix using CPLEX within C++.
In my initial implementation, I do not exploit the sparsity of the constraint matrix, and feed all  zero and non-zero elements of the constraint matrix to CPLEX. However, the problem is so large that it cannot even be load in memory.
So my question is: If I filter out the zero elements, and only feed the nonzero elements of the constraint matrix to CPLEX, will that help with memory issue?
If this helps, please share any advice/insight that you may have on the implementation side since this is a large problem with different classes of variables. I was running out of English alphabets when I was writing the formulation :)

Thanks

------------------------------
Amin
------------------------------
#DecisionOptimization