Originally posted by: Perth2

Hi. I have a MIP model as an IloCplex object with millons of IloIntVar variables in it. After solving the model, I can access the solution with these IloCplex methods:

• double getValue(IloNumVar var) - Returns the solution value for a variable.
• double[] getValues(IloNumVar[] var) - Returns solution values for an array of variables.

However, in my case, only very, very few of these variables are non-zero. Using the methods above, I will have to iterate through the millions of variables to find the very few ones that are non-zero. Since I have to do this many times, this degrades performance of my model. Is there a way to get the array of non-zero IloIntVar variables directly, without having to iterate?

Thanks, Perth2

No, there is no API to just get the non-zero values. After all, CPLEX would have to do the exact same scan that you do.

Note however that it should be much faster to get all values in one shot using getValues(IloNumVar[]) than to query individual values in a loop using getValue(IloNumVar).

Originally posted by: Perth2

Thanks for your answer, Daniel.

1. If CPLEX keeps track of what is in the basis, this operation would be possible without a scan. It would seem to me as a natural thing when solving LPs to keep track of what is in the basis. So if CPLEX does not do this, then why not?

2. It seems that it is possible through the C API in a callback context to query the basis, see this post: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014436225 This is done through a call to CPXgetbhead(). However, I can not find the same functionality through the Java API. Are some parts of CPLEX only accessible through the C API?

3. Can you tell me why it would be much faster to get all values and then loop through them to find non-zero elements, instead of looping and getting each variable value individually? Is CPLEX searching for the individual variable when using getValue(IloNumVar)?

1. There are two problems with using the basis to identify the nonzero variables. First, variables with finite upper bounds (or nonzero lower bounds) may be nonbasic but nonzero. Second, CPLEX has the basis for the problem after preprocessing. Not all the original variables survive preprocessing, and I would not be shocked if some variables in the model CPLEX is actually solving are inear combinations of original variables. So CPLEX will have to reverse the preprocessing transformations to find which of your variables are nonzero.

2. Yes, some of the low-level access stuff is available only through the C API. Even C++ users are out of luck.

3. For one thing, there's some overhead in setting up function calls and processing their results. You pay that "fixed cost" each time when you iterate over individual variables.

If you are using Java 8, you may be able to speed things up a bit using streams. Use an IntStream of indices, filter it by keeping indices of nonzero entries (remember to allow for some rounding error -- treat x as zero if |x| < epsilon), collect the survivors in a List, then iterate over that list to match nonzero values to their variables. You should be able to use parallel streams to speed this up (if you have multiple cores/processors).

On second thought, the Java 8 streams idea may be a nonstarter. I just ran a little experiment, fetching nonzero values for a 100x100 assignment problem. I ran it on a quad-core PC using CPLEX 12.6.1. Calling getValues() was somewhere around 5x to 7x faster than calling getValue() repeatedly. I tried different stream approaches (after calling getValues()), and they were consistently about 40x to 50x slower than calling getValues() and then just looping through the returned vector. This was true whether I used a single stream or parallel streams.

I'm new to streams, so I might be screwing the pooch here, but it looks like getValues() and then a for loop is the fastest route.

Originally posted by: Perth2

Thank you, Daniel and Paul, you have been a great help.