Originally posted by: SystemAdmin

Hi

I'm implementing a column geneartion based algorithm as described in:

https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14939755&#14939755

As described this requires that i have a large number of IloOplModel objects existing at the same time, which gives me a speed up, but it really eats memory. The flow for creating the objects is as:
var subOplArrays = new Array(numVessels);
var subCplexArrays = new Array(numVessels);

...

for(var ves in MasterDataElmts.Vessels) {
subCplexArrays[i] = new IloCplex();
subOplArrays[i] = new IloOplModel(subDef, subCplexArrays[i]);

...
subOplArrays[i].addDataSource(subData);
subOplArrays[i].generate();
subOplArrays[i].settings.mainEndEnabled = true;
subCplexArrays[i].epgap = subProblemGap;
MasterDataElmts.bunkerPlanUniqueId = MasterDataElmts.bunkerPlanUniqueId +1;
writeln("Model object created for vessel: "+i);
i++;
}

this setup restricts me to have at most 100 subproblems due to mem issues, as each model takes about 13 MB. And i'd really like to go much higher.

So my question, what can i do to reduce memery load ? I've tried to make sure that subData only holds the absolutely necessary, and i cant reduce the model it self more than ive done.

Looking forward to any hints!

Christian
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Why don't you reuse the same sub model objects?
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

subData is different for each sub problem and i would have to recreate the whole matrix, objective and rhs. So it would be better to destroy and recreate.
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

You can call generate() for each sub problem instance (on the same model object), and only update the input data... I think I don't understand what is your problem, could you please clarify
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Sorry i'll elaborate some more.

Originally i did as you suggested, calling generate for each subproblem in each iteration.

But with this approach the "generate()" call ends up using 90 % or more of the computational time, as each of my subproblems are usually easily solved. So the whole idea of this different approach is to avoid the generate calls, by simply updating the changed dual values in the subproblems, as all other values are the same from iteration to iteration.

The problem is now that all these subproblems eat all the mem, so it does scale to sufficiently large instances.
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

In the code above I see a call to sub problem generate() inside the loop, so how did you reduce the number of calls there?
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

As stated, i've only posted the part of the code creating the object. If you looked at the post of above link you'll get an idea of what the modyfing and solving code looks like.

Cheers, Chrsitian
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

I don't understand your question in 5 minutes given all the data. There is a good chance I am not smart enough :), hope someone helps you.
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Thanks ;) Sorry I cant explain better. But basically i need each IloOplModel to take up much less mem than its doing. So I'm looking for any way to do this ? Maybe it can be stored in sparse form ?
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

May be after completing the sub model run, store the required result somewhere and drop/reuse the big object.
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

No. That would resemble the original approach.
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Christian,

One way could be to decrease the number of CPLEX instance.
I can see that you create an CPLEX instance for each subproblem.

for(var ves in MasterDataElmts.Vessels) 
{ ..... subCplexArrays[i] = 

new IloCplex(); ....
}

You can use only one instance of CPLEX.
If your model solve fast on only 1 or 2 threads you can have several CPLEX instance with a fix number of thread according the number of core of your machine.

then you can do something like:

CplexArrays[i] = 

new Array() 

for(var i in 1:nbcore) 
{CplexArrays[i] = 

new  IloCplex();
}   

for(var ves in MasterDataElmts.Vessels) 
{ ..... subOplArrays[i] = 

new IloOplModel(subDef, CplexArrays[i%CplexArrays.length]); ....
}

you can find information about loding Model structure in the CPLEX help here:
Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.ide.help/OPL_Studio/opllanguser/topics/uss_langtut_fc_multiple_search_6.html

if you are part of IBM Academic Initiative you can access to teaching material about CPLEX and OPL.
the CPLEX_OPL.zip is for you especially the lesson 18: Performance Tuning
Linkhttps://www14.software.ibm.com/webapp/iwm/web/reg/download.do?source=ai-course-bao&S_PKG=2VPS0&lang=en#
I hope it helps

best regards
Stephen Hall
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Again,

you can try to reduce the number of thread of CPLEX. It will reduce the memory usage.

the second point is to isolate a subproblem in .mod and .dat. After you can run it and analyze it in the console. You would be able to find where the memory is used in the subproblem and reduce it size by modifying the model. Tuning example are:

• use tuples instead of array
• use Range instead of Set
• use internal data instead of external data
• use integer id key instead of string
• ..... find other hint in CPLEX_OPL training book from Academic Initiative. see previous post

after modification you can replicate the changes into your original model.

let me know what are the results.

Stephen Hall
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Stephen,

I'm not sure i can do this, as i update the objective in each cplex subproblem instance, in each iteration, as:

subCplexArrays[j].setObjCoef( subOplArrays[j].purchaseContract[c], c.price + dualAdjust);

as i cant use:

subOplArrays[j].getObjective().setCoef(subOplArrays[j].purchaseContract[c], c.price + dualAdjust);

as my problem is a MIP (and thus non linear). So i would end up calling Generate() in each iteration again, to ensure that model and cplex matched.
I have reduced number of threads to 1.
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Christian,

Sorry you are right for the use of only one instance of CPLEX as when you generate you populate the CPLEX instance with one problem.

But you can still try:

• force the number of CPLEX Thread to 1 Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.cplex.help/CPLEX/Parameters_reference/topics/Threads.html
• isolate one subproblem for one iteration and see where the memory is consumed.

There is an option to tell CPLEX to limit memory usage if possible: Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.cplex.help/CPLEX/Parameters_reference/topics/MemoryEmphasis.html

look in the doc for some hint

• Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.ide.help/OPL_Studio/opllanguser/topics/uss_perf_and_memoryusage_1.html
• Linkftp://public.dhe.ibm.com/common/ssi/sa/wh/n/wsw14059usen/WSW14059USEN.PDF

#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Stephen,

Thnanks for your constructive answers.

I am already running with 1 thread and i've set the reduce memoty switch (in .opl file, i guess thats sufficient?). These helped a bit.

But its not my perception that there is a massive memory usage in cplex itself. When stepping through the code, its the "generate()" calls of the ilog model that eats the majority of the mem, so i think the solution should be found in opl somehow.

Thanks, Chr
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Christian,

I guess you are right and we are back to your 1st assessment and tittle of this thread; minimizing model memory

For that you need to investigate were the memory is used in your Submodel.
1. isolate one iteration of a subproblem in .mod and .dat (in your master run you can extract the internal and external data in a .dat)
2. Run the subproblem
3. use the profiler to see where the memory is used. Tutorial of the profiler here: Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplide/topics/opl_ide_profil.html

for example the use of tuples normally decrease the memory usage.
the internal data of an objet are loaded in memory only if needed, while external data are all loaded from start.

I hope you will find some solution to decrease the model size.
keep me posted

MVH.
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Stephen,

I've added a profiler dump of the full problem (master and subproblem), but it lacks details to really unxderstand the issue.

So you're right that I need to: "isolate one iteration of a subproblem in .mod and .dat (in your master run you can extract the internal and external data in a .dat)", but how to ? I've looked through the help and documentation, but hasnt been able to find any ?

Thanks, Christian
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Christian,
You can have a look at this topic:Performance and memory usage
http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/opllanguser/topics/opl_languser_perf.html

You have already the .mod of your subproblem in your project.
the only thing that you need is the .dat. below is how to do it:
CPLEX Help:http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/refjsopl/html/IloOplModel.html

if (

true) 
{ 
//dump OPL data source var dumpFileName = 
"pscextdata.dat"; writeln(
"dumping data to : ", dumpFileName); var edf= 

new IloOplOutputFile(dumpFileName); var ed=thisOplModel.printExternalData(
"PSC data"); edf.write(ed); edf.close(); 
}

you have the same for internal Data : "printExternalData("title")"
Def of Internal and External Data http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplinterfaces/topics/opl_interfaces_work_memory.html

I have seen another way to control save the entire model:
"exportModel(path)" Exports the CPLEX model to a file.
"importModel(path)" Import the CPLEX model from a file.
CPLEX help: http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/refjsopl/html/IloCplex.html
I don't know if it would be faster then generate() but it is an option.
good luck
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Stephen,

Thanks, It worked. I succesfully exported the internal and external data to .Dat files. Following i've been able to run the subproblems alone and review the profiling, pls refer to attached for details.

But im not able to get much value out of this, the Mem usage is reported at 6.7 MB, a bit less than my experience from the whole program execution, but comparable. Going through the high mem users of these 6.7 i see: "CPLEX MIP Optimization", EXTRACT and LOAD_MODEL posts. But none of these seem to indicate that my model is mem-inefficiently constructed or I'm putting too much (unnneeded) data into it ? So am i looking at it the right way ? I cant find the insight that should help me lower the mem usage.

I still think it should be possible to reduce mem usage, if you look at the tab "Engine Log", you'll see that the model is quite small, only 3151 non-zeros, i've cant see why this should take up 6.7 MB ?

Cheers, Christian
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Christian,

good new you were able to run the subproblem alone.

But you missed to attache the file, please send it
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Again,

from your file I can see that some memory are allocated to EXTRACT, and I am suppose is to extract some constraints.

you can avoid that by:
1: not labeling the constraint
2: not extracting the data for the profiler http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplide/topics/opl_ide_profil_examin_turn.html
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

Hi Stephen,

I've tried this. Did very little to change mem consumption. Or actually nothing ;) I can scale to the same problem sizes with and without the profiling, although the mem consumption is a little bit less

Thanks, Christian
#DecisionOptimization
#OPLusingCPLEXOptimizer

Originally posted by: SystemAdmin

I've talked with Stephen and Alex Fleischer and they've resolved my issue. I ended up:

Having one subproblem at a time, creating the models when needed, solving and destroying afterwards. With a lean model and using the cmd version as "oplrun -deploy" the generate() calls are fast and each iteration runs smoothly without taking up much mem.
Thanks for the help.

Christian
#DecisionOptimization
#OPLusingCPLEXOptimizer