Originally posted by: merda30esimi

Dear All,

I am trying to solve an MIP Constraint Optimisation Problem, representing an Electrical Power System. Because of its large scale we has either computation speed or Out of memory issues. We therfore move from a 4cores to an 8cores and finally to a 12cores computer with 24GB RAM.

Now we can handle our 'reduced model' with 24 GB, reaching up to 97% of memory usage.

However, despite the increase number of cores we noticed different behavior depending on the computation phase. Here below a portion of the log file to explain the matter

1) Phase1: Cores usage: 12% with 8 cores, 8% with 12 cores (1 core only working)
.........................
Probing time = 0.20 sec.
Tried aggregator 2 times.
MIP Presolve eliminated 586 rows and 1359 columns.
MIP Presolve modified 12 coefficients.
Aggregator did 4 substitutions.
Reduced MIP has 55437 rows, 162904 columns, and 302357 nonzeros.
Reduced MIP has 4892 binaries, 0 generals, 0 SOSs, and 14676 indicators.
Presolve time = 17.34 sec.
Found feasible solution after 25.69 sec. Objective = 7.6230e+011
Probing time = 0.13 sec.
Clique table members: 21.
MIP emphasis: integer feasibility.
MIP search method: dynamic search.
Parallel mode: opportunistic, using up to 2 threads.
Initializing dual steep norms . . .

2) Phase2: Cores usage: 25% with 8 cores, 8% with 16 cores (2 cores only working)

Intermediate phase
Iteration log . . .
Iteration: 1 Scaled dual infeas = 0.008000
Iteration: 7 Dual objective = -1284030499.782925
Iteration: 850 Dual objective = -1239278203.094896
Iteration: 1578 Dual objective = -1239201458.666793
Iteration: 2124 Dual objective = -1239201375.257585
Iteration: 2723 Dual objective = -1239201258.050264
Iteration: 3423 Dual objective = -1239201123.880705
Iteration: 4223 Dual objective = -1239105865.575989
Iteration: 4948 Dual objective = -1239105739.439498
Iteration: 5532 Dual objective = -1239105655.235284
Perturbation started.
Iteration: 5663 Dual objective = -1239105654.234894
Iteration: 6344 Dual objective = -1238937518.725422
Iteration: 7007 Dual objective = -1238937410.436373
Iteration: 7555 Dual objective = -1238937346.628754
Iteration: 8126 Dual objective = -1238937280.491217
Iteration: 8749 Dual objective = -1238937167.403522
Iteration: 9447 Dual objective = -1238937039.554304
Iteration: 10176 Dual objective = -1238936942.297044
Iteration: 10844 Dual objective = -1238936848.871474
Iteration: 11489 Dual objective = -1238936766.647102
Iteration: 12163 Dual objective = -1238936673.092103
Iteration: 12801 Dual objective = -1238936601.942516
Iteration: 13363 Dual objective = -1238936497.411635
Iteration: 14028 Dual objective = -1238936402.168913
Iteration: 14683 Dual objective = -1238936340.936554
Iteration: 15351 Dual objective = -1238936284.561273
Iteration: 15981 Dual objective = -1238936211.417532
Iteration: 16643 Dual objective = -1238936111.997017
Iteration: 17132 Dual objective = -1238936048.659764
Iteration: 17742 Dual objective = -1238935985.549742
Iteration: 18481 Dual objective = -1238935907.372630
Iteration: 18935 Dual objective = -1238935852.272569
Iteration: 19370 Dual objective = -1238935831.026211
Iteration: 19876 Dual objective = -1238935750.682635
Elapsed time = 11.02 sec. (20000 iterations).
Iteration: 20314 Dual objective = -1238935672.034832
Iteration: 20799 Dual objective = -1238935612.885073
Iteration: 21315 Dual objective = -1238935484.308744
Iteration: 21767 Dual objective = -1238935439.086106
Iteration: 22206 Dual objective = -1238935394.104510
Iteration: 22670 Dual objective = -1238935350.045757
Iteration: 23188 Dual objective = -1238935264.173426
Iteration: 23650 Dual objective = -1238935222.132782
Iteration: 24035 Dual objective = -1238935173.409771
Iteration: 24387 Dual objective = -1238935141.899602
Iteration: 24754 Dual objective = -1238935012.911323
Iteration: 25240 Dual objective = -1238934915.004854
Iteration: 25718 Dual objective = -1238934851.049295
Iteration: 26169 Dual objective = -1238934757.915850
Iteration: 26624 Dual objective = -1238934719.733904
Elapsed time = 21.42 sec. (27000 iterations).
Iteration: 27133 Dual objective = -1238934658.318412
Iteration: 27499 Dual objective = -1238934486.868574
Iteration: 27949 Dual objective = -1238934446.417578
Iteration: 28418 Dual objective = -1238934411.462626
Iteration: 28858 Dual objective = -1238934305.388979
Iteration: 29300 Dual objective = -1238934188.462519
Iteration: 29743 Dual objective = -1238934037.500304
Iteration: 30178 Dual objective = -1238933985.730039
Iteration: 30587 Dual objective = -1238933875.184673
Iteration: 30934 Dual objective = -1238933818.206025
Iteration: 31337 Dual objective = -1238933787.228208
Iteration: 31782 Dual objective = -1238933674.086860
Iteration: 32163 Dual objective = -1238933618.743589
Iteration: 32497 Dual objective = -1238933426.757722
Iteration: 32919 Dual objective = -1238933315.633631
Elapsed time = 32.45 sec. (33000 iterations).
Iteration: 33223 Dual objective = -1238933256.504607
Iteration: 33565 Dual objective = -1238933178.275423
Iteration: 33980 Dual objective = -1238933069.589626
Iteration: 34321 Dual objective = -1238933014.173693
Iteration: 34709 Dual objective = -1238932971.285056
Iteration: 35066 Dual objective = -1238932935.902741
Iteration: 35494 Dual objective = -1238932733.489005
Iteration: 36051 Dual objective = -1238932570.959894
Iteration: 36458 Dual objective = -1238932512.899527
Iteration: 36843 Dual objective = -1197171275.100300
Iteration: 37194 Dual objective = -1197170733.233619
Iteration: 37561 Dual objective = -1197170286.051941
Elapsed time = 43.83 sec. (38000 iterations).
Iteration: 38022 Dual objective = -1197169892.139103
Iteration: 38397 Dual objective = -1197169480.513651
Iteration: 38947 Dual objective = -477285554.295024
Iteration: 39546 Dual objective = -477285427.935321
Iteration: 40151 Dual objective = -476340762.588369
Iteration: 40835 Dual objective = -476340698.736680
Iteration: 41435 Dual objective = -474518597.717195
Iteration: 42102 Dual objective = -474518511.417632
Iteration: 42738 Dual objective = -474518437.318096
Iteration: 43306 Dual objective = -474518392.834008
Iteration: 43934 Dual objective = -474518340.668706
Iteration: 44640 Dual objective = -474518266.952559
Iteration: 45203 Dual objective = -474518173.703738
Iteration: 45797 Dual objective = -474518118.323706
Iteration: 46374 Dual objective = -474518068.647153
Iteration: 47076 Dual objective = -474185315.754157
Iteration: 47565 Dual objective = -474185282.736812
Iteration: 48126 Dual objective = -474185202.237043
Iteration: 48795 Dual objective = -472762337.134023
Iteration: 49431 Dual objective = -472762282.716635
Iteration: 49915 Dual objective = -472762231.359373
Elapsed time = 55.02 sec. (50000 iterations).
Iteration: 50632 Dual objective = -472762156.531527
Iteration: 51215 Dual objective = -472762093.043497
Iteration: 51743 Dual objective = -472761983.577297
Iteration: 52188 Dual objective = -472761909.956192
Iteration: 52669 Dual objective = -472761881.865173
Iteration: 53275 Dual objective = -472761765.533680
Iteration: 53767 Dual objective = -472761688.110148
Iteration: 54273 Dual objective = -472761672.746421
Iteration: 54697 Dual objective = -472761584.411811
Iteration: 55196 Dual objective = -472761546.087347
Iteration: 55616 Dual objective = -472761511.971044
Iteration: 56095 Dual objective = -472761488.187124
Iteration: 56665 Dual objective = -472761409.967309
Elapsed time = 65.53 sec. (57000 iterations).
Iteration: 57208 Dual objective = -472440031.074818
Iteration: 57738 Dual objective = -472439973.250320
Iteration: 58161 Dual objective = -472439880.040036
Iteration: 58596 Dual objective = -472439822.639527
Iteration: 59090 Dual objective = -472439779.186432
Iteration: 59747 Dual objective = -471335636.939909
Iteration: 60162 Dual objective = -471335616.932866
Iteration: 60580 Dual objective = -471335561.708247
Iteration: 61015 Dual objective = -471335528.446305
Iteration: 61628 Dual objective = -470677978.397196
Iteration: 62078 Dual objective = -470677905.352422
Iteration: 62593 Dual objective = -470677844.380185
Iteration: 62973 Dual objective = -470677807.816327
Iteration: 63576 Dual objective = -470677766.050818
Elapsed time = 76.61 sec. (64000 iterations).
Iteration: 64145 Dual objective = -470677750.006886
Iteration: 64604 Dual objective = -470677687.501804
Iteration: 65073 Dual objective = -470677656.403811
Iteration: 65528 Dual objective = -470677581.639561
Iteration: 66030 Dual objective = -470677453.789468
Iteration: 66369 Dual objective = -470677430.984535
Iteration: 66776 Dual objective = -470677349.501885
Iteration: 67128 Dual objective = -470677309.325360
Iteration: 67562 Dual objective = -470677206.293155
Iteration: 68081 Dual objective = -470677176.675245
Iteration: 68581 Dual objective = -470677087.389962
Iteration: 68986 Dual objective = -470677014.880281
Elapsed time = 87.70 sec. (69000 iterations).
Iteration: 69445 Dual objective = -470676756.409179
Iteration: 69781 Dual objective = -448425496.263071
Iteration: 70235 Dual objective = -448424545.649152
Iteration: 70725 Dual objective = -249550302.912931
Iteration: 71249 Dual objective = -249550189.783258
Iteration: 71923 Dual objective = -247744511.741346
Iteration: 72435 Dual objective = -247744443.020117
Iteration: 73099 Dual objective = -246889810.878958
Iteration: 73626 Dual objective = -246429249.037946
Iteration: 74159 Dual objective = -245333578.365887
Iteration: 74926 Dual objective = -206723204.604222
Iteration: 75415 Dual objective = -206669238.338307
Iteration: 75957 Dual objective = -206669156.251568
Iteration: 76433 Dual objective = -206669066.767623
Iteration: 76859 Dual objective = -206669026.665802
Elapsed time = 98.84 sec. (77000 iterations).
Iteration: 77524 Dual objective = -205883211.912465
Iteration: 78080 Dual objective = -205833961.273547
Iteration: 78702 Dual objective = -204924379.089614
Iteration: 79420 Dual objective = -204189024.766828
Iteration: 80001 Dual objective = -203320859.195666
Iteration: 80449 Dual objective = -203320618.543475
Iteration: 80973 Dual objective = 98062135.372827
Iteration: 81573 Dual objective = 327986118.333362
Iteration: 82102 Dual objective = 336815097.874144
Iteration: 82696 Dual objective = 340087242.225160
Iteration: 83158 Dual objective = 341838013.383719
Iteration: 83677 Dual objective = 352742817.164179
Iteration: 84237 Dual objective = 354217443.817579
Iteration: 84618 Dual objective = 354217612.431324
Iteration: 85027 Dual objective = 354217806.566296
Iteration: 85403 Dual objective = 354217896.491753
Iteration: 85993 Dual objective = 354217946.067616
Iteration: 86600 Dual objective = 354278951.691543
Iteration: 87096 Dual objective = 354302444.499503
Iteration: 87792 Dual objective = 354363056.691550
Iteration: 88695 Dual objective = 356863989.911453
Iteration: 88976 Dual objective = 356864039.671179
Elapsed time = 108.84 sec. (89000 iterations).
Iteration: 89530 Dual objective = 356864099.198124
Iteration: 89920 Dual objective = 357513906.370102
Iteration: 90280 Dual objective = 357594978.437404
Iteration: 90806 Dual objective = 358264326.273824
Iteration: 91414 Dual objective = 359188129.728224
Iteration: 92215 Dual objective = 367590522.119441
Iteration: 92946 Dual objective = 368150305.071456
Iteration: 93512 Dual objective = 368489175.799610
Iteration: 94014 Dual objective = 368489233.829158
Iteration: 94602 Dual objective = 369143574.684704
Iteration: 95128 Dual objective = 369213722.771754
Iteration: 95601 Dual objective = 378122480.353835
Iteration: 96154 Dual objective = 380379291.840489
Iteration: 96752 Dual objective = 380581950.533439
Iteration: 97383 Dual objective = 381019592.206809
Iteration: 97871 Dual objective = 381165366.904851
Iteration: 98360 Dual objective = 381251354.585207
Iteration: 98925 Dual objective = 382212664.553821
Iteration: 99248 Dual objective = 382212755.090891
Iteration: 99662 Dual objective = 382212880.436720
Elapsed time = 120.28 sec. (100000 iterations).
Iteration: 100056 Dual objective = 382212961.896902
Iteration: 100661 Dual objective = 382780452.151478
Iteration: 101224 Dual objective = 382912904.322507
Iteration: 101660 Dual objective = 383088864.791841
Iteration: 102330 Dual objective = 383208807.342386
Iteration: 102826 Dual objective = 383208863.225022
Iteration: 103134 Dual objective = 391852794.672942
Iteration: 103763 Dual objective = 422626079.205094
Iteration: 104218 Dual objective = 434674721.984612
Iteration: 104689 Dual objective = 671224961.341730
Iteration: 105084 Dual objective = 671570123.600178
Iteration: 105638 Dual objective = 689057133.746030
Iteration: 106087 Dual objective = 702557681.139527
Iteration: 106575 Dual objective = 710903692.122442
Iteration: 107327 Dual objective = 719092861.999936
Iteration: 107837 Dual objective = 719554487.101861
Iteration: 108572 Dual objective = 730857358.092684
Iteration: 108863 Dual objective = 1122112650.194340
Elapsed time = 130.59 sec. (109000 iterations).
Iteration: 109241 Dual objective = 1122112681.118912
Iteration: 109645 Dual objective = 1123888657.305108
Iteration: 110122 Dual objective = 1186230793.548790
Iteration: 110844 Dual objective = 1186376155.312050
Iteration: 111356 Dual objective = 1252019438.968947
Iteration: 111837 Dual objective = 1256762441.913758
Iteration: 112328 Dual objective = 1256890297.017273
Iteration: 112873 Dual objective = 1274880728.089063
Iteration: 113196 Dual objective = 1274880762.830209
Iteration: 113512 Dual objective = 1274928420.031252
Iteration: 113978 Dual objective = 1284642712.360188
Iteration: 114611 Dual objective = 1298892844.552650
Iteration: 114899 Dual objective = 1298942642.812950
Iteration: 115478 Dual objective = 1299469439.596363
Iteration: 116018 Dual objective = 1300372745.102486
Iteration: 116508 Dual objective = 1300564195.330233
Iteration: 117046 Dual objective = 1313326881.801222
Iteration: 117715 Dual objective = 1316182076.607020
Elapsed time = 140.59 sec. (118000 iterations).
Iteration: 118210 Dual objective = 1317014558.893524
Iteration: 118658 Dual objective = 1317202227.743913
Iteration: 119030 Dual objective = 1318092762.513054
Iteration: 119429 Dual objective = 1318242242.844770
Iteration: 119841 Dual objective = 1326850103.769571
Iteration: 120353 Dual objective = 1328201301.691032
Iteration: 120737 Dual objective = 1328231207.392220
Iteration: 121149 Dual objective = 1328728922.476810
Iteration: 121450 Dual objective = 1328937610.521888
Iteration: 121893 Dual objective = 1329079116.495683
Iteration: 122223 Dual objective = 1329278854.381740
Iteration: 122575 Dual objective = 1329365989.341807
Iteration: 122772 Dual objective = 1330307401.714160
Iteration: 123150 Dual objective = 1330318586.743193
Iteration: 123580 Dual objective = 1330493792.470525
Iteration: 123904 Dual objective = 1330493799.099311
Elapsed time = 151.01 sec. (124000 iterations).
Iteration: 124238 Dual objective = 1330581637.119796
Iteration: 124472 Dual objective = 1330581641.108418
Iteration: 124800 Dual objective = 1330601831.852972
Iteration: 125128 Dual objective = 1330601835.115500
Iteration: 125448 Dual objective = 1330612102.005490
Iteration: 125903 Dual objective = 1332071200.252931
Iteration: 126231 Dual objective = 1332071203.314456
Iteration: 126575 Dual objective = 1332073730.713030
Iteration: 126879 Dual objective = 1332073731.127527
Iteration: 127147 Dual objective = 1332073731.727803
Removing perturbation.
Root relaxation solution time = 157.51 sec.

3) Phase3: Cores usage: 12-24% with 8 cores, 8-16% with 12 cores (1-2 cores only working)

Nodes Cuts/
Node Left Objective IInf Best Integer Best Bound ItCnt Gap

• 0+ 0 7.62304e+011 127244 ---
• 0+ 0 8.34091e+009 127244 ---
• 0+ 0 3.95689e+009 127244 ---
• 0+ 0 3.95069e+009 127244 ---
• 0+ 0 3.77935e+009 127244 ---

0 0 1.33207e+009 659 3.77935e+009 1.33207e+009 127244 64.75%

• 0+ 0 3.52247e+009 1.33207e+009 127244 62.18%

0 0 1.34804e+009 652 3.52247e+009 Cuts: 789 133602 61.73%
0 0 1.36638e+009 617 3.52247e+009 Cuts: 727 140396 61.21%
0 0 1.38164e+009 609 3.52247e+009 Cuts: 652 146705 60.78%

• 0+ 0 2.97567e+009 1.38164e+009 146705 53.57%

Repair heuristic found nothing.

4) Phase4: Heavy duty computation, Cores usage jumps to 100% stable for both 8 and 12 cores

0 2 1.38164e+009 609 2.97567e+009 1.38206e+009 146705 53.55%
Elapsed real time = 362.63 sec. (tree size = 0.01 MB, solutions = 7)
10 11 1.41971e+009 499 2.97567e+009 1.39408e+009 152262 53.15%
Impl Bds: 377
20 21 1.58948e+009 456 2.97567e+009 1.39408e+009 189287 53.15%
Impl Bds: 1332

• 30+ 29 2.66726e+009 1.39408e+009 241459 47.73%

Impl Bds: 1073
30 31 1.61681e+009 338 2.66726e+009 1.39408e+009 241459 47.73%
Impl Bds: 184
40 41 1.70245e+009 254 2.66726e+009 1.39408e+009 273206 47.73%
Impl Bds: 429
50 48 2.00223e+009 180 2.66726e+009 1.39408e+009 401210 47.73%
Impl Bds: 480
60 58 1.75697e+009 163 2.66726e+009 1.39408e+009 410460 47.73%
70 67 1.75709e+009 154 2.66726e+009 1.39408e+009 415888 47.73%
80 75 1.93024e+009 152 2.66726e+009 1.39408e+009 416850 47.73%
90 83 1.79459e+009 168 2.66726e+009 1.39408e+009 418188 47.73%
100 94 1.75735e+009 172 2.66726e+009 1.39408e+009 418296 47.73%
Elapsed real time = 965.86 sec. (tree size = 5.68 MB, solutions = 8)
110 103 1.75735e+009 164 2.66726e+009 1.39408e+009 419940 47.73%
120 112 1.75781e+009 163 2.66726e+009 1.39408e+009 423966 47.73%
130 121 1.75781e+009 170 2.66726e+009 1.39408e+009 424411 47.73%
140 132 1.75781e+009 171 2.66726e+009 1.39408e+009 424978 47.73%
150 142 2.22989e+009 166 2.66726e+009 1.39408e+009 425961 47.73%
160 152 1.75781e+009 167 2.66726e+009 1.39408e+009 427194 47.73%
170 162 1.75781e+009 156 2.66726e+009 1.39408e+009 428424 47.73%
180 172 2.00269e+009 153 2.66726e+009 1.39408e+009 431621 47.73%
190 182 1.75925e+009 162 2.66726e+009 1.39408e+009 434779 47.73%
200 191 2.00383e+009 145 2.66726e+009 1.39408e+009 437645 47.73%
Elapsed real time = 1159.78 sec. (tree size = 8.81 MB, solutions = 8)
210 201 1.75925e+009 156 2.66726e+009 1.39408e+009 439238 47.73%
220 211 1.86205e+009 154 2.66726e+009 1.39408e+009 439323 47.73%
230 221 2.00386e+009 126 2.66726e+009 1.39408e+009 439423 47.73%
240 231 2.00388e+009 125 2.66726e+009 1.39408e+009 440930 47.73%
250 240 2.09928e+009 125 2.66726e+009 1.39408e+009 441014 47.73%
260 251 2.00388e+009 125 2.66726e+009 1.39408e+009 441111 47.73%
270 261 1.75928e+009 151 2.66726e+009 1.39408e+009 441213 47.73%
280 271 1.75928e+009 152 2.66726e+009 1.39408e+009 442229 47.73%
290 281 1.76280e+009 169 2.66726e+009 1.39408e+009 446142 47.73%
..................................................................................

My worry is that by increasing the number of Cores further, which I needed to have a suitable computation speed. Whatever the number of cores are available in the machine this would affect the Phase4 computation time. Wherease I am not able to enhance the speed during phase 1,2 and 3, so that my problem cannot be solved in due time (i.e. I accept max 7 days of running).
My question is:

Is there any option or solution to better use the CPUs during phase 1,2 and 3 ?

Thanks in advance for any help, hints or suggestion you may provide.

Andrea Mansoldo
EIRGRID
Dublin
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: John Cui

Are you sure you are using 12 cores, I saw

Parallel mode: opportunistic, using up to 2 threads.

This means you are only using 2 threads to solve your problem.
John Cui
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: John Cui

And also, can you tell us which CPLEX version are you using?

John Cui
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

Thanks Cui for your reply.

Actually this log file has been generated running on a two core machine. Indeed I am using 12 cores at present. The Posted list is just to single out different calculation phases and relevant CPU usage I recorded with the 12 core machine.

I am using CPLEX through GAMS environment. Version is 12.4.0.0

Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: John Cui

For solving root LP part, did you try parallal barrier?
John Cui
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

Chui,

this is from GAMS/CPLEX manual. Do you mean to set up lpmethod to 6 (at present I use default)?

"
lpmethod (integer)
Species which LP algorithm to use. If left at the default value (0 for automatic), and a primal-feasible
basis is available, primal simplex will be used. If no primal-feasible basis is available, and threads is equal
to 1, dual simplex will be used. If threads is greater than 1 and no primal-feasible basis is available, the
concurrent option will be used.
Sifting may be useful for problems with many more variables than equations.
The concurrent option runs multiple methods in parallel. The rst thread uses dual simplex. The second
thread uses barrier. The next thread uses primal simplex. Remaining threads are used by the barrier run.
The solution is returned by rst method to nish.
(default = 0)
0 Automatic
1 Primal Simplex
2 Dual Simplex
3 Network Simplex
4 Barrier
5 Sifting
6 Concurrent "

Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: John Cui

For pure LP, yes.
But for MIP, you need to use below parameters:

mip limits auxrootthreads -1
mip strategy startalgorithm 4

Can you attach your model to here?
John Cui
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

Thanks John,

actually in GAMS auxrootthreads has to be the number of the machine cores-1

I attach the model case. The main GAMS file is ESPAUT_V18.gms

Thanks for your help

Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: John Cui

Sorry, I don't have GAMS, so could you please output your model to a .sav model file, then attach it to here?

John Cui
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

Thanks John,

actually in GAMS auxrootthreads has to be the number of the machine cores-1

I attach the model case. The main GAMS file is ESPAUT_V18.gms

Thanks for your help

Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

Here are the save files, 5 model are present into the program 2 lp and 3 mip.

I hope this helps

Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

By the way the issue is found in both situation,

in particular, generating the model (8% usage)
Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: John Cui

CPLEX 12.4 with 2 threads on windows 7 64bits, can solve your 5 models in reasonable memory and performance.

Welcome to IBM(R) ILOG(R) CPLEX(R) Interactive Optimizer 12.4.0.1 with Simplex, Mixed Integer & Barrier Optimizers 5725-A06 5725-A29 5724-Y48 5724-Y49 5724-Y54 5724-Y55 Copyright IBM Corp. 1988, 2011.  All Rights Reserved.   Type 
'help' 

for a list of available commands. Type 
'help' followed by a command name 

for more information on commands.   CPLEX> r BusBar.sav Problem 
'BusBar.sav' read. Read time =    0.02 sec. CPLEX> o Parallel mode: deterministic, using up to 2 threads 

for concurrent optimization. Tried aggregator 1 time. LP Presolve eliminated 36 rows and 24638 columns. Reduced LP has 13 rows, 64 columns, and 113 nonzeros. Initializing dual steep norms . . .   Iteration log . . . Iteration:     1   Dual objective     =     515392452.343262 Reinitializing dual norms . . .   Dual simplex solved model.     Dual simplex - Optimal:  Objective = 5.1539245235e+008 Solution time =    0.08 sec.  Iterations = 1 (0) Deterministic time = 11.97 ticks  (153.45 ticks/sec)   CPLEX> r LFE.sav Problem 
'LFE.sav' read. Read time =    0.11 sec. CPLEX> o Tried aggregator 3 times. MIP Presolve eliminated 32398 rows and 51821 columns. MIP Presolve modified 48 coefficients. Aggregator did 6146 substitutions. Reduced MIP has 23826 rows, 54853 columns, and 86256 nonzeros. Reduced MIP has 0 binaries, 0 generals, 0 SOSs, and 0 indicators. Probing time =    0.00 sec. Tried aggregator 1 time. MIP Presolve eliminated 179 rows and 424 columns. Reduced MIP has 23647 rows, 54429 columns, and 85716 nonzeros. Reduced MIP has 0 binaries, 0 generals, 0 SOSs, and 0 indicators. Presolve time =    0.48 sec. Probing time =    0.00 sec. MIP emphasis: balance optimality and feasibility. MIP search method: dynamic search. Parallel mode: deterministic, using up to 2 threads. Root relaxation solution time =    1.79 sec.   Nodes                                         Cuts/  Node  Left     Objective  IInf  Best Integer    Best Bound    ItCnt     Gap   *     0     0      integral     0  6.28268e+010  6.28268e+010      716    0.00% Elapsed real time =   2.45 sec. (tree size =  0.00 MB, solutions = 1)   Root node processing (before b&c): Real time             =    1.97 Parallel b&c, 2 threads: Real time             =    0.00 Sync time (average)   =    0.00 Wait time (average)   =    0.00 ------- Total (root+branch&cut) =    1.97 sec.   Solution pool: 2 solutions saved.   MIP - Integer optimal solution:  Objective = 6.2826805735e+010 Solution time =    2.51 sec.  Iterations = 716  Nodes = 0 Deterministic time = 1258.71 ticks  (501.08 ticks/sec)   CPLEX> r OPFCON.sav Problem 
'OPFCON.sav' read. Read time =    0.02 sec. CPLEX> o Parallel mode: deterministic, using up to 2 threads 

for concurrent optimization. Tried aggregator 1 time. LP Presolve eliminated 36 rows and 24638 columns. Reduced LP has 13 rows, 64 columns, and 113 nonzeros. Initializing dual steep norms . . .   Iteration log . . . Iteration:     1   Dual objective     =     515392452.343262 Reinitializing dual norms . . .   Dual simplex solved model.     Dual simplex - Optimal:  Objective = 5.1539245235e+008 Solution time =    0.03 sec.  Iterations = 1 (0) Deterministic time = 11.97 ticks  (386.10 ticks/sec)   CPLEX> r OPFFIX.sav Problem 
'OPFFIX.sav' read. Read time =    0.11 sec. CPLEX> o Parallel mode: deterministic, using up to 2 threads 

for concurrent optimization. Tried aggregator 1 time. LP Presolve eliminated 13291 rows and 23155 columns. Aggregator did 9228 substitutions. Reduced LP has 34555 rows, 97813 columns, and 237656 nonzeros. Initializing dual steep norms . . .   Iteration log . . . Iteration:     1   Dual objective     = -2554580624622.187500 Perturbation started. Iteration:   809   Dual objective     =  -85744641732.385376 Iteration:  1482   Dual objective     =  -38337411019.391029 Iteration:  2062   Dual objective     =   34314443337.655422 Iteration:  2936   Dual objective     =  141868852703.249180 Iteration:  3440   Dual objective     =  157034749745.110570 Iteration:  4191   Dual objective     =  165660106008.071870 Iteration:  4881   Dual objective     =  167263454618.385560 Iteration:  5543   Dual objective     =  167813329809.026580 Iteration:  6215   Dual objective     =  168178601379.355740 Iteration:  6743   Dual objective     =  168504618189.821010 Iteration:  7340   Dual objective     =  168701287061.675510 Iteration:  7944   Dual objective     =  168790306791.744510 Iteration:  8314   Dual objective     =  168844369271.585170 Iteration:  8939   Dual objective     =  168878894688.134920 Iteration:  9479   Dual objective     =  168934434063.149630 Iteration:  9796   Dual objective     =  168957484228.528560 Iteration: 10161   Dual objective     =  168964559590.033600 Iteration: 10535   Dual objective     =  168970437158.539670 Iteration: 11023   Dual objective     =  168971307666.738530 Iteration: 11581   Dual objective     =  168971307666.740720 Iteration: 12318   Dual objective     =  168971307666.749080 Iteration: 12976   Dual objective     =  168971307666.751340 Iteration: 13722   Dual objective     =  168971307666.763580 Iteration: 14320   Dual objective     =  168971307666.773280 Iteration: 15042   Dual objective     =  169080185301.876280 Iteration: 15744   Dual objective     =  169080185301.880830 Iteration: 16396   Dual objective     =  169080185301.883850 Iteration: 17085   Dual objective     =  169080185301.881900 Iteration: 17557   Dual objective     =  169080185301.882170 Iteration: 18140   Dual objective     =  169091827740.767400 Iteration: 18737   Dual objective     =  169091827740.767550 Iteration: 19017   Dual objective     =  169091827740.765590 Iteration: 19304   Dual objective     =  169091827740.765590 Iteration: 19533   Dual objective     =  169091827740.765690 Iteration: 19798   Dual objective     =  169091827740.765720 Iteration: 20075   Dual objective     =  169091827740.771180 Iteration: 20352   Dual objective     =  169091827740.775570 Iteration: 20618   Dual objective     =  169091827740.778990 Iteration: 20875   Dual objective     =  169091827740.779910 Elapsed time =   10.90 sec. (21000 iterations). Iteration: 21184   Dual objective     =  169091827740.789220 Iteration: 21784   Dual objective     =  169091827740.800480 Iteration: 22282   Dual objective     =  169091827740.828280 Iteration: 22781   Dual objective     =  169220139456.918180 Iteration: 23378   Dual objective     =  169223572033.403720 Iteration: 23961   Dual objective     =  169229119755.155180 Iteration: 24495   Dual objective     =  169247865060.510310 Iteration: 24997   Dual objective     =  169256108516.289280 Iteration: 25520   Dual objective     =  169256108516.291840 Removing perturbation. Markowitz threshold set to 0.5 Iteration: 25521   Dual objective     =  169256108514.185730 Reinitializing dual norms . . .   Dual simplex solved model.     Dual simplex - Optimal:  Objective = 1.6925610851e+011 Solution time =   14.18 sec.  Iterations = 25521 (0) Deterministic time = 5092.82 ticks  (359.13 ticks/sec)   CPLEX> r OPF.sav Problem 
'OPF.sav' read. Read time =    0.11 sec. CPLEX> o Tried aggregator 3 times. MIP Presolve eliminated 19528 rows and 32366 columns. MIP Presolve modified 124 coefficients. Aggregator did 20174 substitutions. Reduced MIP has 22669 rows, 77699 columns, and 214702 nonzeros. Reduced MIP has 0 binaries, 0 generals, 0 SOSs, and 0 indicators. Probing time =    0.02 sec. Tried aggregator 1 time. Presolve time =    0.59 sec. Probing time =    0.02 sec. MIP emphasis: balance optimality and feasibility. MIP search method: dynamic search. Parallel mode: deterministic, using up to 2 threads. Root relaxation solution time =   12.37 sec.   Nodes                                         Cuts/  Node  Left     Objective  IInf  Best Integer    Best Bound    ItCnt     Gap   0     0     unbounded                                      20305   Root node processing (before b&c): Real time             =   12.48 Parallel b&c, 2 threads: Real time             =    0.00 Sync time (average)   =    0.00 Wait time (average)   =    0.00 ------- Total (root+branch&cut) =   12.48 sec.     MIP - Integer infeasible or unbounded. Current MIP best bound is infinite. Solution time =   13.09 sec.  Iterations = 20305  Nodes = 0 Deterministic time = 5776.22 ticks  (441.30 ticks/sec)

John Cui
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

John,

thanks for this input.

I know that this is easy, despite any option. But because it is reduced. I will send you the big one. The yesterday updating was that memory is less of an issue with some manual poilishing.
But the question is how to have CPU exploited in the root and generation model phases. Did you check the CPU usage during the running.

Keep in touch

Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: John Cui

I found primal simplex is the best solver in phase 2 to solve the root LP.
So I suggest you to set root node's lpmethod to primal simplex.

John Cui
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

Thanks John for the follow up,

I will do it.

Find attached a bigger case, where the issue become relevant. I ran the program on a 12, 24 and 32 machine and always two cores are working in the root phase.
This may become an issue when a large case like this has to be solved. In principle we cannot drop the root and preliminary phase when increasing the number of core available.
So that if we zeroing the heavy duty phase computation where the CPU works at 100%, by using an high core number computer, we could never reach a solution because the preliminary root phase, with two cores only, does not provide enough speed to produce a solution in due time.

Is there any possibility to enhance the CPU utilisation at an higher percentage for any phase ? Is this issue due to the algorithm ?

Note that I kept auxroothreads set to N-1 available cores.
John did you get the attache file ?

Apparently the dimensions do not allow to send it to you.

Please let me know
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

aTTACHED FILE 65MB
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

Can you please provide alternatives to upload 65MB files ?

Thanks

Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: SystemAdmin

Some thoughts on your problem: I think your analysis is mostly correct and the low CPU utilization you see is related to the fact that some phases of the solution process are inherently sequential and cannot be parallelized well.
Phase 1: This is presolve. Presolve attempts to eliminate redundant constraints, simplify constraints etc. This is not parallelized in CPLEX, so CPLEX will never use more than a single CPU for presolve.
Phase 2: This is solving the initial LP relaxation. In default settings CPLEX does not necessarily exploit all available CPUs here. There are several algorithms to solve the initial LP relaxation. The most prominent are primal and dual simplex and the barrier algorithm. primal and dual simplex both are sequential algorithms and will not use more than a single CPU. The barrier algorithm can be parallelized nicely and will try to use all available CPUs. By choosing a different algorithm for solving the initial LP relaxation you may increase CPU utilization. But notice that using more and more cores is not always a good idea. At some point the communication overhead between cores/threads will cross a threshold and using more cores will actually slow down things. You may have to experiment with the number of cores. CPLEX also implements a so called "concurrent" solve for the initial LP relaxation: It runs simplex and barrier in parallel and uses the result of the fastest algorithm.
Phase 3: This is separation of cuts at the root node. This is again inherently sequential and low CPU utilization is kind of expected.
Phase 4: This is the parallel tree search. The process is highly parallel and you should see all CPUs busy most of the time.

In order to make use of the idle CPUs during inherently sequentially phases CPLEX has parameter auxrootthreads that John already mentioned. This parameter tells CPLEX that it is fine to use that many threads to run different things "in the background". CPLEX will run additional algorithms (for example heuristics). It will run whatever it deems useful (which may well be nothing). So by using the auxrootthreads parameter you may be able to increase CPU utilization.

If I understand correctly you also lose a significant amount of time in model generation. Model generation is not part of CPLEX. This is all in GAMS. You will have to talk to GAMS people to learn whether this can be sped up if multiple cores are available.

If you add more processors to increase speed then the following comments apply:
Phase 1: This will not get faster by adding more processors. CPLEX may run additional things in the background if you provide more processors but phase 1 (presolve) will not get faster itself. Presolve is usually very fast, so that should not be a problem in general. If you still think that it is to slow you can try to disable presolve.
Phase 2: This can be sped up by switching to parallel algorithms like barrier. As mentioned above, there is a point at which adding more machines will not make things faster but only slower. The "best" number of cores depends on the individual problem characteristics.
Phase 3: This may get faster when switching to parallel barrier. If you cannot get it fast enough then you can consider turning off some of the cuts or even all cuts.
Phase 4: This phase will benefit most from more cores. However, there is again a problem-dependent threshold above which adding more cores will only slow down things.

I hope this helps.
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

Thanks Daniel for your detailed explanation. Some attempts are then needed. Indeed we do not encounter major issues in the Barrier implementation, where the use is reasonable and the solution is performed in max 5 hours for the largest case.

Phase 3 is really the major burden. We di not implement anything to control this phase. Actually, by turning it off we enter in Phase 4 strightforward but then the MIP is larger. I understand therefore that some cuts options have to be considered looking for the best compromise in each problem.

I wanted also to ask you about the binary file that has been asled by john. I got the translator file from GAMS so that I am now in a position of producing this file.

I think, by running CPLEX with the binary, a great improvement can be obtained in all the process. Do you know what could be the scale of this.
What should be the computer requirements to run such a file ?

Thanks

Andrea
#CPLEXOptimizers
#DecisionOptimization

Originally posted by: merda30esimi

I have one more question about the log file of the root phase (number 3)

In the headings it appears an IINF title, third column which I understood are the number of Integer Variables to be fixed. During the root progression the number flactuates decreasing after a while to a value lower the the number of initial variables.

My question is, does this number represent the Integer variables which are worth to consider ?? In other words can I run first a root case with a very High tolerance in order to screen the Universal set and use the outcome of the root and cut analysis for the MIP optimisation ?
I noticed for instance that starting with 5500 Integer variable, with a 95% tolerance I has 2500 IINF index which could help to progress faster afterwards.
IINF Best current I sol
0 0 1.34804e+009 652 3.52247e+009 Cuts: 789 133602 61.73%
0 0 1.36638e+009 617 3.52247e+009 Cuts: 727 140396 61.21%
0 0 1.38164e+009 609 3.52247e+009 Cuts: 652 146705 60.78%

0+ 0 2.97567e+009 1.38164e+009 146705 53.57%
Andrea
#CPLEXOptimizers
#DecisionOptimization