Decision Optimization

Decision Optimization

Delivers prescriptive analytics capabilities and decision intelligence to improve decision-making.

 View Only
Expand all | Collapse all

Out of memory on MIP

  • 1.  Out of memory on MIP

    Posted Sat March 19, 2011 06:23 AM

    Originally posted by: SaqibIlyas


    Hello everyone
    I have an MIP problem as defined in the attached LP file, alongwith an MST file describing an initial solution. I was getting error termination due to CPLEX v 12.2 being out of memory. Following some online resources as well as the User's Manual, I used the following parameters:

    set workmem 2000
    set mip limits treememory 1800 
    set workdir {my home directory}
    set mip strategy file 2
    set mip strategy probe 3
    


    I am still getting an out of memory:

    MIP - Memory limit exceeded, integer feasible:  Objective =  5.4508110000e+02
    Current MIP best bound =  5.3864353036e+02 (gap = 6.43757, 1.18%)
    Solution time =  211.63 sec.  Iterations = 217682  Nodes = 4666 (3840)
    


    Isn't CPLEX supposed to go on by writing some nodes to file in this case? How should I proceed?
    #CPLEXOptimizers
    #DecisionOptimization


  • 2.  Re: Out of memory on MIP

    Posted Sun March 20, 2011 05:37 AM

    Originally posted by: SystemAdmin


    set mip limits treememory 1800
    

    instructs CPLEX to stop as soon as the tree becomes larger than 1800 MB. This parameter sets a limit on the total size of the tree. Just remove that line.
    #CPLEXOptimizers
    #DecisionOptimization


  • 3.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 05:40 AM

    Originally posted by: SaqibIlyas


    Thanks. That helped. However, now, after giving the following commands:

    set mip strategy nodeselect 0
    set workmem 1500
    set workdir {myhomedir}
    set mip strategy file 3
    set mip strategy probe 3
    


    The optimization still terminates with error, giving the following output:

    There may be further error information in the clone logs.
     
    Cover cuts applied:  2048
    Zero-half cuts applied:  33
    Gomory fractional cuts applied:  13
     
    Root node processing (before b&c):
      Real time             =   38.80
    Parallel b&c, 8 threads:
      Real time             = 3686.78
      Sync time (average)   =  113.53
      Wait time (average)   =  504.77
                              -------
    Total (root+branch&cut) = 3725.58 sec.
    CPLEX Error  1803: Failure on temporary file write.
     
    Solution pool: 1 solution saved.
     
    MIP - Error termination, no tree:  Objective =  5.4508110000e+02
    Current MIP best bound =  5.3865897252e+02 (gap = 6.42213, 1.18%)
    Solution time = 3729.96 sec.  Iterations = 2418722  Nodes = 120379 (114454)
    


    I could not find any details in the clone logs. How do I proceed now?
    Thanks
    #CPLEXOptimizers
    #DecisionOptimization


  • 4.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 06:29 AM

    Originally posted by: SaqibIlyas


    Based on an earlier thread, I did an strace on the problem:

    strace -e trace=open /opt/cplex122/cplex/bin/x86_sles10_4.1/cplex < input
    


    where input contains all the commands I mentioned earlier in this thread to solve this problem. The last strace lines I saw were:

    open("/home/cchchen/cpx2qMZXX/cpx2552", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 14
    open("/home/cchchen/cpx2qMZXX/cpx2553", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 14
    open("/home/cchchen/cpx2qMZXX/cpx2554", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 14
    


    These don't appear to be indicative of an error. So, I guess the "no tree" in the output probably is indicative of what the actual error might be. Any help?
    Thanks
    #CPLEXOptimizers
    #DecisionOptimization


  • 5.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 06:39 AM

    Originally posted by: SystemAdmin


    May it be that your hard disk is full?
    Are there already other temporary CPLEX files in your home directory, or were this the first files that CPLEX wanted to write?
    #CPLEXOptimizers
    #DecisionOptimization


  • 6.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 06:44 AM

    Originally posted by: SaqibIlyas


    These were the first files written by CPLEX. It created a temporary folder and then wrote thousands of files to it right in front of my eyes (the strace output shows some). The / folder is shown to be only 39% full, so I doubt that disk space would be the issue.
    #CPLEXOptimizers
    #DecisionOptimization


  • 7.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 06:48 AM

    Originally posted by: SystemAdmin


    Hmm, very strange. Can you reproduce this issue? And if so, does CPLEX terminate at the same point?
    #CPLEXOptimizers
    #DecisionOptimization


  • 8.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 06:59 AM

    Originally posted by: SaqibIlyas


    Yea. I've already attached the lp and mst files. The commands in the input file are:

    read rbmilp.1.10.20.txt lp
    read mipstart.1.10.20.txt mst
    set workmem 1500
    set workdir /home/cchchen
    set mip strategy file 3
    set mip strategy probe 3
    opt
    


    in case you would like to try this out at your end. I am using CPLEX v 12.2.
    Thanks
    #CPLEXOptimizers
    #DecisionOptimization


  • 9.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 07:03 AM

    Originally posted by: SystemAdmin


    Yes, I would like to try. Could you please give precise details about the machine you were using (in particular OS and number of cores). Additionally, if you still have it, the log file would be very useful as well.

    Thanks,

    Tobias
    #CPLEXOptimizers
    #DecisionOptimization


  • 10.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 07:22 AM

    Originally posted by: SaqibIlyas


    For some reason, the forum is saying "disallowed content detected" on my message. I am pasting the text in the attached file instead, then.
    #CPLEXOptimizers
    #DecisionOptimization


  • 11.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 07:27 AM

    Originally posted by: SystemAdmin


    Do you still have the log file somewhere? This would help to easily check whether my run is producing the same solving path as yours.
    #CPLEXOptimizers
    #DecisionOptimization


  • 12.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 07:52 AM

    Originally posted by: SaqibIlyas


    I'm not quite sure if it has been overwritten. It is generally appended to, when you run another optimization, right?
    #CPLEXOptimizers
    #DecisionOptimization


  • 13.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 07:58 AM

    Originally posted by: SystemAdmin


    Yes, this is the log file I wanted to have. Nevertheless, I think the issue is clear: your home partition ran out of space. Please just try again using a different workdir (on a partition that has more space left).
    #CPLEXOptimizers
    #DecisionOptimization


  • 14.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 09:18 AM

    Originally posted by: SaqibIlyas


    Indeed, it was "no space left on device"
    #CPLEXOptimizers
    #DecisionOptimization


  • 15.  Re: Out of memory on MIP

    Posted Wed March 23, 2011 08:57 AM

    Originally posted by: SaqibIlyas


    I set the working dir to a mount point containing 11 GB free space. Even when there is still about 9 GB free on that mount, I get an out of memory status. Here is the output after I did an strace on "write":

    write(14, "\0\254C\1\0\356&}\nw\254F@Q\262\262\377\377I\273?\24\1u\263N\314{@Z\242\367"..., 339968) = 339968
    write(14, "\1\0^M\1\0_M\1\0`M\1\0aM\1\0bM\1\0cM\1\0\0dM\1\0e"..., 1441) = 1441
    write(1, "\nThere may be further error info"..., 59
    There may be further error information in the clone logs.
    ) = 59
    write(13, "CPLEX Error 1001: Out of memory"..., 34) = 34
    write(1, "\n", 1
    )                       = 1
    write(1, "Cover cuts applied: 6479\n", 26Cover cuts applied:  6479
    ) = 26
    write(1, "Zero-half cuts applied: 104\n", 29Zero-half cuts applied:  104
    ) = 29
    write(1, "Gomory fractional cuts applied: "..., 35Gomory fractional cuts applied:  6
    ) = 35
    write(1, "\nRoot node processing (before b&"..., 36
    Root node processing (before b&c):
    ) = 36
    write(1, " Real time = 1210.6"..., 34  Real time             = 1210.69
    ) = 34
    write(1, "Parallel b&c, 8 threads:\n", 25Parallel b&c, 8 threads:
     
    ) = 25
    write(1, " Real time = 4516.8"..., 34  Real time             = 4516.85
    ) = 34
    write(1, " Sync time (average) = 165.5"..., 34  Sync time (average)   =  165.52
    ) = 34
    write(1, " Wait time (average) = 446.5"..., 34  Wait time (average)   =  446.59
    ) = 34
    write(1, " ------"..., 34                          -------
    ) = 34
    write(1, "Total (root+branch&cut) = 5727.5"..., 39Total (root+branch&cut) = 5727.53 sec.
    ) = 39
    write(1, "Warning: MIP starts not construc"..., 69Warning: MIP starts not constructed because of out-of-memory status.
    ) = 69
    write(1, "\n", 1
    )                       = 1
    write(1, "Solution pool: 8 solutions saved"..., 34Solution pool: 8 solutions saved.
    ) = 34
    write(1, "\n", 1
    )                       = 1
    write(1, "MIP - Error termination, integer"..., 74MIP - Error termination, integer feasible:  Objective =  4.4631860000e+02
    ) = 74
    write(1, "Current MIP best bound = 4.3534"..., 66Current MIP best bound =  4.3534126226e+02 (gap = 10.9773, 2.46%)
    ) = 66
    write(1, "Solution time = 5748.27 sec. It"..., 74Solution time = 5748.27 sec.  Iterations = 3966194  Nodes = 49244 (34961)
    ) = 74
    write(1, "\n", 1
    )                       = 1
    write(3, "\nThere may be further error info"..., 740) = 740
    write(1, "CPLEX> ", 7CPLEX> )
    


    Also attaching the log file. The commands in the input file are:

    read rbmilp.5.30.20.txt lp
    read mipstart.5.30.20.txt mst
    set workmem 500
    set workdir /tmp3
    set mip strategy file 3
    set mip strategy probe 3
    set mip strategy nodeselect 0
    opt
    


    The strace command is:

    strace -e trace=write /opt/cplex122/cplex/bin/x86_sles10_4.1/cplex < input
    

    #CPLEXOptimizers
    #DecisionOptimization


  • 16.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 07:30 AM

    Originally posted by: SystemAdmin


    Since the problem is a file write it would be more meaningful to do a
    strace -e trace=write ...
    

    in this case and see if any of the calls to write() returns with 0 or -1.
    You say your disk is not full since there is still room on '/'. Is your home directory on the same partition/disk as '/'? Does 'df -h /home/cchchen' also report lots of free space? And does it report that free space also during the solve?
    #CPLEXOptimizers
    #DecisionOptimization


  • 17.  Re: Out of memory on MIP

    Posted Mon March 21, 2011 07:47 AM

    Originally posted by: SaqibIlyas


    Yes, good point. When I df my home directory it showed it as 99% full, with about 700 MB available space. That might well be causing it.
    #CPLEXOptimizers
    #DecisionOptimization