Originally posted by: Mr.Hai

Hello,

I tried to run an example of distributed MIP optimization on a single machine with multiple cores (4 cores in this case) using OPEN MP. Here is the script that I use to specify the master and workers. 

 

~/opt/openmpi/bin$ mpirun -x "LD_LIBRARY_PATH=/home/daohai/IBM/cplex/bin/x86-64_linux:/home/daohai/opt/openmpi/lib" -tag-output -localhost localhost1 -localhost localhost2 -localhost localhost3 -localhost localhost4 /home/daohai/IBM/cplex/bin/x86-64_linux/cplex -mpi -libpath="/home/daohai/opt/openmpi/lib" -mpilib="/home/daohai/opt/openmpi/lib/libmpi.so" -mpiapi="openmpi"

 

However, it turns out that: 

mpirun was unable to launch the specified application as it could not find an executable:

Executable: -localhost
Node: daohai-Inspiron-5523

 

Does anyone know what mistakes that I had in above script?

 

Thanks

#DecisionOptimization
#MathematicalProgramming-General

Originally posted by: Mr.Hai

I should add more: when I change the script to become: 

daohai@daohai-Inspiron-5523:~/opt/openmpi/bin$ mpirun -x "LD_LIBRARY_PATH=/home/daohai/IBM/cplex/bin/x86-64_linux:/home/daohai/opt/openmpi/lib" -tag-output -host localhost -host localhost -host localhost -host localhost /home/daohai/IBM/cplex/bin/x86-64_linux/cplex -mpi -libpath="/home/daohai/opt/openmpi/lib" -mpilib="/home/daohai/opt/openmpi/lib/libmpi.so" -mpiapi="openmpi"

 

and here is the response: 

[daohai-Inspiron-5523:01365] [[56193,0],0] mca_oob_tcp_recv_handler: invalid message type: 15
[daohai-Inspiron-5523:01365] [[56193,0],0] mca_oob_tcp_recv_handler: invalid message type: 15
[daohai-Inspiron-5523:01365] [[56193,0],0] mca_oob_tcp_recv_handler: invalid message type: 15
[daohai-Inspiron-5523:01365] [[56193,0],0] mca_oob_tcp_recv_handler: invalid message type: 15
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.

 

So, the question is how to specify the master and workers in the single machine case?

 

Thanks so much if anyone could suggest something

#DecisionOptimization
#MathematicalProgramming-General

Originally posted by: Mr.Hai

If I use process transport protocol by writing the following configuration file (.vmc): 

<?xml version="1.0"?>
<vmc>
  <machine name="localhost">
    <transport type="process">
      <cmdline>
         <item value="ssh"/>
         <item value="localhost"/>
         <item value="~/home/daohai/IBM//cplex/bin/x86-64_linux/cplex"/>
         <item value="-worker=process"/>
         <item value="-stdio"/>
         <item value="-libpath=~/home/daohai/IBM//cplex/bin/x86-64_linux"/>
      </cmdline>
    </transport>
    <param name="threads" value="2" />
  </machine>

<machine name="localhost">
    <transport type="process">
      <cmdline>
         <item value="ssh"/>
         <item value="localhost"/>
         <item value="/home/daohai/IBM//cplex/bin/x86-64_linux/cplex"/>
         <item value="-worker=process"/>
         <item value="-stdio"/>
         <item value="-libpath=/home/daohai/IBM//cplex/bin/x86-64_linux"/>
      </cmdline>
    </transport>
    <param name="threads" value="3" />
  </machine>

<machine name="localhost">
    <transport type="process">
      <cmdline>
         <item value="ssh"/>
         <item value="localhost"/>
         <item value="/home/daohai/IBM//cplex/bin/x86-64_linux/cplex"/>
         <item value="-worker=process"/>
         <item value="-stdio"/>
         <item value="-libpath=/home/daohai/IBM//cplex/bin/x86-64_linux"/>
      </cmdline>
    </transport>
    <param name="threads" value="4" />
  </machine>
</vmc>

 

And in the interactive mode, I perform three following commands

read config.vmc

read knapsack.lp

mipopt

 

and here is the response:

CPLEX Error  1814: Failed to load dynamically loadable library.

 

My question is why i cannot load the library since I precisely indicate the bin files in vmc script

 

Thanks for your help

 

#DecisionOptimization
#MathematicalProgramming-General

Can you please try removing the '~' from the various directories specified in the VMC file? It looks wrong to have this there and in any case, '~' is something that only works in a shell, it is not auto-expanded by the VMC reader.

Can you also try to put /home/daohai/IBM//cplex/bin/x86-64_linux into your LD_LIBRARY_PATH environment variable before launching CPLEX?

#DecisionOptimization
#MathematicalProgramming-General

Originally posted by: Mr.Hai

Thanks for the suggestion. I did the following

1. Remove the ~ in vmc file

2. Here is the content of bashrc file: 

# add cplex 
PATH=$PATH:/home/daohai/IBM/cplex/bin/x86-64_linux/
export PATH

# add mpi
$PATH:/home/daohai/opt/openmpi/bin/
export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/home/daohai/opt/openmpi/lib

 

3. When I run in interactive mode: 

CPLEX> read config.vmc
VMC file 'config.vmc' read.
CPLEX> read knapsack.lp
Problem 'knapsack.lp' read.
Read time = 0.00 sec. (0.00 ticks)
CPLEX> mipopt
CPLEX Error  1814: Failed to load dynamically loadable library.

Error termination, CPLEX Error  1814.
Solution time =    0.00 sec.
Deterministic time = 0.00 ticks  (0.00 ticks/sec)

 

Still, it produces the same problem! Notice that the process transport protocol is used. 

 

Do you have any insight/advice on this case?

 

 

 

#DecisionOptimization
#MathematicalProgramming-General

First of all, these lines in your bashrc look pretty odd:

$PATH:/home/daohai/opt/openmpi/bin/
export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/home/daohai/opt/openmpi/lib

I guess that what you want to have is

PATH="$PATH:/home/daohai/opt/openmpi/bin/"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/daohai/opt/openmpi/lib"

However, this is unrelated to the failure using the process transport. For the distributed parallel MIP to work you need to have in LD_LIBRARY_PATH the directory that contains the CPLEX shared libraries, which seems to be /home/daohai/IBM/cplex/bin/x86-64_linux in your case. So try adding this to your bashrc:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/daohai/IBM/cplex/bin/x86-64_linux

In the interactive optimizer you can verify that LD_LIBRARY_PATH is as expected by

CPLEX> x echo $LD_LIBRARY_PATH

This prints the current value of the LD_LIBRARY_PATH environment variable. That value should contain /home/daohai/IBM/cplex/bin/x86-64_linux.

#DecisionOptimization
#MathematicalProgramming-General

Originally posted by: Mr.Hai

I add the following line, as you suggested, to the bashrc and here are the last lines:

 

# add cplex 
PATH=$PATH:/home/daohai/IBM/cplex/bin/x86-64_linux/
export PATH

# add mpi
PATH=$PATH:/home/daohai/opt/openmpi/bin/
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/daohai/opt/openmpi/lib"

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/daohai/IBM/cplex/bin/x86-64_linux

 

and when I issue in interactive mode: CPLEX> x echo $LD_LIBRARY_PATH 

the result is: $LD_LIBRARY_PATH:/home/daohai/opt/openmpi/lib, which is not as expected!

 

I tried to comment on this line #export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/daohai/opt/openmpi/lib"

However, the result is the same

 

So, do you know what is cause behind that?

 

Moreover, with the mipopt, it is automatically solved in parallel with maximum number of cores. So, I wonder how to distinguish two cases: assume that the first one with loading vmc configuration works and the second case when simply executing mipopt command. Or what are the different between two cases?

Thanks

 

 

#DecisionOptimization
#MathematicalProgramming-General

I don't know why you still get the wrong LD_LIBRARY_PATH in the interactive. From the output it looks like you still have that \$LD_LIBRARY_PATH (with the backslash) somewhere in your startup files. What you can try to do is to explicitly set LD_LIBRARY_PATH in the shell right before starting the interactive:

you@home> export LD_LIBRARY_PATH=/home/daohai/opt/openmpi/lib:/home/daohai/IBM/cplex/bin/x86-64_linux
you@home> cplex

With this you should get the expected LD_LIBRARY_PATH in the interactive.

As for your second question: When you issue 'mipopt' then the interactive optimizer checks whether a VMC is loaded. If there is a VMC then it performs distributed parallel MIP using this VMC. If there is no VMC loaded then it just runs shared memory parallel MIP.

#DecisionOptimization
#MathematicalProgramming-General

Originally posted by: Mr.Hai

Yes, there is error from my part. Now, it seems to work but I do not well understand

1. Case 1: If I set the value of thread to 1 for all machines in .vmc file (single computer with 4 cores, so I setup three distributed solvers)

Here is the result: Note that "Parallel mode: none, using 1 thread"

Tried aggregator 2 times.
Aggregator did 1 substitutions.
Reduced MIP has 2 rows, 3 columns, and 6 nonzeros.
Reduced MIP has 0 binaries, 1 generals, 0 SOSs, and 0 indicators.
Presolve time = 0.00 sec. (0.01 ticks)
Running distributed MIP on 3 solvers.
Setting up 3 distributed solvers.
Setup time = 0.00 sec. (0.00 ticks)
Starting ramp-up.

Found incumbent of value 34.000000 after 0.00 sec. (0.00 ticks)
MIP emphasis: balance optimality and feasibility.
MIP search method: dynamic search.
Parallel mode: none, using 1 thread.
Root relaxation solution time = 0.00 sec. (0.00 ticks)

        Nodes                                         Cuts/
   Node  Left     Objective  IInf  Best Integer    Best Bound    ItCnt     Gap

*     0+    0                           34.0000      163.0000           379.41%
      0     0      125.2083     1       34.0000      125.2083        3  268.26%
*     0+    0                          122.5000      125.2083             2.21%
      0     0        cutoff            122.5000                      3    0.00%
Elapsed time = 0.00 sec. (0.03 ticks, tree = 0.01 MB, solutions = 2)

Root node processing (before b&c):
  Real time             =    0.00 sec. (0.03 ticks)
Sequential b&c:
  Real time             =    0.00 sec. (0.00 ticks)
                          ------------
Total (root+branch&cut) =    0.00 sec. (0.03 ticks)
Ramp-up : worker 0 terminated with lpstat 101
Ramp-up : worker 1 terminated with lpstat 101
Ramp-up : worker 2 terminated with lpstat 101

Ramp-up finished (winner: 2).
Ramp-up time = 0.01 sec. (0.03 ticks)

 

2. Case 2: If I put the values for threads are 2, 3, 4 correspondingly 

Here is the result. Notice that "Parallel mode: deterministic, using up to 2 threads"

Tried aggregator 2 times.
Aggregator did 1 substitutions.
Reduced MIP has 2 rows, 3 columns, and 6 nonzeros.
Reduced MIP has 0 binaries, 1 generals, 0 SOSs, and 0 indicators.
Presolve time = 0.00 sec. (0.01 ticks)
Running distributed MIP on 3 solvers.
Setting up 3 distributed solvers.
Setup time = 0.00 sec. (0.00 ticks)
Starting ramp-up.

Found incumbent of value 34.000000 after 0.00 sec. (0.00 ticks)
MIP emphasis: balance optimality and feasibility.
MIP search method: dynamic search.
Parallel mode: deterministic, using up to 2 threads.
Root relaxation solution time = 0.00 sec. (0.00 ticks)

        Nodes                                         Cuts/
   Node  Left     Objective  IInf  Best Integer    Best Bound    ItCnt     Gap

*     0+    0                           34.0000      163.0000           379.41%
      0     0      125.2083     1       34.0000      125.2083        3  268.26%
*     0+    0                          122.5000      125.2083             2.21%
      0     0        cutoff            122.5000                      3    0.00%
Elapsed time = 0.02 sec. (0.03 ticks, tree = 0.01 MB, solutions = 2)

Root node processing (before b&c):
  Real time             =    0.02 sec. (0.03 ticks)
Parallel b&c, 2 threads:
  Real time             =    0.00 sec. (0.00 ticks)
  Sync time (average)   =    0.00 sec.
  Wait time (average)   =    0.00 sec.
                          ------------
Total (root+branch&cut) =    0.02 sec. (0.03 ticks)
Ramp-up : worker 0 terminated with lpstat 101
Ramp-up : worker 1 terminated with lpstat 101
Ramp-up : worker 2 terminated with lpstat 101

Ramp-up finished (winner: 2).
Ramp-up time = 0.04 sec. (0.04 ticks)

Solution pool: 1 solution saved.

MIP - Integer optimal solution:  Objective =  1.2250000000e+02
Solution time =   13.65 sec.  Iterations = 3  Nodes = 0
Deterministic time = 0.05 ticks  (0.00 ticks/sec)

 

3. So my questions are:

a. Could you please elaborate on the impact of setting the value for thread? How it is mapped to number of cores

b. I am going to extend this framework to HPC system and HPC system at my school works only with MPI. In order to create .vmc file for this case, it is necessary to know the host name. So, I wonder whether the host name is equivalent to the name of the node that I request from HPC. 

Here are the possible situation that I would like to work on

a. I request 1 node with 16 cores and I would like to use those 16 cores to run distributed mip algorithms instead of concurrent optimization. What the .vmc file would look like? Note that the process transport protocol is not supported at our HPC system

b. I request 2 nodes and each nodes with 8 cores. I wonder how the algorithm will treat 8 cores for each node: whether it runs concurrent optimization algorithm on each node or it treats 8 cores of each node as a single one?

c. For the case of 2 nodes and each nodes with 8 cores, I wonder how could I use such 8x2=16 cores in parallel for distributed algorithms rather than simply indicating 2 nodes for parallel? 

c. Is there any way to access to parameters for distributed algorithms (ramp-up time, etc)?

 

Thank so much

 

#DecisionOptimization
#MathematicalProgramming-General

The output you see is expected. Note that the output shown for rampup is the engine log for the first worker. So in both cases you see the number of threads you asked for on a single worker, right?

For MPI, workers are not identified by their host name in the VMC file. Instead they are identified by their MPI rank, so you don't need to know their respective host names, see documentation.

For all the questions you raised, I think you just have to specify the appropriate number of threads in the VMC file. The number of threads specified in the VMC file for a single worker is the number of threads CPLEX will use in that particular worker. For example, if you have a 16 core machine and want to run 16 single-threaded workers on that machine, then your VMC should have 16 <machine> elements, each of which has the thread count set to 1. If you have instead 2 nodes with 8 cores each, leave the thread parameters at their default values and list only two <machine>s in your VMC (the two nodes) then by default CPLEX will use 8 cores in each of the workers (where 8 is the number of cores the OS reports for each of the nodes).

#DecisionOptimization
#MathematicalProgramming-General

Originally posted by: Mr.Hai

Thanks for the detail answer. It becomes much clearer to me now. 

1. The output you see is expected. Note that the output shown for rampup is the engine log for the first worker. So in both cases you see the number of threads you asked for on a single worker, right?

Yes, it is exactly what I've seen. So, in a single local machine with multiple cores, I have no doubt anymore

 

2. I tried to use the HPC system at my school and here is the situation that I encountered:

a.  I requested 1 node with 16 cores, and 2GB for each core, 1 thread per core

b. Since the system use MPI, I used " set distmpi config mpi" to automatically generate the vmc file. 

c. However, when I submit the job and look into the cplex.log file, it turns out that: CPLEX still use concurrent optimization (16 threads). I have not seen a similar thing like that "Running distributed MIP on N solvers"

 

My questions are:

1) For a single node, Does this command "set distmpi config mpi" work? Since what I expect is that the maximum number of core (e.g., 16) will be used as distributed solver. If that command does not work for single node case, is this the right way that one must write the vmc file with 16 worker elements to  use distributed MIP algorithm?

 

2) How can I access to the vmc file generated by this command "set distmip config mpi"? 

3) If I use Matlab and also would like to use the auto-configuration vmc. How could I do instead of reading file like this: cpx.readVMConfig(vmconfig)?

 

Thanks

#DecisionOptimization
#MathematicalProgramming-General

1) For a single node, Does this command "set distmpi config mpi" work? Since what I expect is that the maximum number of core (e.g., 16) will be used as distributed solver. If that command does not work for single node case, is this the right way that one must write the vmc file with 16 worker elements to  use distributed MIP algorithm?

Whether this command works or not does not depend on the number of cores/nodes. It only depends on the number of machines in the MPI universe in which you spawned the MPI process. The command will use all machines in the MPI universe for distributed parallel MIP.

How can I access to the vmc file generated by this command "set distmip config mpi"? 

Have you tried 'display vmconfig'?

If I use Matlab and also would like to use the auto-configuration vmc. How could I do instead of reading file like this: cpx.readVMConfig(vmconfig)?

This auto-configuration is only available in the interactive.

#DecisionOptimization
#MathematicalProgramming-General