High Performance Computing Group

Do you run lsfstartup on LSF master host?

Hi,

There is only one master host in my setup.

Make sure no lsf service running on the node, then run lim -2 to get more debug log.

Hi,

I confirm that there is no other LSF service running.

I share below the debug log for your review:

# lim -2
Reading configuration from /usr/share/lsf/conf/lsf.conf
Oct 15 12:02:00 2025 17941 6 3.4.0 Lim starting...
Oct 15 12:02:00 2025 17941 6 3.4.0 LIM is running in advanced workload execution mode.
Oct 15 12:02:00 2025 17941 6 3.4.0 Master LIM is not running in EGO_DISABLE_UNRESOLVABLE_HOST mode.
Oct 15 12:02:00 2025 17941 5 3.4.0 lim -2
Oct 15 12:02:00 2025 17941 6 3.4.0 LIM is running as IBM Spectrum LSF Community Edition.
Oct 15 12:02:00 2025 17941 6 3.4.0 reCheckClass: numhosts 1 so reset exchIntvl to 15.00
Oct 15 12:02:06 2025 17941 4 3.4.0 initNewMaster: This is now the master host.
Oct 15 12:02:06 2025 17941 7 3.4.0 main: Is this an MQ broker candidate? 0.
Oct 15 12:02:06 2025 17945 6 3.4.0 getDaemonPath2_: Can't access /home/lsf-admin/Desktop/lsfsce10.2.0.15-x86_64/lsf/lsf10.1_lsfinstall/pim: No such file or directory. Trying /usr/share/lsf/10.1/linux3.10-glibc2.17-x86_64/etc.
Oct 15 12:02:06 2025 17945 5 3.4.0 /usr/share/lsf/10.1/linux3.10-glibc2.17-x86_64/etc/pim -2
Oct 15 12:02:11 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1
Oct 15 12:02:16 2025 17941 Last message repeated 1 time(s).
Oct 15 12:02:21 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=2
Oct 15 12:02:26 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1
Oct 15 12:02:31 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=2
Oct 15 12:02:36 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1
Oct 15 12:02:41 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=2
Oct 15 12:02:46 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1
Oct 15 12:02:51 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=2
Oct 15 12:02:56 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1
Oct 15 12:03:01 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=2
Oct 15 12:03:06 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1
Oct 15 12:03:11 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=2
Oct 15 12:03:16 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1
Oct 15 12:03:21 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=2
Oct 15 12:03:26 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1
Oct 15 12:03:31 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=2
Oct 15 12:03:36 2025 17941 7 3.4.0 announceMaster: send to 1 hosts this period, cnt=1

Hi,

I managed to start the cluster:

# lsfstartup
Starting up all LIMs ...
Do you really want to start up LIM on all hosts ? [y/n]y
Start up LIM on <lsf-server-host> ...... Warning: Permanently added 'lsf-server-host' (ED25519) to the list of known hosts.
root@lsf-server-host's password:
done

Waiting for Master LIM to start up ...  Master LIM is ok
Starting up all RESes ...
Do you really want to start up RES on all hosts ? [y/n]y
root@lsf-server-host's password:  ......
done

Starting all server daemons on LSBATCH hosts ...
Do you really want to start up server batch daemon on all hosts ? [y/n] y
root@lsf-server-host's password: lsf-server-host> ......
done

Done starting up LSF daemons on the local LSF cluster ...

However, I don't see that the LIM service in my client host starts.

Before the installation on the server host, I updated the /etc/hosts to point to the client host accordingly and I can list it by lshosts command on the master host:

# lshosts
HOST_NAME      type    model  cpuf ncpus maxmem maxswp server RESOURCES
lsf-server-  X86_64 Intel_E5  12.5     2   1.6G   1.9G    Yes (mg)
lsf-client- UNKNOWN UNKNOWN_   1.0     -      -      -     No ()

LSF services shouldn't run on LSF client host.

Hi,

Thank you for the information. Can you please help me understand how the client host communicates with the server host (and submit jobs)? I couldn't the information in the installation guide.

I run lsadmin limstartup command on the client host before running a bsub command on it. Then I get an error message when I try to run bsub -I ls command:

# lsadmin limstartup

Starting up LIM on <lsf-client-host> ...... done

$ bsub -I ls

LSF daemon (LIM) not responding ... still trying

LSF daemon (LIM) not responding ... still trying

Failed in an LSF library call: Server host LIM configuration is not ready yet. Job not submitted

As said you shouldn't start lim service on a client host. LSF client host communicates to LSF master node directly for job submission. lsf.conf file is the place LSF client host to find master list and communication ports info.  

Hi,

Thank you for the follow up. I noticed that I mistakenly allowed TCP instead of UDP connections on the server host. After fixing the firewall rules on the server host, I am able to submit jobs from the client host:

[lsf-admin@lsf-client-host ~]$ bsub -R "select[type==any]" -o 251019.txt ls

Job <215> is submitted to default queue <normal>.

[lsf-admin@lsf-server-host ~]$ cat 251019.txt

Sender: LSF System <lsf-admin@lsf-server-host>

Subject: Job 215: <ls> in cluster <lsf-cluster> Done

Job <ls> was submitted from host <lsf-client-host> by user <lsf-admin> in cluster <lsf-cluster> at Sun Oct 19 22:40:49 2025

Job was executed on host(s) <lsf-server-host>, in queue <normal>, as user <lsf-admin> in cluster <lsf-cluster> at Sun Oct 19 22:40:50 2025

</home/lsf-admin> was used as the home directory.

</home/lsf-admin> was used as the working directory.

Started at Sun Oct 19 22:40:50 2025

Terminated at Sun Oct 19 22:40:50 2025

Results reported at Sun Oct 19 22:40:50 2025

Your job looked like:

------------------------------------------------------------

# LSBATCH: User input

ls

------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time :                                   0.02 sec.

    Max Memory :                                 -

    Average Memory :                             -

    Total Requested Memory :                     -

    Delta Memory :                               -

    Max Swap :                                   -

    Max Processes :                              -

    Max Threads :                                -

    Run time :                                   2 sec.

    Turnaround time :                            1 sec.

The output (if any) follows:

setup