Originally posted by: ErezHadad

Hi,

I deployed Conductor on RHEL 7.2 on Intel, in two VMs - master (4GB, 2 cores), and compute (1GB, 2 cores).

My problem is - when I create Spark Instance Groups, they consistently fail a short time after being created.

What fails is either the Spark Master for batch or (if I add a notebook service), the master UI service for the notebook.

I tried with the bundled Spark 1.4.1, and also with the add-on of Spark 1.5.2, with and without Zeppelin (0.5 for 1.4.1, 0.5.5 for 1.5.2).

The root cause seems to be failure in binding to a port 7077. Here is a sample log from a SIG with only Spark 1.4.1, no notebook:

-----------------------------------------------

16/04/06 13:34:07 INFO Master: Starting Spark master at spark://ipcs-master:7077
16/04/06 13:34:07 INFO Master: Running Spark version 1.4.1
16/04/06 13:34:07 ERROR MasterWebUI: Failed to bind MasterWebUI
java.net.BindException: Address already in use: Service 'MasterUI' failed after 0 retries!
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:460)
    at sun.nio.ch.Net.bind(Net.java:452)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:253)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:86)
    at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.spark-project.jetty.server.Server.doStart(Server.java:293)
    at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:228)
    at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238)
    at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238)
    at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2018)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
    at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2009)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:117)
    at org.apache.spark.deploy.master.Master.preStart(Master.scala:271)
    at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
    at org.apache.spark.deploy.master.Master.aroundPreStart(Master.scala:80)
    at akka.actor.ActorCell.create(ActorCell.scala:580)
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
    at akka.dispatch.Mailbox.run(Mailbox.scala:219)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
16/04/06 13:34:07 INFO Utils: Shutdown hook called

-----------------------------------------------

I verified that port 7077 is indeed free (lsof -i :7077 returns no output).

I disabled SELinux in case it was the cause - still same failure.

Please help me resolve this.

Regards,

Erez Hadad

Originally posted by: ErezHadad

Problem resolved:

It seems that the deployed Spark service (I used defaults) tries to bind the Web UI of Spark ('MasterUI' in the log above) to port 8080 - which is the PMC port. Because of that, it failed.

To resolve, when you setup Spark in the SIG, click on the "Default Configuration" link, then select "Environment Variables" from the list and change the SPARK_MASTER_UI_PORT from 8080 to something else - I used 8090.

Originally posted by: Kuan.F.

Thank you for the input here. We actually have some mechanism built inside to avoid this kind of port conflict issue.

May I know what are the environment setting for your Conductor installation there? 8080 is not our default port for PMC.

Originally posted by: ErezHadad

Hi Kuan,

I have not changed any environment settings beyond the default, except disabling SSL.

Here is a documentation link showing that PMC is installed by default at port 8080 when SSL is disabled: https://www.ibm.com/support/knowledgecenter/SSVH2B_1.1.0/get_started/locating_pmc.dita?lang=en

 I will try to deploy Conductor on RHEL 7.1 in the hope that it should resolve some of the issues, and this time I will not disable SSL.

Another disturbing issue: When deploying a new SIG, port binding for Spark services (Networking->spark.port.maxRetries) is configured to 16. However, the binding operation gives up in case of used port immediately - after  0 retries (it says so specifically in the exception log) - is that also expected behavior?

Regards,

Erez Hadad

Originally posted by: Kuan.F.

Hi Erez,

    Regarding to the retry count issue, that configuration does not apply to daemons in spark instance group, coz we do our own port management on those daemons.