【上海校区】Spark（2.3.1）集群安装【一篇就够】

操作系统：Linux（CenOS7）

Spark集群至少需要有JDK、Hadoop、Scala、Spark

一、安装Spark依赖的Scala（每个节点都安装）

1.1 下载Scala（2.11.12）

wget "https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz"

1.2 解压、配置环境变量

vi /etc/profile 编辑文件，追加如下内容：

# scala environment
export SCALA_HOME=/usr/scala-2.11.12
export PATH=$PATH:$SCALA_HOME/bin

1.3 验证Scala安装成功

[root@master usr]# source /etc/profile <== 使配置生效
[root@master usr]# scala -version
Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL

二、安装Spark（每个节点都安装）

2.1 下载Spark（2.3.1）

wget "http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz"

2.2 解压、配置环境变量

vi /etc/profile 编辑文件，追加如下内容：

# spark environment
export SPARK_HOME=/usr/spark-2.3.1-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin <==因为$SPARK_HOME/sbin目录下一些文件名和$HADOOP_HOME/bin目录下一些文件名相同，为了避免文件名冲突，这里不添加 $SPARK_HOME/sbin目录

2.3 配置conf目录下文件

2.3.1 新建spark-env.sh文件

[root@master conf]# cp spark-env.sh.template spark-env.sh

[root@master conf]# vi spark-env.sh 编辑文件，增加如下内容：

JAVA_HOME=/usr/java/jdk1.8.0_162

HADOOP_HOME=/usr/hadoop/hadoop-2.9.0
HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

SCALA_HOME=/usr/scala-2.11.12

SPARK_HOME=/usr/spark-2.3.1-bin-hadoop2.7
SPARK_MASTER_IP=master
SPARK_EXECUTOR_MEMORY=1G

2.3.2 新建slaves文件

[root@master conf]# cp slaves.template slaves
[root@master conf]# vi slaves 编辑文件，增加如下内容：

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# A Spark Worker will be started on each of the machines listed below.
slave1
slave2

三、启动Spark集群

因为Spark是依赖于Hadoop提供的分布式文件系统的，所以在启动Spark之前，先确保Hadoop在正常运行。（Hadoop安装与启动）

在Hadoop正常运行的情况下，在master（Hadoop的NameNode，即Spark的master节点）上，

/usr/spark-2.3.1-bin-hadoop2.7/sbin目录下执行：

[root@master sbin]# ./start-all.sh <==前面必须有 ./ ，否则会执行$HADOOP_HOME/bin下的start-all.sh脚本
starting org.apache.spark.deploy.master.Master, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out
slave1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.out
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.out

访问Spark集群URL：

四、运行Spark提供的计算圆周率的示例程序

4.1 使用单机local模式提交任务

local模式也就是本地模式，即在本地机器上单机执行程序。使用这个模式，并不需要启动Hadoop集群，也不需要启动Spark集群，只要有一台机器上安装了JDK、Scala、Spark即可运行。

[root@master sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master local /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
2018-09-09 21:21:21 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-09 21:21:22 INFO  SparkContext:54 - Running Spark version 2.3.1
2018-09-09 21:21:22 INFO  SparkContext:54 - Submitted application: Spark Pi
2018-09-09 21:21:22 INFO  SecurityManager:54 - Changing view acls to: root
2018-09-09 21:21:22 INFO  SecurityManager:54 - Changing modify acls to: root
2018-09-09 21:21:22 INFO  SecurityManager:54 - Changing view acls groups to:
2018-09-09 21:21:22 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-09-09 21:21:22 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 21:21:23 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 44606.
2018-09-09 21:21:23 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-09-09 21:21:23 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-09-09 21:21:23 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-09-09 21:21:23 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-09-09 21:21:23 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-d76c7c4a-0859-4c11-aba0-7d8e88efdec6
2018-09-09 21:21:24 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-09-09 21:21:24 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2018-09-09 21:21:24 INFO  log:192 - Logging initialized @6994ms
2018-09-09 21:21:24 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
2018-09-09 21:21:24 INFO  Server:414 - Started @7429ms
2018-09-09 21:21:25 INFO  AbstractConnector:278 - Started ServerConnector@e3cee7b{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:21:25 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6ab72419{/jobs,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51684e4a{/jobs/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6ce1f601{/jobs/job,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e886a5b{/jobs/job/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@d816dde{/stages,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e33c391{/stages/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6c451c9c{/stages/stage,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@47747fb9{/stages/stage/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3113a37{/stages/pool,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@213e3629{/stages/pool/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e9658b5{/storage,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a7b6f69{/storage/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@20312893{/storage/rdd,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@70eecdc2{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c41709a{/environment,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7db0565c{/environment/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54ec8cc9{/executors,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52eacb4b{/executors/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5528a42c{/executors/threadDump,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a551a63{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a6f5124{/static,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77b21474{/,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52d10fb8{/api,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22680f52{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60d84f61{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-09-09 21:21:25 INFO  SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar at spark://master:44606/jars/spark-examples_2.11-2.3.1.jar with timestamp 1536499285929
2018-09-09 21:21:26 INFO  Executor:54 - Starting executor ID driver on host localhost
2018-09-09 21:21:26 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46568.
2018-09-09 21:21:26 INFO  NettyBlockTransferService:54 - Server created on master:46568
2018-09-09 21:21:26 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-09-09 21:21:26 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 46568, None)
2018-09-09 21:21:27 INFO  BlockManagerMasterEndpoint:54 - Registering block manager master:46568 with 413.9 MB RAM, BlockManagerId(driver, master, 46568, None)
2018-09-09 21:21:27 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 46568, None)
2018-09-09 21:21:27 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 46568, None)
2018-09-09 21:21:27 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@18a645fd{/metrics/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:30 INFO  SparkContext:54 - Starting job: reduce at SparkPi.scala:38
2018-09-09 21:21:30 INFO  DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
2018-09-09 21:21:30 INFO  DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2018-09-09 21:21:30 INFO  DAGScheduler:54 - Parents of final stage: List()
2018-09-09 21:21:30 INFO  DAGScheduler:54 - Missing parents: List()
2018-09-09 21:21:30 INFO  DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2018-09-09 21:21:31 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)
2018-09-09 21:21:32 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)
2018-09-09 21:21:32 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on master:46568 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:21:32 INFO  SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039
2018-09-09 21:21:32 INFO  DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
2018-09-09 21:21:32 INFO  TaskSchedulerImpl:54 - Adding task set 0.0 with 2 tasks
2018-09-09 21:21:32 INFO  TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7853 bytes)
2018-09-09 21:21:32 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2018-09-09 21:21:32 INFO  Executor:54 - Fetching spark://master:44606/jars/spark-examples_2.11-2.3.1.jar with timestamp 1536499285929
2018-09-09 21:21:33 INFO  TransportClientFactory:267 - Successfully created connection to master/192.168.11.128:44606 after 146 ms (0 ms spent in bootstraps)
2018-09-09 21:21:33 INFO  Utils:54 - Fetching spark://master:44606/jars/spark-examples_2.11-2.3.1.jar to /tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5/userFiles-5292d290-2b31-4dc6-bfd6-c85e4cdaaae6/fetchFileTemp6719743492164655300.tmp
2018-09-09 21:21:34 INFO  Executor:54 - Adding file:/tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5/userFiles-5292d290-2b31-4dc6-bfd6-c85e4cdaaae6/spark-examples_2.11-2.3.1.jar to class loader
2018-09-09 21:21:34 INFO  Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 867 bytes result sent to driver
2018-09-09 21:21:34 INFO  TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7853 bytes)
2018-09-09 21:21:34 INFO  Executor:54 - Running task 1.0 in stage 0.0 (TID 1)
2018-09-09 21:21:34 INFO  TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 2050 ms on localhost (executor driver) (1/2)
2018-09-09 21:21:34 INFO  Executor:54 - Finished task 1.0 in stage 0.0 (TID 1). 867 bytes result sent to driver
2018-09-09 21:21:34 INFO  TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 318 ms on localhost (executor driver) (2/2)
2018-09-09 21:21:34 INFO  TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2018-09-09 21:21:34 INFO  DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 3.901 s
2018-09-09 21:21:34 INFO  DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 4.356085 s
Pi is roughly 3.146475732378662
2018-09-09 21:21:35 INFO  AbstractConnector:318 - Stopped Spark@e3cee7b{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:21:35 INFO  SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-09-09 21:21:35 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-09-09 21:21:35 INFO  MemoryStore:54 - MemoryStore cleared
2018-09-09 21:21:35 INFO  BlockManager:54 - BlockManager stopped
2018-09-09 21:21:35 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2018-09-09 21:21:35 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-09-09 21:21:35 INFO  SparkContext:54 - Successfully stopped SparkContext
2018-09-09 21:21:35 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-09-09 21:21:35 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5
2018-09-09 21:21:35 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-ba114487-f80c-4684-a962-870139ca248d
[root@master sbin]#

4.2 使用独立的Spark集群模式提交任务

这种模式也就是Standalone模式，使用独立的Spark集群模式提交任务，需要先启动Spark集群，但是不需要启动Hadoop集群。
启动Spark集群的方法是进入$SPARK_HOME/sbin目录下，执行start-all.sh脚本，启动成功后，可以访问下面的地址看是否成功：http://master:8080/

[root@master sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077 /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
2018-09-09 21:33:01 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-09 21:33:02 INFO  SparkContext:54 - Running Spark version 2.3.1
2018-09-09 21:33:02 INFO  SparkContext:54 - Submitted application: Spark Pi
2018-09-09 21:33:02 INFO  SecurityManager:54 - Changing view acls to: root
2018-09-09 21:33:02 INFO  SecurityManager:54 - Changing modify acls to: root
2018-09-09 21:33:02 INFO  SecurityManager:54 - Changing view acls groups to:
2018-09-09 21:33:02 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-09-09 21:33:02 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 21:33:03 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 44732.
2018-09-09 21:33:03 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-09-09 21:33:03 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-09-09 21:33:03 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-09-09 21:33:03 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-09-09 21:33:03 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-8089bd51-572f-4c01-93a8-b0a9d5d78014
2018-09-09 21:33:03 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-09-09 21:33:03 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2018-09-09 21:33:04 INFO  log:192 - Logging initialized @6786ms
2018-09-09 21:33:04 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
2018-09-09 21:33:04 INFO  Server:414 - Started @7198ms
2018-09-09 21:33:04 INFO  AbstractConnector:278 - Started ServerConnector@6b9267b{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:33:04 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4fdfa676{/jobs,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@38875e7d{/jobs/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e886a5b{/jobs/job,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e33c391{/jobs/job/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6c451c9c{/stages,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31c269fd{/stages/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@372b0d86{/stages/stage,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@213e3629{/stages/stage/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e9658b5{/stages/pool,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a7b6f69{/stages/pool/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@20312893{/storage,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@70eecdc2{/storage/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c41709a{/storage/rdd,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7db0565c{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54ec8cc9{/environment,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52eacb4b{/environment/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5528a42c{/executors,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a551a63{/executors/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a6f5124{/executors/threadDump,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1edb61b1{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@ec2bf82{/static,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41c07648{/,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1fe8d51b{/api,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@39c11e6c{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@324dcd31{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-09-09 21:33:05 INFO  SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar at spark://master:44732/jars/spark-examples_2.11-2.3.1.jar with timestamp 1536499985512
2018-09-09 21:33:06 INFO  StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://master:7077...
2018-09-09 21:33:06 INFO  TransportClientFactory:267 - Successfully created connection to master/192.168.11.128:7077 after 139 ms (0 ms spent in bootstraps)
2018-09-09 21:33:07 INFO  StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180909213306-0000
2018-09-09 21:33:07 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40542.
2018-09-09 21:33:07 INFO  NettyBlockTransferService:54 - Server created on master:40542
2018-09-09 21:33:07 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-09-09 21:33:07 INFO  StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180909213306-0000/0 on worker-20180909210806-192.168.11.129-41138 (192.168.11.129:41138) with 1 core(s)
2018-09-09 21:33:07 INFO  StandaloneSchedulerBackend:54 - Granted executor ID app-20180909213306-0000/0 on hostPort 192.168.11.129:41138 with 1 core(s), 1024.0 MB RAM
2018-09-09 21:33:07 INFO  StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180909213306-0000/1 on worker-20180909210808-192.168.11.130-41969 (192.168.11.130:41969) with 1 core(s)
2018-09-09 21:33:07 INFO  StandaloneSchedulerBackend:54 - Granted executor ID app-20180909213306-0000/1 on hostPort 192.168.11.130:41969 with 1 core(s), 1024.0 MB RAM
2018-09-09 21:33:07 INFO  StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180909213306-0000/0 is now RUNNING
2018-09-09 21:33:08 INFO  StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180909213306-0000/1 is now RUNNING
2018-09-09 21:33:08 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 40542, None)
2018-09-09 21:33:08 INFO  BlockManagerMasterEndpoint:54 - Registering block manager master:40542 with 413.9 MB RAM, BlockManagerId(driver, master, 40542, None)
2018-09-09 21:33:08 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 40542, None)
2018-09-09 21:33:08 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 40542, None)
2018-09-09 21:33:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@56ccd751{/metrics/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:10 INFO  StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
2018-09-09 21:33:13 INFO  SparkContext:54 - Starting job: reduce at SparkPi.scala:38
2018-09-09 21:33:13 INFO  DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
2018-09-09 21:33:13 INFO  DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2018-09-09 21:33:13 INFO  DAGScheduler:54 - Parents of final stage: List()
2018-09-09 21:33:13 INFO  DAGScheduler:54 - Missing parents: List()
2018-09-09 21:33:14 INFO  DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2018-09-09 21:33:15 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)
2018-09-09 21:33:16 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)
2018-09-09 21:33:16 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on master:40542 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:33:16 INFO  SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039
2018-09-09 21:33:16 INFO  DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
2018-09-09 21:33:16 INFO  TaskSchedulerImpl:54 - Adding task set 0.0 with 2 tasks
2018-09-09 21:33:19 INFO  CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.129:49616) with ID 0
2018-09-09 21:33:19 INFO  TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, 192.168.11.129, executor 0, partition 0, PROCESS_LOCAL, 7857 bytes)
2018-09-09 21:33:20 INFO  CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.130:40952) with ID 1
2018-09-09 21:33:20 INFO  TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, 192.168.11.130, executor 1, partition 1, PROCESS_LOCAL, 7857 bytes)
2018-09-09 21:33:20 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.11.129:34978 with 413.9 MB RAM, BlockManagerId(0, 192.168.11.129, 34978, None)
2018-09-09 21:33:20 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.11.130:44746 with 413.9 MB RAM, BlockManagerId(1, 192.168.11.130, 44746, None)
2018-09-09 21:33:23 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on 192.168.11.129:34978 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:33:23 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on 192.168.11.130:44746 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:33:24 INFO  TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 4487 ms on 192.168.11.129 (executor 0) (1/2)
2018-09-09 21:33:25 INFO  TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 5012 ms on 192.168.11.130 (executor 1) (2/2)
2018-09-09 21:33:25 INFO  TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2018-09-09 21:33:25 INFO  DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 10.560 s
2018-09-09 21:33:25 INFO  DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 11.549968 s
Pi is roughly 3.1398956994784974
2018-09-09 21:33:25 INFO  AbstractConnector:318 - Stopped Spark@6b9267b{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:33:25 INFO  SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-09-09 21:33:25 INFO  StandaloneSchedulerBackend:54 - Shutting down all executors
2018-09-09 21:33:25 INFO  CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asking each executor to shut down
2018-09-09 21:33:25 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-09-09 21:33:25 INFO  MemoryStore:54 - MemoryStore cleared
2018-09-09 21:33:25 INFO  BlockManager:54 - BlockManager stopped
2018-09-09 21:33:25 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2018-09-09 21:33:25 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-09-09 21:33:25 INFO  SparkContext:54 - Successfully stopped SparkContext
2018-09-09 21:33:25 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-09-09 21:33:25 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-ca417e29-82e1-4084-b518-46fde5110317
2018-09-09 21:33:25 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-bc263b74-64a0-426c-840c-74c6084d131a
[root@master sbin]#

4.3 用yarn-client模式执行计算程序

这种模式也叫On-Yarn模式，主要包括yarn-Client和yarn-Cluster两种模式。在这种模式下提交任务，需要先启动Hadoop集群，然后再启动Spark集群。

[root@master sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
2018-09-09 21:51:06 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-09 21:51:06 INFO  SparkContext:54 - Running Spark version 2.3.1
2018-09-09 21:51:06 INFO  SparkContext:54 - Submitted application: Spark Pi
2018-09-09 21:51:07 INFO  SecurityManager:54 - Changing view acls to: root
2018-09-09 21:51:07 INFO  SecurityManager:54 - Changing modify acls to: root
2018-09-09 21:51:07 INFO  SecurityManager:54 - Changing view acls groups to:
2018-09-09 21:51:07 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-09-09 21:51:07 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 21:51:07 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 37707.
2018-09-09 21:51:07 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-09-09 21:51:08 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-09-09 21:51:08 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-09-09 21:51:08 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-09-09 21:51:08 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-f4bcc309-8d7a-4238-bcfa-5587921eec3b
2018-09-09 21:51:08 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-09-09 21:51:08 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2018-09-09 21:51:09 INFO  log:192 - Logging initialized @7824ms
2018-09-09 21:51:09 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
2018-09-09 21:51:09 INFO  Server:414 - Started @8272ms
2018-09-09 21:51:09 INFO  AbstractConnector:278 - Started ServerConnector@2c444798{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:51:09 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e392345{/jobs,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1bdaa23d{/jobs/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@79f227a9{/jobs/job,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@50d68830{/jobs/job/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e53135d{/stages,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7674a051{/stages/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a7704c{/stages/stage,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@323e8306{/stages/stage/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@a23a01d{/stages/pool,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4acf72b6{/stages/pool/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7561db12{/storage,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3301500b{/storage/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@24b52d3e{/storage/rdd,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15deb1dc{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e9c413e{/environment,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@57a4d5ee{/environment/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5af5def9{/executors,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a45c42a{/executors/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@36dce7ed{/executors/threadDump,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@47a64f7d{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@33d05366{/static,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54a3ab8f{/,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1968a49c{/api,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c20be82{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-09-09 21:51:10 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@13c612bd{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-09-09 21:51:10 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-09-09 21:51:10 INFO  SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar at spark://master:37707/jars/spark-examples_2.11-2.3.1.jar with timestamp 1536501070145
2018-09-09 21:51:13 INFO  RMProxy:98 - Connecting to ResourceManager at master/192.168.11.128:8032
2018-09-09 21:51:14 INFO  Client:54 - Requesting a new application from cluster with 2 NodeManagers
2018-09-09 21:51:14 INFO  Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-09-09 21:51:14 INFO  Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead
2018-09-09 21:51:14 INFO  Client:54 - Setting up container launch context for our AM
2018-09-09 21:51:15 INFO  Client:54 - Setting up the launch environment for our AM container
2018-09-09 21:51:15 INFO  Client:54 - Preparing resources for our AM container
2018-09-09 21:51:20 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-09-09 21:51:52 INFO  Client:54 - Uploading resource file:/tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74/__spark_libs__7942925809039923414.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0002/__spark_libs__7942925809039923414.zip
2018-09-09 21:52:39 INFO  Client:54 - Uploading resource file:/tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74/__spark_conf__6224445360548244890.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0002/__spark_conf__.zip
2018-09-09 21:52:40 INFO  SecurityManager:54 - Changing view acls to: root
2018-09-09 21:52:40 INFO  SecurityManager:54 - Changing modify acls to: root
2018-09-09 21:52:40 INFO  SecurityManager:54 - Changing view acls groups to:
2018-09-09 21:52:40 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-09-09 21:52:40 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 21:52:40 INFO  Client:54 - Submitting application application_1536498065211_0002 to ResourceManager
2018-09-09 21:52:40 INFO  YarnClientImpl:273 - Submitted application application_1536498065211_0002
2018-09-09 21:52:40 INFO  SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1536498065211_0002 and attemptId None
2018-09-09 21:52:41 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:41 INFO  Client:54 -
   client token: N/A
   diagnostics: AM container is launched, waiting for AM container to Register with RM
   ApplicationMaster host: N/A
   ApplicationMaster RPC port: -1
   queue: default
   start time: 1536501160345
   final status: UNDEFINED
   tracking URL: http://master:8088/proxy/application_1536498065211_0002/
   user: root
2018-09-09 21:52:42 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:43 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:44 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:45 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:46 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:47 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:48 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:49 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:50 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:51 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:52 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:53 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:54 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:55 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:56 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:57 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:58 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:59 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:00 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:01 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:02 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:03 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:04 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:05 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:06 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:07 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:08 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:09 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:10 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:11 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:12 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:13 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:14 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:15 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:16 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:17 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:18 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:19 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:20 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:21 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:22 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:23 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:24 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:25 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:26 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:27 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:28 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:29 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:30 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:31 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:32 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:33 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:34 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:35 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:36 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:37 INFO  YarnClientSchedulerBackend:54 - Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> master, PROXY_URI_BASES -> http://master:8088/proxy/application_1536498065211_0002), /proxy/application_1536498065211_0002
2018-09-09 21:53:37 INFO  JettyUtils:54 - Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2018-09-09 21:53:37 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:38 INFO  Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:39 INFO  YarnSchedulerBackend$YarnSchedulerEndpoint:54 - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
2018-09-09 21:53:39 INFO  Client:54 - Application report for application_1536498065211_0002 (state: RUNNING)
2018-09-09 21:53:39 INFO  Client:54 -
   client token: N/A
   diagnostics: N/A
   ApplicationMaster host: 192.168.11.129
   ApplicationMaster RPC port: 0
   queue: default
   start time: 1536501160345
   final status: UNDEFINED
   tracking URL: http://master:8088/proxy/application_1536498065211_0002/
   user: root
2018-09-09 21:53:39 INFO  YarnClientSchedulerBackend:54 - Application application_1536498065211_0002 has started running.
2018-09-09 21:53:39 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39617.
2018-09-09 21:53:39 INFO  NettyBlockTransferService:54 - Server created on master:39617
2018-09-09 21:53:39 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-09-09 21:53:40 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 39617, None)
2018-09-09 21:53:40 INFO  BlockManagerMasterEndpoint:54 - Registering block manager master:39617 with 413.9 MB RAM, BlockManagerId(driver, master, 39617, None)
2018-09-09 21:53:40 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 39617, None)
2018-09-09 21:53:40 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 39617, None)
2018-09-09 21:53:40 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c2c3947{/metrics/json,null,AVAILABLE,@Spark}
2018-09-09 21:53:41 INFO  YarnClientSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
2018-09-09 21:54:01 INFO  SparkContext:54 - Starting job: reduce at SparkPi.scala:38
2018-09-09 21:54:01 INFO  DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
2018-09-09 21:54:01 INFO  DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2018-09-09 21:54:01 INFO  DAGScheduler:54 - Parents of final stage: List()
2018-09-09 21:54:01 INFO  DAGScheduler:54 - Missing parents: List()
2018-09-09 21:54:01 INFO  DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2018-09-09 21:54:14 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)
2018-09-09 21:54:14 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)
2018-09-09 21:54:14 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on master:39617 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:54:14 INFO  SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039
2018-09-09 21:54:14 INFO  DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
2018-09-09 21:54:14 INFO  YarnScheduler:54 - Adding task set 0.0 with 2 tasks
2018-09-09 21:54:32 WARN  YarnScheduler:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2018-09-09 21:54:44 WARN  YarnScheduler:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2018-09-09 21:54:46 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.129:59958) with ID 2
2018-09-09 21:54:46 INFO  TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, slave1, executor 2, partition 0, PROCESS_LOCAL, 7864 bytes)
2018-09-09 21:54:46 INFO  BlockManagerMasterEndpoint:54 - Registering block manager slave1:39770 with 413.9 MB RAM, BlockManagerId(2, slave1, 39770, None)
2018-09-09 21:54:48 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on slave1:39770 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:54:50 INFO  TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, slave1, executor 2, partition 1, PROCESS_LOCAL, 7864 bytes)
2018-09-09 21:54:50 INFO  TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 3931 ms on slave1 (executor 2) (1/2)
2018-09-09 21:54:50 INFO  TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 338 ms on slave1 (executor 2) (2/2)
2018-09-09 21:54:50 INFO  YarnScheduler:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2018-09-09 21:54:50 INFO  DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 48.440 s
2018-09-09 21:54:50 INFO  DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 48.819130 s
Pi is roughly 3.144115720578603
2018-09-09 21:54:50 INFO  AbstractConnector:318 - Stopped Spark@2c444798{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:54:50 INFO  SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-09-09 21:54:50 INFO  YarnClientSchedulerBackend:54 - Interrupting monitor thread
2018-09-09 21:54:50 INFO  YarnClientSchedulerBackend:54 - Shutting down all executors
2018-09-09 21:54:50 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down
2018-09-09 21:54:51 INFO  SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
2018-09-09 21:54:51 INFO  YarnClientSchedulerBackend:54 - Stopped
2018-09-09 21:54:51 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-09-09 21:54:51 INFO  MemoryStore:54 - MemoryStore cleared
2018-09-09 21:54:51 INFO  BlockManager:54 - BlockManager stopped
2018-09-09 21:54:51 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2018-09-09 21:54:51 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-09-09 21:54:51 INFO  SparkContext:54 - Successfully stopped SparkContext
2018-09-09 21:54:51 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-09-09 21:54:51 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74
2018-09-09 21:54:51 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-f085e710-596c-4c23-b436-c879d61d4a01
[root@master sbin]#

4.4 用yarn-cluster模式执行计算程序

[root@master sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
2018-09-09 22:00:59 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-09 22:01:03 INFO  RMProxy:98 - Connecting to ResourceManager at master/192.168.11.128:8032
2018-09-09 22:01:04 INFO  Client:54 - Requesting a new application from cluster with 2 NodeManagers
2018-09-09 22:01:04 INFO  Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-09-09 22:01:04 INFO  Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2018-09-09 22:01:04 INFO  Client:54 - Setting up container launch context for our AM
2018-09-09 22:01:04 INFO  Client:54 - Setting up the launch environment for our AM container
2018-09-09 22:01:04 INFO  Client:54 - Preparing resources for our AM container
2018-09-09 22:01:09 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-09-09 22:01:21 INFO  Client:54 - Uploading resource file:/tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25/__spark_libs__624052493228396690.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0003/__spark_libs__624052493228396690.zip
2018-09-09 22:02:25 INFO  Client:54 - Uploading resource file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0003/spark-examples_2.11-2.3.1.jar
2018-09-09 22:02:40 INFO  Client:54 - Uploading resource file:/tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25/__spark_conf__5330471617982538290.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0003/__spark_conf__.zip
2018-09-09 22:02:40 INFO  SecurityManager:54 - Changing view acls to: root
2018-09-09 22:02:40 INFO  SecurityManager:54 - Changing modify acls to: root
2018-09-09 22:02:40 INFO  SecurityManager:54 - Changing view acls groups to:
2018-09-09 22:02:40 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-09-09 22:02:40 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 22:02:40 INFO  Client:54 - Submitting application application_1536498065211_0003 to ResourceManager
2018-09-09 22:02:40 INFO  YarnClientImpl:273 - Submitted application application_1536498065211_0003
2018-09-09 22:02:41 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:41 INFO  Client:54 -
   client token: N/A
   diagnostics: AM container is launched, waiting for AM container to Register with RM
   ApplicationMaster host: N/A
   ApplicationMaster RPC port: -1
   queue: default
   start time: 1536501760869
   final status: UNDEFINED
   tracking URL: http://master:8088/proxy/application_1536498065211_0003/
   user: root
2018-09-09 22:02:42 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:43 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:44 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:46 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:47 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:48 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:49 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:50 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:51 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:52 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:53 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:54 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:55 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:56 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:57 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:58 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:59 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:00 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:01 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:02 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:03 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:04 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:05 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:06 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:07 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:08 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:09 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:10 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:11 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:12 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:13 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:14 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:15 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:16 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:17 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:18 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:19 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:20 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:21 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:22 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:23 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:24 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:25 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:26 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:27 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:28 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:29 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:30 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:31 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:32 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:33 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:34 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:35 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:36 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:37 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:38 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:39 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:40 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:41 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:42 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:43 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:44 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:45 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:46 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:47 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:48 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:49 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:50 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:51 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:52 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:53 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:54 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:55 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:56 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:57 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:58 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:59 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:00 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:01 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:02 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:03 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:04 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:05 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:06 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:07 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:08 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:09 INFO  Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:10 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:10 INFO  Client:54 -
   client token: N/A
   diagnostics: N/A
   ApplicationMaster host: 192.168.11.130
   ApplicationMaster RPC port: 0
   queue: default
   start time: 1536501760869
   final status: UNDEFINED
   tracking URL: http://master:8088/proxy/application_1536498065211_0003/
   user: root
2018-09-09 22:04:11 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:12 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:13 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:14 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:15 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:16 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:17 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:18 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:19 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:20 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:21 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:22 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:23 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:24 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:25 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:26 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:27 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:28 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:29 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:30 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:31 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:32 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:33 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:34 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:35 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:36 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:37 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:38 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:39 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:40 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:41 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:42 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:43 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:46 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:47 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:48 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:49 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:50 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:51 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:53 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:54 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:55 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:56 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:57 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:58 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:59 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:00 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:01 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:02 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:03 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:04 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:05 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:06 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:07 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:08 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:10 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:11 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:12 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:13 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:14 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:15 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:16 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:17 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:18 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:19 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:20 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:21 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:22 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:23 INFO  Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:24 INFO  Client:54 - Application report for application_1536498065211_0003 (state: FINISHED)
2018-09-09 22:05:24 INFO  Client:54 -
   client token: N/A
   diagnostics: N/A
   ApplicationMaster host: 192.168.11.130
   ApplicationMaster RPC port: 0
   queue: default
   start time: 1536501760869
   final status: SUCCEEDED
   tracking URL: http://master:8088/proxy/application_1536498065211_0003/
   user: root
2018-09-09 22:05:24 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-09-09 22:05:24 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-0f174e46-416d-45dd-bb59-51c652b4a70d
2018-09-09 22:05:24 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25
[root@master sbin]#
使用yarn-cluster模式计算，结果没有输出在控制台，结果写在了Hadoop集群的日志中

点击红框application id

点击Logs

点击stdout

完！！！

不二晨 · 不二晨

很不错，受教了

帐号		自动登录	找回密码
密码			加入黑马

【上海校区】Spark（2.3.1）集群安装【一篇就够】

1 个回复