黑马程序员技术交流社区

标题: 【上海校区】大数据平台运维之Sqoop [打印本页]

作者: 不二晨 时间: 2018-12-17 10:22
标题: 【上海校区】大数据平台运维之Sqoop
本帖最后由不二晨于 2018-12-18 17:56 编辑

在master和slaver节点安装Sqoop Clients，完成后，在master节点查看Sqoop的版本信息，将操作命令和输出结果以文本形式提交到答题框中。

[root@master ~]# sqoop version

Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist!Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumuloinstallation.

17/05/07 06:56:25 INFO sqoop.Sqoop: Running Sqoopversion: 1.4.6.2.4.3.0-227

Sqoop 1.4.6.2.4.3.0-227

git commit id d296ad374bd38a1c594ef0f5a2d565d71e798aa6

Compiled by jenkins on Sat Sep 10 00:58:52 UTC 2016

38.使用Sqoop工具列出master节点中MySQL中所有数据库，将操作命令和输出结果以文本形式提交到答题框中。

[root@master ~]# sqoop list-databases --connectjdbc:mysql://localhost --username root --password bigdata

Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist!Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumuloinstallation.

17/05/07 07:02:34 INFO sqoop.Sqoop: Running Sqoopversion: 1.4.6.2.4.3.0-227

17/05/07 07:02:34 WARN tool.BaseSqoopTool: Settingyour password on the command-line is insecure. Consider using -P instead.

17/05/07 07:02:34 INFO manager.MySQLManager: Preparingto use a MySQL streaming resultset.

information_schema

ambari

mysql

performance_schema

39.使用Sqoop工具列出master节点中MySQL中ambari数据库中所有的数据表，将操作命令和输出结果以文本形式提交到答题框中。

[root@master ~]# sqoop list-tables --connectjdbc:mysql://localhost/ambari --username root --password bigdata

Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist!Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumuloinstallation.

17/05/07 07:07:01 INFO sqoop.Sqoop: Running Sqoopversion: 1.4.6.2.4.3.0-227

17/05/07 07:07:01 WARN tool.BaseSqoopTool: Settingyour password on the command-line is insecure. Consider using -P instead.

17/05/07 07:07:02 INFO manager.MySQLManager: Preparingto use a MySQL streaming resultset.

ClusterHostMapping

QRTZ_BLOB_TRIGGERS

QRTZ_CALENDARS

QRTZ_CRON_TRIGGERS

QRTZ_FIRED_TRIGGERS

QRTZ_JOB_DETAILS

QRTZ_LOCKS

QRTZ_PAUSED_TRIGGER_GRPS

QRTZ_SCHEDULER_STATE

QRTZ_SIMPLE_TRIGGERS

QRTZ_SIMPROP_TRIGGERS

QRTZ_TRIGGERS

adminpermission

adminprincipal

adminprincipaltype

adminprivilege

adminresource

adminresourcetype

alert_current

alert_definition

alert_group

alert_group_target

alert_grouping

alert_history

alert_notice

alert_target

alert_target_states

ambari_sequences

artifact

blueprint

blueprint_configuration

clusterEvent

cluster_version

clusterconfig

clusterconfigmapping

clusters

clusterservices

clusterstate

confgroupclusterconfigmapping

configgroup

configgrouphostmapping

execution_command

groups

hdfsEvent

host_role_command

host_version

hostcomponentdesiredstate

hostcomponentstate

hostconfigmapping

hostgroup

hostgroup_component

hostgroup_configuration

hosts

hoststate

job

kerberos_descriptor

kerberos_principal

kerberos_principal_host

key_value_store

mapreduceEvent

members

metainfo

repo_version

request

requestoperationlevel

requestresourcefilter

requestschedule

requestschedulebatchrequest

role_success_criteria

servicecomponentdesiredstate

serviceconfig

serviceconfighosts

serviceconfigmapping

servicedesiredstate

stack

stage

task

taskAttempt

topology_host_info

topology_host_request

topology_host_task

topology_hostgroup

topology_logical_request

topology_logical_task

topology_request

upgrade

upgrade_group

upgrade_item

users

viewentity

viewinstance

viewinstancedata

viewinstanceproperty

viewmain

viewparameter

viewresource

widget

widget_layout

widget_layout_user_widget

workflow

40.在MySQL中创建名为xiandian的数据库，在xiandian数据库中创建xd_phy_course数据表，其数据表结构如表1所示。使用Hive工具来创建数据表xd_phy_course，将phy_course_xd.txt导入到该表中，其中xd_phy_course表的数据结构如表2所示。使用Sqoop工具将hive数据仓库中的xd_phy_course表导出到master节点的MySQL中xiandain 数据库的xd_phy_course表。将以上操作命令和输出结果以文本形式提交到答题框中。

[root@master ~]# mysql -uroot -pbigdata

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 37

Server version: 5.5.44-MariaDB MariaDB Server

Copyright (c) 2000, 2015, Oracle, MariaDB CorporationAb and others.

Type 'help;' or '\h' for help. Type '\c' to clear thecurrent input statement.

MariaDB [(none)]> create database xiandian;

Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> use xiandian;

Database changed

MariaDB [xiandian]> create table xd_phy_course(stnamevarchar(20),stID int(1),class varchar(20),opt_cour varchar(20));

Query OK, 0 rows affected (0.20 sec)

hive> create table xd_phy_course3 (stnamestring,stID int,class string,opt_cour string) row format delimited fieldsterminated by '\t' lines terminated by '\n';

OK

Time taken: 3.136 seconds

hive> load data local inpath'/root/phy_course_xd.txt' into table xd_phy_course3;

Loading data to table default.xd_phy_course3

Table default.xd_phy_course3 stats: [numFiles=1,totalSize=89444]

OK

Time taken: 1.129 seconds

[root@master ~]# sqoop export --connectjdbc:mysql://localhost:3306/xiandian --username root --password bigdata --tablexd_phy_course --hcatalog-database sopdm --hcatalog-table xd_phy_course

Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist!Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumuloinstallation.

17/05/07 07:28:29 INFO sqoop.Sqoop: Running Sqoopversion: 1.4.6.2.4.3.0-227

17/05/07 07:28:29 WARN tool.BaseSqoopTool: Settingyour password on the command-line is insecure. Consider using -P instead.

17/05/07 07:28:30 INFO manager.MySQLManager: Preparingto use a MySQL streaming resultset.

17/05/07 07:28:30 INFO tool.CodeGenTool: Beginningcode generation

17/05/07 07:28:30 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:28:30 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:28:30 INFO orm.CompilationManager:HADOOP_MAPRED_HOME is /usr/hdp/2.4.3.0-227/hadoop-mapreduce

Note:/tmp/sqoop-root/compile/8a4718bb8203f0691782456590ed01ec/xd_phy_course.javauses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

17/05/07 07:28:32 INFO orm.CompilationManager: Writingjar file:/tmp/sqoop-root/compile/8a4718bb8203f0691782456590ed01ec/xd_phy_course.jar

17/05/07 07:28:32 INFO mapreduce.ExportJobBase:Beginning export of xd_phy_course

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in[jar:file:/usr/hdp/2.4.3.0-227/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindingsfor an explanation.

SLF4J: Actual binding is of type[org.slf4j.impl.Log4jLoggerFactory]

17/05/07 07:28:32 INFO mapreduce.ExportJobBase:Configuring HCatalog for export job

17/05/07 07:28:32 INFO hcat.SqoopHCatUtilities: ConfiguringHCatalog specific details for job

17/05/07 07:28:32 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:28:32 INFO hcat.SqoopHCatUtilities:Database column names projected : [stname, stid, class, opt_cour]

17/05/07 07:28:32 INFO hcat.SqoopHCatUtilities:Database column name - info map :

      opt_cour: [Type : 12,Precision : 20,Scale : 0]

      stid :[Type : 4,Precision : 1,Scale : 0]

      class :[Type : 12,Precision : 20,Scale : 0]

      stname :[Type : 12,Precision : 20,Scale : 0]

17/05/07 07:28:33 INFO hive.metastore: Trying toconnect to metastore with URI thrift://slaver1:9083

17/05/07 07:28:33 INFO hive.metastore: Connected tometastore.

17/05/07 07:28:33 ERROR tool.ExportTool: EncounteredIOException running export job: java.io.IOException:NoSuchObjectException(message:sopdm.xd_phy_course table not found)

[root@master ~]# sqoop export --connectjdbc:mysql://localhost:3306/xiandian --username root --password bigdata --tablexd_phy_course --hcatalog-table xd_phy_course  Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist! Accumulo importswill fail.

Please set $ACCUMULO_HOME to the root of your Accumuloinstallation.

17/05/07 07:29:48 INFO sqoop.Sqoop: Running Sqoopversion: 1.4.6.2.4.3.0-227

17/05/07 07:29:48 WARN tool.BaseSqoopTool: Settingyour password on the command-line is insecure. Consider using -P instead.

17/05/07 07:29:48 INFO manager.MySQLManager: Preparingto use a MySQL streaming resultset.

17/05/07 07:29:48 INFO tool.CodeGenTool: Beginningcode generation

17/05/07 07:29:48 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:29:48 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:29:48 INFO orm.CompilationManager:HADOOP_MAPRED_HOME is /usr/hdp/2.4.3.0-227/hadoop-mapreduce

Note:/tmp/sqoop-root/compile/35d4b31b4d93274ba6bde54b3e56a821/xd_phy_course.javauses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

17/05/07 07:29:50 INFO orm.CompilationManager: Writingjar file:/tmp/sqoop-root/compile/35d4b31b4d93274ba6bde54b3e56a821/xd_phy_course.jar

17/05/07 07:29:50 INFO mapreduce.ExportJobBase:Beginning export of xd_phy_course

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in[jar:file:/usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in[jar:file:/usr/hdp/2.4.3.0-227/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Seehttp://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type[org.slf4j.impl.Log4jLoggerFactory]

17/05/07 07:29:50 INFO mapreduce.ExportJobBase:Configuring HCatalog for export job

17/05/07 07:29:50 INFO hcat.SqoopHCatUtilities:Configuring HCatalog specific details for job

17/05/07 07:29:50 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:29:50 INFO hcat.SqoopHCatUtilities:Database column names projected : [stname, stid, class, opt_cour]

17/05/07 07:29:50 INFO hcat.SqoopHCatUtilities:Database column name - info map :

      opt_cour: [Type : 12,Precision : 20,Scale : 0]

      stid :[Type : 4,Precision : 1,Scale : 0]

      class :[Type : 12,Precision : 20,Scale : 0]

      stname :[Type : 12,Precision : 20,Scale : 0]

17/05/07 07:29:51 INFO hive.metastore: Trying toconnect to metastore with URI thrift://slaver1:9083

17/05/07 07:29:51 INFO hive.metastore: Connected tometastore.

17/05/07 07:29:51 INFO hcat.SqoopHCatUtilities:HCatalog full table schema fields = [stname, stid, class, opt_cour]

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities:HCatalog table partitioning key fields = []

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities:HCatalog projected schema fields = [stname, stid, class, opt_cour]

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities:HCatalog job : Hive Home = /usr/hdp/current/hive-client

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities:HCatalog job:  HCatalog Home =/usr/hdp/2.4.3.0-227//sqoop/../hive-hcatalog

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingjar files under /usr/hdp/2.4.3.0-227//sqoop/../hive-hcatalog/share/hcatalog todistributed cache

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/2.4.3.0-227/hive-hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/2.4.3.0-227/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/2.4.3.0-227/hive-hcatalog/share/hcatalog/hive-hcatalog-pig-adapter-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/2.4.3.0-227/hive-hcatalog/share/hcatalog/hive-hcatalog-pig-adapter.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/2.4.3.0-227/hive-hcatalog/share/hcatalog/hive-hcatalog-server-extensions-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/2.4.3.0-227/hive-hcatalog/share/hcatalog/hive-hcatalog-server-extensions.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/2.4.3.0-227/hive-hcatalog/share/hcatalog/hive-hcatalog-streaming-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/2.4.3.0-227/hive-hcatalog/share/hcatalog/hive-hcatalog-streaming.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingjar files under /usr/hdp/2.4.3.0-227//sqoop/../hive-hcatalog/lib to distributedcache

17/05/07 07:29:52 WARN hcat.SqoopHCatUtilities: Nofiles under /usr/hdp/2.4.3.0-227/sqoop/../hive-hcatalog/lib to add todistributed cache for hcatalog job

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingjar files under /usr/hdp/current/hive-client/lib to distributed cache

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/joda-time-2.5.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/commons-io-2.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-jdbc.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/ranger-hive-plugin-shim-0.5.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/commons-lang-2.6.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-metastore.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/ranger-plugin-classloader-0.5.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-beeline.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/commons-logging-1.1.3.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/curator-framework-2.6.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-exec-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/commons-math-2.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-serde.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-jdbc-1.2.1000.2.4.3.0-227-standalone.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/curator-recipes-2.6.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-jdbc-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/commons-pool-1.5.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/jta-1.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-metastore-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/geronimo-annotation_1.0_spec-1.1.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-serde-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/commons-vfs2-2.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/jpam-1.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-service-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-service.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/libfb303-0.9.2.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-shims-common.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/libthrift-0.9.2.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-shims-scheduler.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/log4j-1.2.16.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/httpclient-4.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/ST4-4.0.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/geronimo-jaspic_1.0_spec-1.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/accumulo-core-1.7.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/geronimo-jta_1.1_spec-1.1.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/accumulo-fate-1.7.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/groovy-all-2.4.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/oro-2.0.8.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/accumulo-start-1.7.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/guava-14.0.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/accumulo-trace-1.7.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-shims.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/activation-1.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/httpcore-4.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/ant-1.9.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-cli-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/ant-launcher-1.9.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/ivy-2.4.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/antlr-2.7.7.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-cli.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/antlr-runtime-3.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-common-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-ant-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/apache-log4j-extras-1.2.17.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/janino-2.7.6.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/asm-commons-3.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/jcommander-1.32.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/asm-tree-3.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/jdo-api-3.0.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/avro-1.7.5.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-common.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/bonecp-0.8.0.RELEASE.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-accumulo-handler-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/calcite-avatica-1.2.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/calcite-core-1.2.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-ant.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/json-20090211.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/calcite-linq4j-1.2.0.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/maven-scm-api-1.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/commons-cli-1.2.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-exec.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/xz-1.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/commons-codec-1.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-beeline-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/commons-collections-3.2.2.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-contrib.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/commons-compiler-2.7.6.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-hbase-handler.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/commons-compress-1.4.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/jetty-all-7.6.0.v20120127.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/commons-dbcp-1.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-shims-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/curator-client-2.6.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/datanucleus-api-jdo-3.2.6.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-hwi-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/datanucleus-core-3.2.10.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-hwi.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/datanucleus-rdbms-3.2.9.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/jline-2.12.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/derby-10.10.2.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-contrib-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/eigenbase-properties-1.1.5.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-hbase-handler-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/hive-shims-0.20S-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-shims-0.23-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-shims-common-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/hive-shims-scheduler-1.2.1000.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/htrace-core-3.1.0-incubating.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/jetty-all-server-7.6.0.v20120127.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/jsr305-3.0.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/mail-1.4.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/maven-scm-provider-svn-commons-1.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/maven-scm-provider-svnexe-1.4.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/netty-3.7.0.Final.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/opencsv-2.3.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/paranamer-2.3.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/parquet-hadoop-bundle-1.6.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/plexus-utils-1.5.6.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/protobuf-java-2.5.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/regexp-1.3.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/servlet-api-2.5.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/snappy-java-1.0.5.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/stax-api-1.0.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/stringtemplate-3.2.1.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/super-csv-2.2.0.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath: file:/usr/hdp/current/hive-client/lib/velocity-1.5.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingto job classpath:file:/usr/hdp/current/hive-client/lib/zookeeper-3.4.6.2.4.3.0-227.jar

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: Addingjar files under/usr/hdp/2.4.3.0-227//sqoop/../hive-hcatalog/share/hcatalog/storage-handlers todistributed cache (recursively)

17/05/07 07:29:52 WARN hcat.SqoopHCatUtilities: Nofiles under/usr/hdp/2.4.3.0-227/sqoop/../hive-hcatalog/share/hcatalog/storage-handlers toadd to distributed cache for hcatalog job

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities: ConfiguringHCatalog for export job

17/05/07 07:29:52 INFO hcat.SqoopHCatUtilities:Ignoring configuration request for HCatalog info

17/05/07 07:29:53 INFO impl.TimelineClientImpl:Timeline service address: http://slaver1:8188/ws/v1/timeline/

17/05/07 07:29:53 INFO client.RMProxy: Connecting toResourceManager at slaver1/10.0.0.15:8050

17/05/07 07:30:31 INFO mapred.FileInputFormat: Totalinput paths to process : 1

17/05/07 07:30:31 INFO mapreduce.JobSubmitter: numberof splits:1

17/05/07 07:30:32 INFO mapreduce.JobSubmitter:Submitting tokens for job: job_1494136863427_0004

17/05/07 07:30:33 INFO impl.YarnClientImpl: Submittedapplication application_1494136863427_0004

17/05/07 07:30:33 INFO mapreduce.Job: The url to trackthe job: http://slaver1:8088/proxy/application_1494136863427_0004/

17/05/07 07:30:33 INFO mapreduce.Job: Running job:job_1494136863427_0004

17/05/07 07:30:46 INFO mapreduce.Job: Jobjob_1494136863427_0004 running in uber mode : false

17/05/07 07:30:46 INFO mapreduce.Job:  map 0% reduce 0%

17/05/07 07:31:50 INFO mapreduce.Job: Task Id :attempt_1494136863427_0004_m_000000_0, Status : FAILED

Exception from container-launch.

Container id:container_e03_1494136863427_0004_01_000002

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

      atorg.apache.hadoop.util.Shell.runCommand(Shell.java:600)

      atorg.apache.hadoop.util.Shell.run(Shell.java:511)

      atorg.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:783)

      atorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)

      atorg.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)

      atorg.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

      atjava.util.concurrent.FutureTask.run(FutureTask.java:266)

      atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      atjava.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

17/05/07 07:31:57 INFO mapreduce.Job:  map 100% reduce 0%

17/05/07 07:31:58 INFO mapreduce.Job: Jobjob_1494136863427_0004 completed successfully

17/05/07 07:31:58 INFO mapreduce.Job: Counters: 32

      FileSystem Counters

            FILE: Number of bytes read=0

            FILE: Number of bytes written=301829

            FILE: Number of read operations=0

            FILE: Number of large read operations=0

            FILE: Number of write operations=0

            HDFS: Number of bytes read=90953

            HDFS: Number of bytes written=0

            HDFS: Number of read operations=2

            HDFS: Number of large read operations=0

            HDFS: Number of write operations=0

      JobCounters

            Failed map tasks=1

            Launched map tasks=2

            Other local map tasks=1

            Data-local map tasks=1

            Total time spent by all maps in occupied slots (ms)=199983

            Total time spent by all reduces in occupied slots (ms)=0

            Total time spent by all map tasks (ms)=66661

            Total vcore-seconds taken by all map tasks=66661

            Total megabyte-seconds taken by all map tasks=102391296

   Map-Reduce Framework

            Map input records=2050

            Map output records=2050

            Input split bytes=1509

            Spilled Records=0

            Failed Shuffles=0

            Merged Map outputs=0

            GC time elapsed (ms)=69

            CPU time spent (ms)=2560

            Physical memory (bytes)snapshot=278097920

            Virtual memory (bytes) snapshot=3273080832

            Total committed heap usage (bytes)=152043520

      FileInput Format Counters

            Bytes Read=0

      FileOutput Format Counters

            Bytes Written=0

17/05/07 07:31:58 INFO mapreduce.ExportJobBase:Transferred 88.8213 KB in 125.923 seconds (722.2906 bytes/sec)

17/05/07 07:31:58 INFO mapreduce.ExportJobBase:Exported 2050 records.

41.在Hive中创建xd_phy_course数据表，其数据表结构如下表所示。使用Sqoop工具将MySQL中xiandian数据库下xd_phy_course表导入到Hive数据仓库中的xd_phy_course表中。

新：

[root@master ~]# hive

WARNING: Use "yarn jar" to launch YARNapplications.

Logging initialized using configuration in file:/etc/hive/2.4.3.0-227/0/hive-log4j.properties

hive> create table xd_phy_course4 (stnamestring,stID int,class string,opt_cour string) row format delimited fieldsterminated by '\t' lines terminated by '\n';

OK

Time taken: 2.329 seconds

[root@master ~]# sqoop import --connectjdbc:mysql://localhost:3306/xiandian --username root --password bigdata --tablexd_phy_course --hive-import --hive-overwrite --hive-table xd_phy_course4 -m 1--fields-terminated-by '\t' --lines-terminated-by '\n'

Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist!Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumuloinstallation.

17/05/07 07:52:37 INFO sqoop.Sqoop: Running Sqoopversion: 1.4.6.2.4.3.0-227

17/05/07 07:52:37 WARN tool.BaseSqoopTool: Settingyour password on the command-line is insecure. Consider using -P instead.

17/05/07 07:52:37 INFO manager.MySQLManager: Preparingto use a MySQL streaming resultset.

17/05/07 07:52:37 INFO tool.CodeGenTool: Beginningcode generation

17/05/07 07:52:37 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:52:37 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:52:37 INFO orm.CompilationManager: HADOOP_MAPRED_HOMEis /usr/hdp/2.4.3.0-227/hadoop-mapreduce

Note:/tmp/sqoop-root/compile/72163bd11b0ea7dbb76333cc4f9f7c11/xd_phy_course.javauses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

17/05/07 07:52:39 INFO orm.CompilationManager: Writingjar file:/tmp/sqoop-root/compile/72163bd11b0ea7dbb76333cc4f9f7c11/xd_phy_course.jar

17/05/07 07:52:39 WARN manager.MySQLManager: It lookslike you are importing from mysql.

17/05/07 07:52:39 WARN manager.MySQLManager: This transfercan be faster! Use the --direct

17/05/07 07:52:39 WARN manager.MySQLManager: option toexercise a MySQL-specific fast path.

17/05/07 07:52:39 INFO manager.MySQLManager: Settingzero DATETIME behavior to convertToNull (mysql)

17/05/07 07:52:39 INFO mapreduce.ImportJobBase:Beginning import of xd_phy_course

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in[jar:file:/usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in[jar:file:/usr/hdp/2.4.3.0-227/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Seehttp://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

17/05/07 07:52:41 INFO impl.TimelineClientImpl:Timeline service address: http://slaver1:8188/ws/v1/timeline/

17/05/07 07:52:41 INFO client.RMProxy: Connecting toResourceManager at slaver1/10.0.0.15:8050

17/05/07 07:52:43 INFO db.DBInputFormat: Using readcommited transaction isolation

17/05/07 07:52:43 INFO mapreduce.JobSubmitter: numberof splits:1

17/05/07 07:52:44 INFO mapreduce.JobSubmitter:Submitting tokens for job: job_1494136863427_0006

17/05/07 07:52:44 INFO impl.YarnClientImpl: Submittedapplication application_1494136863427_0006

17/05/07 07:52:44 INFO mapreduce.Job: The url to trackthe job: http://slaver1:8088/proxy/application_1494136863427_0006/

17/05/07 07:52:44 INFO mapreduce.Job: Running job:job_1494136863427_0006

17/05/07 07:53:11 INFO mapreduce.Job: Jobjob_1494136863427_0006 running in uber mode : false

17/05/07 07:53:11 INFO mapreduce.Job:  map 0% reduce 0%

17/05/07 07:53:41 INFO mapreduce.Job:  map 100% reduce 0%

17/05/07 07:53:42 INFO mapreduce.Job:  map 0% reduce 0%

17/05/07 07:53:42 INFO mapreduce.Job: Task Id :attempt_1494136863427_0006_m_000000_0, Status : FAILED

Exception from container-launch.

Container id:container_e03_1494136863427_0006_01_000002

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

      atorg.apache.hadoop.util.Shell.runCommand(Shell.java:600)

      atorg.apache.hadoop.util.Shell.run(Shell.java:511)

      atorg.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:783)

      atorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)

      atorg.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)

      atorg.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

      atjava.util.concurrent.FutureTask.run(FutureTask.java:266)

      atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      atjava.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

17/05/07 07:54:09 INFO mapreduce.Job:  map 100% reduce 0%

17/05/07 07:56:36 INFO mapreduce.Job: Jobjob_1494136863427_0006 completed successfully

17/05/07 07:56:36 INFO mapreduce.Job: Counters: 31

      FileSystem Counters

            FILE: Number of bytes read=0

            FILE: Number of bytes written=152552

            FILE: Number of read operations=0

            FILE: Number of large read operations=0

            FILE: Number of write operations=0

            HDFS: Number of bytes read=87

            HDFS: Number of bytes written=89444

            HDFS: Number of read operations=4

            HDFS: Number of large read operations=0

            HDFS: Number of write operations=2

      JobCounters

            Failed map tasks=1

            Launched map tasks=2

            Other local map tasks=2

            Total time spent by all maps in occupied slots (ms)=107508

            Total time spent by all reduces in occupied slots (ms)=0

            Total time spent by all map tasks (ms)=35836

            Total vcore-seconds taken by all maptasks=35836

            Total megabyte-seconds taken by all map tasks=55044096

   Map-Reduce Framework

            Map input records=2050

            Map output records=2050

            Input split bytes=87

            Spilled Records=0

            Failed Shuffles=0

            Merged Map outputs=0

            GC time elapsed (ms)=54

            CPU time spent (ms)=1650

            Physical memory (bytes) snapshot=234340352

            Virtual memory (bytes) snapshot=3241504768

            Total committed heap usage (bytes)=166723584

      FileInput Format Counters

            Bytes Read=0

      FileOutput Format Counters

            Bytes Written=89444

17/05/07 07:56:36 INFO mapreduce.ImportJobBase:Transferred 87.3477 KB in 236.0327 seconds (378.9476 bytes/sec)

17/05/07 07:56:36 INFO mapreduce.ImportJobBase:Retrieved 2050 records.

17/05/07 07:56:36 INFO manager.SqlManager: ExecutingSQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1

17/05/07 07:56:36 INFO hive.HiveImport: Loadinguploaded data into Hive

Logging initialized using configuration in jar:file:/usr/hdp/2.4.3.0-227/hive/lib/hive-common-1.2.1000.2.4.3.0-227.jar!/hive-log4j.properties

OK

Time taken: 1.48 seconds

Loading data to table default.xd_phy_course4

Table default.xd_phy_course4 stats: [numFiles=1,totalSize=89444]

OK
---------------------
【转载】仅作分享，侵删
作者：飞鱼是两个人的名字
原文：https://blog.csdn.net/KamRoseLee/article/details/80269627

作者: 不二晨 时间: 2018-12-18 17:56

奈斯

作者: 梦缠绕的时候 时间: 2018-12-20 16:43

欢迎光临黑马程序员技术交流社区 (http://bbs.itheima.com/)

黑马程序员IT技术论坛 X3.2