说明:Hadoop集群已经搭建完毕,集群上使用的Hadoop-2.5.0。 目的:在window10系统上利用Eclipse配置Hadoop开发环境,编写MapReduce关联Hadoop集群。 准备:JDK环境变量配置、Eclipse、hadoop-2.7.5.tar、hadoop-eclipse-plugin-2.7.3.jar、hadoop-common-2.7.3-bin-master.jar(hadoop-2.7.3的Hadoop不好找了,插件使用的2.7.3版本,如要版本统一可自行下载) Hadoop-2.7.5下载地址:http://mirrors.shu.edu.cn/apache/hadoop/common/ hadoop-eclipse-plugin-2.7.3.jar下载地址:http://download.csdn.net/download/u010185220/10211976
hadoop-common-2.7.3-bin-master.jar下载地址:http://download.csdn.net/download/u010185220/10212069
一、环境搭建 第一步:JDK环境变量配置、Eclipse安装,略; 第二步:Hadoop环境配置 把下载好的Hadoop版本解压的本地一目录,本人使用的是Hadoop-2.7.5。添加系统环境变量:新建变量名HADOOP_HOME,值为Hadoop的解压路径,如D:\hadoop-2.7.5。添加到path中:%HADOOP_HOME%\bin。 第三步:把hadoop-eclipse-plugin-2.7.3.jar包复制到Eclipse目录下的pluguns目录中。重启Eclipse。打开Eclipse->Prefences。可以看到左侧多出了Hadoop Map/Reduce项。
点击多出的Hadoop Map/Reduce项,添加Hadoop解压路径
第四步:解压hadoop-common-2.7.3-bin-master.7z包,把解压得到的bin目录下的hadoop.dll、hadoop.exp、hadoop.lib、winutils.exe等所有文件复制到Hadoop-2.7.5的bin目录下。再把hadoop.dll复制到C:\Windows\System32目录下。 第五步:Eclipse中依次点击:Window->Open Perspective->Map/Reduce,项目结构中出现DFS Locations结构。
第六步:Eclipse中依次点击:Window->Show View ->Other->MapReduce Tools->Map/Reduce Locations。确定(open) 下面的控制台多出了Map/Reduce Locations试图。右键Map/Reduce Locations试图的空白处,选择新建,定义Hadoop集群的链接。Location name任起,Host填写Hadoop的mater的IP地址,port是对应的端口号,这个要与集群上core-site.xml文件中的参数一致,确保能连到集群,User name任起。
core-site.xml的位置根据自己的情况确定,我的在/etc/hadoop/2.5.0.0-1245/0/下,查看方式是:cat /etc/hadoop/2.5.0.0-1245/0/core-site.xml。
填写好以上参数后点击Finish。DFS Locations下出现定义的Hadoop连接信息。点开节点会看到集群上的文件信息。看不到这连接失败,检查上步IP地址及端口的配置是否有误。
若有文件的话,点击其中的某节点中的文件确定能查看文件内容。
此时若不能查看文件内容,若提示是editor could not be initialized. org.eclipse.ui.workbench.texteditor类似的问题,则可能是C:\Windows\System32下的hadoop.dll版本和hadoop-2.7.5/bin下的hadoop.dll版本不一致的原因。 至此,window下Eclipse配置Hadoop开发环境搭建完毕。 二、WordCount示例 第一步、新建项目 :File->new->other->Map/Reduce Project 第二步、src下创建Package,Package下创建WordCount.java类 代码如下(可直接复制粘贴到你的WordCount类):
import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } @SuppressWarnings("deprecation") Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } 在src下创建log4j.properties文件,不然运行程序时候会报错。可先创建txt文本文件,添加内容后修改名字及后缀名,然后复制到项目下。内容如下: # Configure logging for testing:optionally with log file #log4j.rootLogger=debug,appender log4j.rootLogger=info,appender #log4j.rootLogger=error,appender #\u8F93\u51FA\u5230\u63A7\u5236\u53F0 log4j.appender.appender=org.apache.log4j.ConsoleAppender #\u6837\u5F0F\u4E3ATTCCLayout log4j.appender.appender.layout=org.apache.log4j.TTCCLayout
第三步:右键项目,依次Run as ->Run Configurations...->Java Application。选Java Application后点击左上角的New launch application,配置Main标签参数。填写Name(任起),Search...往下拉,找到WordCount,确定。
配置Argument标签参数。注意点在图上已经说明
配置完成后点击Apply,Run。出现类似以下日志,成功。 [pool-6-thread-1] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 115 bytes [pool-6-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit [pool-6-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 1 files, 123 bytes from disk [pool-6-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 0 segments, 0 bytes from memory into reduce [pool-6-thread-1] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments [pool-6-thread-1] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 115 bytes [pool-6-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied. [pool-6-thread-1] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords [pool-6-thread-1] INFO org.apache.hadoop.mapred.Task - Task:attempt_local1399589841_0001_r_000000_0 is done. And is in the process of committing [pool-6-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied. [pool-6-thread-1] INFO org.apache.hadoop.mapred.Task - Task attempt_local1399589841_0001_r_000000_0 is allowed to commit now [pool-6-thread-1] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local1399589841_0001_r_000000_0' to hdfs://192.168.200.240:8020/user/tws/test/_temporary/0/task_local1399589841_0001_r_000000 [pool-6-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce [pool-6-thread-1] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local1399589841_0001_r_000000_0' done. [pool-6-thread-1] INFO org.apache.hadoop.mapred.Task - Final Counters for attempt_local1399589841_0001_r_000000_0: Counters: 29 File System Counters FILE: Number of bytes read=453 FILE: Number of bytes written=292149 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=87 HDFS: Number of bytes written=81 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=3 Map-Reduce Framework Combine input records=0 Combine output records=0 Reduce input groups=9 Reduce shuffle bytes=123 Reduce input records=9 Reduce output records=9 Spilled Records=9 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 Total committed heap usage (bytes)=253231104 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Output Format Counters Bytes Written=81 此时可以在Argument指定的路径(即输出路径,hdfs://IP:端口/路径)下查看结果。
word.txt内容如下:
运行结果:
|