【西安校区】Snappy 安装 - 黑马程序员技术交流社区

1. Snappy 安装过程及验证

gcc c++, autoconf,automake, libtool, Java 6, JAVA_HOME set, Maven 3

B. 下载Snappy snappy-1.1.1.tar.gz

C. 解压，编译并安装动态链接库到本地

2. Hadoop Snappy 源码编译过程

A. 下载Hadoop-Snappy源码（hadoop-snappy-read-only）

http://code.google.com/p/hadoop-snappy/

B. 编译hadoop snappy源码,进入hadoop-snappy-read-only/下执行

mvn package[-Dsnappy.prefix=SNAPPY_INSTALLATION_DIR]

注：如果第二步snappy安装路径是默认的话，即/usr/local/lib，

则此处 [-Dsnappy.prefix=SNAPPY_INSTALLATION_DIR]可以不写，或者

-Dsnappy.prefix=/usr/local/lib

3. Hadoop上Hadoop Snappy 安装配置过程及验证

A. 解压第2步target下hadoop-snappy-0.0.1-SNAPSHOT.tar.gz，解压后，复制lib文件

cp -r /root/modules/snappy-hadoop/target/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/* $HADOOP_HOME/lib/native/Linux-amd64-64/

B. 将第2步target下的hadoop-snappy-0.0.1-SNAPSHOT.jar复制到$HADOOP_HOME/lib 下。

C. 配置hadoop-env.sh，添加：

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

[XML] 纯文本查看 复制代码

<property>

  <name>mapred.output.compress</name>

  <value>false</value>

  <description>

      Should the job outputs be compressed?

  </description>

</property>

 

<property>

  <name>mapred.output.compression.type</name>

  <value>RECORD</value>

  <description>

     If the job outputs are to compressed as SequenceFiles, how should they

     be compressed? Should be one of NONE, RECORD or BLOCK.

  </description>

</property>

 

<property>

  <name>mapred.output.compression.codec</name>

  <value>org.apache.hadoop.io.compress.DefaultCodec</value>

  <description>

        If the job outputs are compressed, how should they be compressed?

  </description>

</property>

 

<property>

  <name>mapred.compress.map.output</name>

  <value>false</value>

  <description>

         Should the outputs of the maps be compressed before being sent

     across the network. Uses SequenceFile compression.

  </description>

</property>

 

<property>

  <name>mapred.map.output.compression.codec</name>

  <value>org.apache.hadoop.io.compress.DefaultCodec</value>

  <description>

  If  the map outputs are compressed, how should they be compressed?

  </description>

</property>

E. 重新启动hadoop。为了验证是否成功，往hdfs上传一个文本文件，敲入一些词组，运行wordcount程序。如果map部分100%完成，即说明我们hadoop snappy安装成功。

4. HBase 配置Snappy及验证

A. 配置HBase lib/native/Linux-amd64-64/ 中的lib文件。HBase中的lib文件，即需要第三步中/root/modules/snappy-hadoop/target/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/ 下的所有lib文件，也需要Hadoop中，$HADOOP_HOME/lib/native/Linux-amd64-64/下的hadoop的lib 文件。简单起见，我们只需要将$HADOOP_HOME/lib/native/Linux- amd64-64/下lib文件，全部复制到相应HBase目录下：

cp -r $HADOOP_HOME/lib/native/Linux-amd64-64/* $HBASE_HOME/lib/native/Linux-amd64-64/

B. 配置HBase环境变量hbase-env.sh

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

D. 验证安装是否成功

hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/testfile snappy

hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/testfile snappy