目录1、问题2、解决(这里先放着,只作为记录)3、通过源码和日志排查过程———————————————————————————-1、问题:

近期公司的HiveService2启动有点慢,需要10分钟.

2、解决(这里先放着,只作为记录):

原因:HiveServer2在启动的时候,需要在hivemetastore中建立一个物化视图,这就是cache.每次查询的时候,查询语句先要通过hivemetastore中的物化视图,找到数据库和表.

初步的解决办法:
注释 HiveServer2.java中的167行
HiveMaterializedViewsRegistry.get().init(sessionHive);

3、通过源码和日志排查过程:

1、使用debug启动HiveService2,(./bin/hive –service hiveserver2 –hiveconf hive.root.logger=DEBUG,console)

2、查看日志发现,HiveService2初始化时,需要读取hive中的数据库和表,部分日志如下:

2018-07-23T16:40:18,216  INFO [main] metastore.HiveMetaStore: 0: get_multi_table : db=tempenterprise tbls=edge_class_courtannouncement_defendant_2_courtannouncement,mid_e_company_investing,mid_e_company_investor,mid_e_courtaannounce_plaintiff,mid_e_courtannouncement_defendant,mid_e_courtannouncement_other,mid_e_lawinstitueannouncement_plaintiff,mid_e_lawinstituteannouncement_defendant,mid_e_lawinstituteannouncement_other,mid_e_lawsuit_defendant,mid_e_lawsuit_demandant,mid_e_lawsuit_other,mid_e_legal_representative,mid_e_personal_investing,mid_e_personal_investor,mid_e_pledge,mid_e_staff,raw_company,raw_courtannouncement,raw_investor,raw_lawinstituteannouncement,raw_lawsuit,raw_natural_person,raw_pledge,raw_pledge_info,raw_staff,v_idmapping_company,v_idmapping_courtannouncement,v_idmapping_lawinstituteannouncement,v_idmapping_lawsuit,v_idmapping_natural_person,v_idmapping_pledge2018-07-23T16:40:18,216  INFO [main] HiveMetaStore.audit: ugi=huiyu ip=unknown-ip-addr  cmd=get_multi_table : db=tempenterprise tbls=edge_class_courtannouncement_defendant_2_courtannouncement,mid_e_company_investing,mid_e_company_investor,mid_e_courtaannounce_plaintiff,mid_e_courtannouncement_defendant,mid_e_courtannouncement_other,mid_e_lawinstitueannouncement_plaintiff,mid_e_lawinstituteannouncement_defendant,mid_e_lawinstituteannouncement_other,mid_e_lawsuit_defendant,mid_e_lawsuit_demandant,mid_e_lawsuit_other,mid_e_legal_representative,mid_e_personal_investing,mid_e_personal_investor,mid_e_pledge,mid_e_staff,raw_company,raw_courtannouncement,raw_investor,raw_lawinstituteannouncement,raw_lawsuit,raw_natural_person,raw_pledge,raw_pledge_info,raw_staff,v_idmapping_company,v_idmapping_courtannouncement,v_idmapping_lawinstituteannouncement,v_idmapping_lawsuit,v_idmapping_natural_person,v_idmapping_pledge2018-07-23T16:40:18,216 DEBUG [main] metastore.ObjectStore: Open transaction: count = 1, isActive = true at:    org.apache.hadoop.hive.metastore.ObjectStore.getTableObjectsByName(ObjectStore.java:1377)2018-07-23T16:40:18,443 DEBUG [main] metastore.ObjectStore: Commit transaction: count = 0, isactive true at:    org.apache.hadoop.hive.metastore.ObjectStore.getTableObjectsByName(ObjectStore.java:1399)

• 1
• 2
• 3
• 4
• 5
• 6
  

3、源码跟踪如下所示:

3.1 HiveService2的main函数

public static void main(String[] args) {    //确认是否加载配置    HiveConf.setLoadHiveServer2Config(true);    try {      //Options初始化，以及设置 Server名称 为hiveserver2      ServerOptionsProcessor oproc = new ServerOptionsProcessor("hiveserver2");      //解析hive配置的参数      ServerOptionsProcessorResponse oprocResponse = oproc.parse(args);      // NOTE: It is critical to do this here so that log4j is reinitialized      // before any of the other core hive classes are loaded      String initLog4jMessage = LogUtils.initHiveLog4j();      LOG.debug(initLog4jMessage);      //打印启动日志      HiveStringUtils.startupShutdownMessage(HiveServer2.class, args, LOG);      // Logger debug message from "oproc" after log4j initialize properly      LOG.debug(oproc.getDebugMessage().toString());      // Call the executor which will execute the appropriate command based on the parsed options      //开启hiveserver2服务 ，启动StartOptionExecutor方法中的  startHiveServer2 方法      oprocResponse.getServerOptionsExecutor().execute();    } catch (LogInitializationException e) {      LOG.error("Error initializing log: " + e.getMessage(), e);      System.exit(-1);    }  }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
  

3.2 启动HiveServer2的方法

其中:oprocResponse.getServerOptionsExecutor().execute();执行了静态类StartOptionExecutor中的startHiveServer2方法:

    /***     * 启动 Starting HiveServer2     * @throws Throwable     省略部分代码     */    private static void startHiveServer2() throws Throwable {        long attempts = 0, maxAttempts = 1;        while (true) {            LOG.info("Starting HiveServer2");            HiveConf hiveConf = new HiveConf();            HiveServer2 server = null;            server = new HiveServer2();            //hive的初始化            server.init(hiveConf);            //开启服务 HiveServer2            server.start();        }    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
  

3.3 启动HiveServer2的时,Hive的初始化
HiveServer2.server.init(hiveConf);作用是Hive的初始化,其中要将hive的数据库和表存储到metastore中,代码如下:

  // Create views registry        try {            Hive sessionHive = Hive.get(hiveConf);            HiveMaterializedViewsRegistry.get().init(sessionHive);        } catch (HiveException e) {            throw new RuntimeException("Failed to get metastore connection", e);        }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
  

hivemetastore中建立一个物化视图,这就是cache.将hive中所有的数据库和表存储到metastore中,如果数据库中的表多的话,启动会变慢.

  /**   * Initialize the registry for the given database. It will extract the materialized views   * that are enabled for rewriting from the metastore for the current user, parse them,   * and register them in this cache.   *   * The loading process runs on the background; the method returns in the moment that the   * runnable task is created, thus the views will still not be loaded in the cache when   * it does.   */  public void init(final Hive db) {    try {      List<Table> tables = new ArrayList<Table>();      for (String dbName : db.getAllDatabases()) {        // TODO: We should enhance metastore API such that it returns only        // materialized views instead of all tables        tables.addAll(db.getAllTableObjects(dbName));      }      pool.submit(new Loader(tables));    } catch (HiveException e) {      LOG.error("Problem connecting to the metastore when initializing the view registry");    }  }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
  

3.4 HiveServer2的启动

  @Override  public synchronized void start() {    super.start();    // If we're supporting dynamic service discovery, we'll add the service uri for this    // HiveServer2 instance to Zookeeper as a znode.    HiveConf hiveConf = this.getHiveConf();    if (hiveConf.getBoolVar(ConfVars.HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY)) {      try {        //读取Zookeeper        addServerInstanceToZooKeeper(hiveConf);      } catch (Exception e) {        LOG.error("Error adding this HiveServer2 instance to ZooKeeper: ", e);        throw new ServiceException(e);      }    }    if (webServer != null) {      try {        webServer.start();        //代表hive已经启动了        LOG.info("Web UI has started on port " + webServer.getPort());      } catch (Exception e) {        LOG.error("Error starting Web UI: ", e);        throw new ServiceException(e);      }    }  }

奈斯，优秀