JAVA高级面试进阶训练营视频教程

admin

　　背景：hadoop版本为3.1.3, 且以伪分布式形式安装，hive版本为3.1.2，hive为hadoop的一个客户端。

1. 安装简要步骤

(1) 官网下载apache-hive-3.1.2-bin.tar.gz，并解压缩

(2) 进入conf配置文件夹，将hive-env.sh.template重命名为hive-env.sh，并在其中增加如下配置

a. hadoop安装目录：export HADOOP_HOME=/home/ws/module/hadoop

b. hive配置目录：export HIVE_CONF_DIR=/home/ws/module/hive/conf

c. hive依赖包：export HIVE_AUX_JARS_PATH=/home/ws/module/hive/lib

(3) 将hive-default.xml.template重命名为hive-site.xml

1) 如果想使用默认的derby数据库，可以修改如下地方：

a. 将配置文件中如下文本中for和transactional的非法字符删除或替换

Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire Exclusive locks fortransactional tables. This ensures that inserts (w/o overwrite) running concurrently are not hidden by the INSERT OVERWRITE.

防止报异常：Hive throws: WstxParsingException: Illegal character entity: expansion character (code 0x8)

b. 将配置文件中所有的${system:java.io.tmpdir}/${system:user.name}修改为绝对路径如/tmp/scratchdir，同理将${system:java.io.tmpdir}/${hive.session.id}_resources中的${system:java.io.tmpdir}也替换为一个绝对路径，否则会报异常：Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D**

c. 在javax.jdo.option.ConnectionURL属性中增加derby的路径，如下所示：

jdbc:derby:/home/ws/module/hive/metastore_db;databaseName=metastore_db;create=true

d. 配置好之后需要创建metada schema: schematool -initSchema -dbType derby

e. 然后执行hive即可对hive命令行进行测试

2) 因为derby数据库默认同时只允许一个用户连接，因此通常hive会使用mysql数据库，具体配置如下：

a. 由于需要链接mysql，因此需要将mysql的jar包(博主采用mysql-connector-java-5.1.47.jar)放置在hive安装目录的lib文件夹下

b. 可以不使用默认的hive-site.xml，而是直接在hive-site.xml只需要录入需要的属性配置信息：

<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=UTF-8&amp;useSSL=false</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    <description>Username to use against metastore database</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>111</value>
    <description>password to use against metastore database</description>
  </property>

  <property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
  </property>

        <property>
                <name>datanucleus.readOnlyDatastore</name>
                <value>false</value>
        </property>
        <property>
                <name>datanucleus.fixedDatastore</name>
                <value>false</value>
        </property>
        <property>
                <name>datanucleus.autoCreateSchema</name>
                <value>true</value>
        </property>
        <property>
                <name>datanucleus.schema.autoCreateAll</name>
                <value>true</value>
        </property>
        <property>
                <name>datanucleus.autoCreateTables</name>
                <value>true</value>
        </property>
<property>
                <name>datanucleus.autoCreateColumns</name>
                <value>true</value>
        </property>
        <property>
                <name>hive.metastore.local</name>
                <value>true</value>
        </property>
        <!-- 显示表的列名 -->
        <property>
                <name>hive.cli.print.header</name>
                <value>true</value>
        </property>
        <!-- 显示数据库名称 -->
        <property>
                <name>hive.cli.print.current.db</name>
                <value>true</value>
        </property>
</configuration>

View Code

c. 创建metada schema: schematool -initSchema -dbType mysql

d. 然后执行hive即可对hive命令行进行测试

2. 异常汇总

(1) 部分遇到的异常，已在如上安装步骤中列出

(2) 异常FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

解决方案：需要创建metada schema：schematool -initSchema -dbType derby/mysql

(3) 关于guava的异常：java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

解决方案：hive 3.1.2中默认的guava包为guava-19.0.jar，在https://mvnrepository.com/artifact/com.google.guava/guava下载比较新的guava包替换即可，博主使用的是guava-27.0-jre.jar

3. 参考

(1) https://askubuntu.com/questions/1073035/failed-hiveexception-java-lang-runtimeexception-unable-to-instantiate-org-apac

(2) https://stackoverflow.com/questions/52783323/hive-throws-wstxparsingexception-illegal-character-entity-expansion-character

(3) hive3.1.2安装 

如有问题，留言联系！