Hadoop中Hive组件如何进行高效部署？

本文共计3285个文字，预计阅读时间需要14分钟。

目录

一、基础环境

二、安装MySQL

1.将所需包复制到 /opt 目录下

2.解压 mysql-5.7.18.zip

3.安装MySQL数据库

4.修改配置

5.启动MySQL

6.查询默认密码

7.初始化数据库

目录

• 一、基础环境
• 二、安装MySQL
  • 1、传输需要的包到/opt目录下
  • 2、卸载自带的MariaDB
  • 3、解压mysql-5.7.18.zip
  • 4、依次安装MySQL数据库
  • 5、修改配置
  • 6、启动MySQL
  • 7、查询MySQL默认密码
  • 8、初始化MySQL
  • 9、进入数据库，添加权限
• 二、配置Hive组件
  • 1、设置Hive环境变量
  • 2、修改Hive组件配置文件
  • 3、重启all.sh
  • 4、初始化数据库
  • 5、启动hive
  • 6、验证

一、基础环境

Hive 组件需要基于之前已部署完毕的 Hadoop 全分布系统，在 master 节点上实现 Hive 组件安装。

1、Hadoop-全分布式配置（全部配置）
2、Hadoop-集群运行（部分配置）

# 查看之前部署的环境 [root@master ~]# su - hadoop [hadoop@master ~]$ jps 47967 Jps [hadoop@master hadoop]$ start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [master] master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out 192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out 192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out starting yarn daemons starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out 192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out 192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out [hadoop@master hadoop]$ jps 44678 NameNode 45030 ResourceManager 45289 Jps 44874 SecondaryNameNode [hadoop@master hadoop]$ exit logout

要确保 Hadoop 系统能够正常运行

二、安装MySQL

注意：使用 root 用户登录

1、传输需要的包到/opt目录下

[root@master ~]# cd /opt/software/ [root@master software]# ls apache-hive-2.0.0-bin.tar.gz jdk-8u152-linux-x64.tar.gz mysql-connector-java-5.1.46.jar hadoop-2.7.1.tar.gz mysql-5.7.18.zip 2、卸载自带的MariaDB

[root@master software]# rpm -qa | grep mariadb mariadb-libs-5.5.52-1.el7.x86_64 [root@master software]# rpm -e --nodeps mariadb-libs-5.5.52-1.el7.x86_64 3、解压mysql-5.7.18.zip

[root@master software]# ls apache-hive-2.0.0-bin.tar.gz jdk-8u152-linux-x64.tar.gz mysql-connector-java-5.1.46.jar hadoop-2.7.1.tar.gz mysql-5.7.18.zip [root@master software]# unzip mysql-5.7.18.zip Archive: mysql-5.7.18.zip creating: mysql-5.7.18/ inflating: mysql-5.7.18/mysql-community-client-5.7.18-1.el7.x86_64.rpm inflating: mysql-5.7.18/mysql-community-common-5.7.18-1.el7.x86_64.rpm inflating: mysql-5.7.18/mysql-community-devel-5.7.18-1.el7.x86_64.rpm inflating: mysql-5.7.18/mysql-community-libs-5.7.18-1.el7.x86_64.rpm inflating: mysql-5.7.18/mysql-community-server-5.7.18-1.el7.x86_64.rpm 4、依次安装MySQL数据库

[root@master software]# cd mysql-5.7.18/ [root@master mysql-5.7.18]# ls mysql-community-client-5.7.18-1.el7.x86_64.rpm mysql-community-libs-5.7.18-1.el7.x86_64.rpm mysql-community-common-5.7.18-1.el7.x86_64.rpm mysql-community-server-5.7.18-1.el7.x86_64.rpm mysql-community-devel-5.7.18-1.el7.x86_64.rpm [root@master mysql-5.7.18]# rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm warning: mysql-community-common-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY Preparing... ################################# [100%] Updating / installing... 1:mysql-community-common-5.7.18-1.e################################# [100%] [root@master mysql-5.7.18]# rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm warning: mysql-community-libs-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY Preparing... ################################# [100%] Updating / installing... 1:mysql-community-libs-5.7.18-1.el7################################# [100%] [root@master mysql-5.7.18]# rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm warning: mysql-community-client-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY Preparing... ################################# [100%] Updating / installing... 1:mysql-community-client-5.7.18-1.e################################# [100%] [root@master mysql-5.7.18]# rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm warning: mysql-community-server-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY Preparing... ################################# [100%] Updating / installing... 1:mysql-community-server-5.7.18-1.e################################# [100%] 5、修改配置

添加配置信息到/etc/my.cnf 文件 symbolic-links=0 的下方

[root@master mysql-5.7.18]# vi /etc/my.cnf default-storage-engine=innodb innodb_file_per_table collation-server=utf8_general_ci init-connect='SET NAMES utf8' character-set-server=utf8 6、启动MySQL

[root@master mysql-5.7.18]# systemctl start mysqld [root@master mysql-5.7.18]# systemctl status mysqld ● mysqld.service - MySQL Server Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2022-04-09 10:12:27 CST; 44s ago Docs: man:mysqld(8) dev.mysql.com/doc/refman/en/using-systemd.html Process: 45577 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS) Process: 45499 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS) Main PID: 45580 (mysqld) CGroup: /system.slice/mysqld.service └─45580 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid Apr 09 10:12:17 master.example.com systemd[1]: Starting MySQL Server... Apr 09 10:12:27 master.example.com systemd[1]: Started MySQL Server. 7、查询MySQL默认密码

[root@master ~]# cat /var/log/mysqld.log | grep password 2022-04-09T02:12:24.671882Z 1 [Note] A temporary password is generated for root@localhost: /RrHz+J-t1fj

默认密码为/RrHz+J-t1fj

8、初始化MySQL

初始化mysql_secure_installation
先输入旧密码/RrHz+J-t1fj
设置新密码
是否更改 root 用户密码n
是否使用设定的密码继续y
是否删除匿名用户y
是否拒绝 root 用户远程登录n
是否删除测试数据库y
是否重新加载授权表y

[root@master ~]# mysql_secure_installation Securing the MySQL server deployment. Enter password for user root: The existing password for the user account root has expired. Please set a new password. New password: Re-enter new password: The 'validate_password' plugin is installed on the server. The subsequent steps will run with the existing configuration of the plugin. Using existing password for root. Estimated strength of the password: 100 Change the password for root ? ((Press y|Y for Yes, any other key for No) : n Estimated strength of the password: 100 Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y By default, a MySQL installation has an anonymous user, allowing anyone to log into MySQL without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? (Press y|Y for Yes, any other key for No) : n ... skipping. Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n ... skipping. By default, MySQL comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? (Press y|Y for Yes, any other key for No) : n ... skipping. Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? (Press y|Y for Yes, any other key for No) : n ... skipping. All done! 9、进入数据库，添加权限

添加 root 用户从本地和远程访问 MySQL 数据库表单的授权

[root@master ~]# mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 12 Server version: 5.7.18 MySQL Community Server (GPL) Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> grant all privileges on *.* to root@'localhost' identified by 'Wangzhigang123!'; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> grant all privileges on *.* to root@'%' identified by 'Wangzhigang123!'; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql> select user,host from mysql.user where user='root'; +------+-----------+ | user | host | +------+-----------+ | root | % | | root | localhost | +------+-----------+ 2 rows in set (0.00 sec) mysql> exit; Bye [root@master ~]# 二、配置Hive组件 1、设置Hive环境变量

# 解压hive文件 [root@master ~]# tar -zxf /opt/software/apache-hive-2.0.0-bin.tar.gz [root@master ~]# mv apache-hive-2.0.0-bin/ /usr/local/src/hive # 修改用户和用户组为 hadoop [root@master ~]# chown -R hadoop:hadoop /usr/local/src/hive # 添加配置信息 [root@master ~]# vi /etc/profile # set hive environment export HIVE_HOME=/usr/local/src/hive export PATH=$PATH:$HIVE_HOME/bin [root@master ~]# source /etc/profile 2、修改Hive组件配置文件

[root@master ~]# su - hadoop [hadoop@master ~]$ cp /usr/local/src/hive/conf/hive-default.xml.template /usr/local/src/hive/conf/hive-site.xml [hadoop@master ~]$ vi /usr/local/src/hive/conf/hive-site.xml # 设置 MySQL 数据库连接 <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value> <description>JDBC connect string for a JDBC metastore</description> # 配置 MySQL 数据库 root 的密码 <name>javax.jdo.option.ConnectionPassword</name> <value>Wangzhigang123!</value> <description>password to use against metastore database</description> # 验证元数据存储版本一致性,默认 false <name>hive.metastore.schema.verification</name> <value>false</value> # 配置数据库驱动 <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> # 配置数据库用户名 <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>Username to use against metastore database</description> # 配置目录信息（共4处） <name>hive.querylog.location</name> <value>/usr/local/src/hive/tmp</value> <description>Location of Hive run time structured log file</description> <name>hive.exec.local.scratchdir</name> <value>/usr/local/src/hive/tmp</value> <name>hive.downloaded.resources.dir</name> <value>/usr/local/src/hive/tmp/resources</value> <name>hive.server2.logging.operation.log.location</name> <value>/usr/local/src/hive/tmp/operation_logs</value> [hadoop@master ~]$ mkdir -p /usr/local/src/hive/tmp{resources,operation_logs} 3、重启all.sh

[hadoop@master ~]$ cp /opt/software/mysql-connector-java-5.1.46.jar /usr/local/src/hive/lib/ [hadoop@master ~]$ stop-all.sh This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh Stopping namenodes on [master] master: stopping namenode 192.168.100.20: stopping datanode 192.168.100.30: stopping datanode Stopping secondary namenodes [0.0.0.0] 0.0.0.0: stopping secondarynamenode stopping yarn daemons stopping resourcemanager 192.168.100.20: stopping nodemanager 192.168.100.30: stopping nodemanager no proxyserver to stop [hadoop@master ~]$ start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [master] master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out 192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out 192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out starting yarn daemons starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out 192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out 192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out 4、初始化数据库

[hadoop@master ~]$ schematool -initSchema -dbType mysql which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Metastore connection URL: jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: root Starting metastore schema initialization to 2.0.0 Initialization script hive-schema-2.0.0.mysql.sql Initialization script completed schemaTool completed 5、启动hive

[hadoop@master ~]$ hive which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. hive> show databases; OK default Time taken: 0.241 seconds, Fetched: 1 row(s) hive> exit; [hadoop@master ~]$ 6、验证

[hadoop@master ~]$ mysql -uroot -pWangzhigang123! mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 24 Server version: 5.7.18 MySQL Community Server (GPL) Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | hive | | mysql | | performance_schema | | sys | +--------------------+ 5 rows in set (0.00 sec) mysql> exit; Bye [hadoop@master ~]$ jps 47057 NameNode 47411 ResourceManager 47254 SecondaryNameNode 47967 Jps [hadoop@master ~]$

slave1节点

[root@slave1 ~]# su - hadoop [hadoop@slave1 ~]$ jps 43621 Jps 43351 DataNode 43464 NodeManager

slave2节点

[root@slave2 ~]# su - hadoop [hadoop@slave2 ~]$ jps 41607 Jps 41337 DataNode 41450 NodeManager

声明：未经许可，不得转载
原文地址：www.cnblogs.com/wzgwzg/p/16121326.html