MENU

系统环境

  • 操作系统:Centos 7 X64-
  • CDH版本:5.14.0

相关包下载

CDH 下载地址

  1. CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel
  2. CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel.sha
  3. manifest.json

Cloudera Manager

  • cloudera-manager-centos7-cm5.14.0_x86_64.tar.gz

JDK

  • jdk-8u162-linux-x64.tar.gz

MySQL

  • mysql-connector-java-5.1.22-bin.jar
  • mysql-server-5.1.71-1.el6.x86_64

准备工作:系统环境搭建

在root用户下操作

网络配置(所有节点)
vi /etc/hosts

10.116.160.54 hadoop00

10.116.160.55 hadoop01

10.116.160.56 hadoop02

10.116.160.57 hadoop03

10.116.160.59 hadoop04
打通SSH,设置无密码登录(所有节点)

一路回车 ssh-keygen -t rsa

以hadoop00为例

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop01
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop02
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop03
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop04
关闭IPV6(所有节点)

vi /etc/sysctl.conf

禁用整个系统所有接口的IPV6

net.ipv6.conf.all.disable_ipv6 = 1

生效 sysctl -p /etc/sysctl.conf

关闭SELINUX(所有节点)

vi /etc/sysconfig/selinux

修改 SELINUX = disabled (注意别修改错)

关闭防火墙(所有节点)

直接关闭防火墙 systemctl stop firewalld.service

禁止firewall开机启动 systemctl disable firewalld.service

检查状态

service firewalld status
/bin/systemctl status firewalld.service

关闭或禁用防火墙

service firewalld stop
/bin/systemctl stop firewalld.service 
打开句柄限制(所有节点)

vi /etc/security/limits.conf

根据当前用户名,添加

*    soft    nofile    327680
*    hard    nofile      327680
hdfs    soft    nproc    131072
hdfs    hard    nproc     131072
mapred    soft     nproc     131072
mapred    hard    nproc     131072
hbase    soft     nproc     131072
hbase    hard    nproc     131072
zookeeper    soft     nproc     131072
zookeeper    hard    nproc    131072
hive    soft    nproc     131072
hive    hard    nproc     131072
root    soft     nproc     131072
root    hard    nproc     131072
cloudera-scm    soft     nproc    131072
cloudera-scm    hard    nproc    131072

执行命令 vi /etc/pam.d/login

添加 session required pam_limits.so

NTP时间同步(待完善)

在其中一台机器上面搭建ntp服务器

为服务器和客户机安装ntp ntpdate yum install ntp ntpdate -y

安装JDK(所有节点)

查看版本

java -version
rpm -qa | grep java

卸载 rpm -e --nodeps [包名]

解压

mkdir /usr/java
tar -zxvf jdk-8u162-linux-x64.tar.gz -C /usr/java

配置环境变量

vi ~/.bash_profile
export JAVA_HOME=/usr/java/jdk1.8.0_162
export PATH=$JAVA_HOME/bin:$PATH

生效 source ~/.bash_profile

验证 echo $JAVA_HOME # /usr/java/jdk1.8.x_yyy

分发

scp -r /usr/java root@hadoop01:/usr/
scp ~/.bash_profile root@hadoop01:~/
安装配置MySQL(主节点)

检查系统是否有安装过的mysql,需要卸载干净 rpm -qa|grep -i mysql

安装

rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum -y remove mariadb-libs
yum install mysql-community-server
yum install mysql-server

启动服务

service mysqld start
/bin/systemctl start mysqld.service

设置开机启动

chkconfig mysqld on
systemctl enable mysqld.service

设置root初始密码 mysqladmin -u root password '123456';

进入mysql命令行 mysql -uroot -p

创建数据库hive

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

授权root用户在主节点拥有所有数据库的访问权限

grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;

grant all privileges on 库名.表名 to '用户名'@'IP地址' identified by '密码' with grant option;

刷新 flush privileges;

正式开工

  1. 安装Cloudera Manager Server & Agent
主节点解压

默认安装在/opt下

tar -zxvf cloudera-manager-centos7-cm5.14.0_x86_64.tar.gz -C /opt

为 Cloudera Manager 5 建立数据库

将mysql-connector-java-5.1.22-bin-jar放到/opt/cm-5.14.0/share/cmf/lib中

mv mysql-connector-java-5.1.22-bin-jar /opt/cm-5.14.0/share/cmf/lib

初始化CM5的数据库,登录主节点

/opt/cm-5.14.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123456 --scm-host localhost scm scm scm

Agent配置
vi /opt/cm-5.14.0/etc/cloudera-scm-agent/config.ini
server_host=hadoop00

同步

scp -r /opt/cm-5.14.0 root@hadoop01:/opt/
scp -r /opt/cm-5.14.0 root@hadoop02:/opt/
在所有节点创建 cloudera-scm 用户

useradd --system --home=/opt/cm-5.14.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

准备 Parcels,用以安装 CDH5

将CDH5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中

相关的文件如下:

  • CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel
  • CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel.sha1 # .sha1 文件重命名为 .sha
  • manifest.json
相关启动脚本

主节点 启动服务端

/opt/cm-5.14.0/etc/init.d/cloudera-scm-server start

全部机器 启动Agent服务

/opt/cm-5.14.0/etc/init.d/cloudera-scm-agent start

启动的是service脚本,停止stop,重启restart

CDH5的安装配置

在浏览器输入:http://10.116.160.86:7180(hadoop00的IP地址)

用户名密码均为admin