大数据技术
2024年2月1日

CDH安装(一)

作者 高劲松

一、服务器规划

宿主机情况

虚拟机准备

二、安装前准备

网络配置

配置固定IP，并且保证能够上外网

vi /etc/sysconfig/network-scripts/ifcfg-ens33

IPADDR=10.168.1.21

NETMASK=255.255.255.0

GATEWAY=10.168.1.1

BOOTPROTO=static

[root@localhost ~]# vi /etc/resolv.conf

# Generated by NetworkManager

nameserver 8.8.8.8

service network start

设置主机名

hostnamectl set-hostname 命令用于设置主机名。

使用方法为： hostnamectl set-hostname [new_hostname] 。

例如，要将主机名更改为 “mycomputer”，可以使用命令 hostnamectl set-hostname mycomputer 。

设置etc/hosts（所有节点）

10.168.1.31 node1

10.168.1.32 node2

10.168.1.33 node3

关闭防火墙与selinux文件

systemctl stop firewalld

systemctl disable firewalld.service

关闭selinux

输入vim /etc/selinux/config,将SELINUX=disabled

临时关闭 selinux

setenforce 0

永久关闭 selinux

sed -i “s/^SELINUX=enforcing/SELINUX=disabled/g” /etc/selinux/config

设置 SSH 免密码登录（主节点）

1.ssh-keygen -t rsa

2.cd ~/.ssh

3.# 将公钥导入到认证文件

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

4.# 认证文件合并，输入 yes 回车再输入服务器密码

ssh-copy-id node1

安装 ntp

yum install -y ntp

启动ntp

systemctl start ntpd

开机自动启动

systemctl enable ntpd

查看ntp状态

systemctl status ntpd

安装JDK

上传 JDK8 压缩文件

解压

tar -zxvf jdk-8u212-linux-x64.tar.gz

配置当前用户环境变量

vim ~/.bash_profile

在文件中添加如下命令，记得切换自己文件路径

export JAVA_HOME=/usr/java/jdk1.8.0_212

export PATH=$JAVA_HOME/bin:$PATH

立即生效

source ~/.bash_profile

设置vm.swappiness(每台主机都要运行)

sysctl vm.swappiness=10

echo ‘vm.swappiness=10’>> /etc/sysctl.conf

设置透明大页面压缩(每台主机都要运行)

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

安装一些依赖库(每台主机都要运行)

yum install -y vim wget ntp net-tools httpd createrepo

启动httpd(安装主机运行)

systemctl start httpd

[root@localhost jdk1.8.0_131]# systemctl start httpd

[root@localhost jdk1.8.0_131]# cd /var/www/html

[root@localhost html]# ls

 [root@localhost html]#

[root@localhost html]# mkdir -p cloudera-repos/

[root@localhost html]# cd cloudera-repos/

[root@localhost cloudera-repos]# ls

 [root@localhost cloudera-repos]# pwd

/var/www/html/cloudera-repos

[root@localhost cloudera-repos]# mkdir cm6

[root@localhost cloudera-repos]# tree

-bash: tree: command not found

[root@localhost cloudera-repos]# yum install -y tree

配置yum源(每台主机都运行)

(

cat <<EOF

[cloudera-manager]

name=Cloudera Manager 6.3.1

baseurl=http://node1/cloudera-repos/cm6/cm6.3.1/

gpgcheck=0

enabled=1

EOF

) >> /etc/yum.repos.d/cloudera-manager.repo

然后scp过去即可。

上传安装包到相关目录

[root@node1 html]# tree
.
└── cloudera-repos
├── cdh6
│   └── 6.3.1
│   └── parcels
│   ├── CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
│   ├── CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
│   ├── CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
│   └── manifest.json
└── cm6
└── cm6.3.1
├── allkeys.asc
├── repodata
│   ├── 3662f97de72fd44c017bb0e25cee3bc9398108c8efb745def12130a69df2ecb2-filelists.sqlite.bz2
│   ├── 43f3725f730ee7522712039982aa4befadae4db968c8d780c8eb15ae9872cd4d-primary.xml.gz
│   ├── 49e4d60647407a36819f1d8ed901258a13361749b742e3be9065025ad31feb8e-filelists.xml.gz
│   ├── 8afda99b921fd1538dd06355952719652654fc06b6cd14515437bda28376c03d-other.sqlite.bz2
│   ├── b9300879675bdbc300436c1131a910a535b8b5a5dc6f38e956d51769b6771a96-primary.sqlite.bz2
│   ├── e28836e19e07f71480c4dad0f7a87a804dc93970ec5277ad95614e8ffcff0d58-other.xml.gz
│   ├── repomd.xml
│   ├── repomd.xml.asc
│   └── repomd.xml.key
├── RPM-GPG-KEY-cloudera
├── RPMS
│   ├── noarch
│   └── x86_64
│   ├── cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
│   ├── cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
│   ├── cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
│   ├── cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
│   ├── enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
│   └── oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
└── SRPMS

安装Cloudera Manager(安装主机运行)

yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

安装MySQL

卸载自带的mariadb(安装主机运行)

yum -y remove mariadb-libs

安装(安装主机运行)

yum install -y libaio \

&& rpm -ivh mysql-community-common-5.7.16-1.el7.x86_64.rpm \

&& rpm -ivh mysql-community-libs-5.7.16-1.el7.x86_64.rpm \

&& rpm -ivh mysql-community-client-5.7.16-1.el7.x86_64.rpm \

&& rpm -ivh mysql-community-server-5.7.16-1.el7.x86_64.rpm \

&& rpm -ivh mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm

配置my.cnf(安装主机运行)

echo character-set-server=utf8 >> /etc/my.cnf

systemctl enable mysqld && systemctl start mysqld && grep password /var/log/mysqld.log

[root@localhost ~]# systemctl enable mysqld && systemctl start mysqld && grep password /var/log/mysqld.log

2024-01-31T08:04:53.335184Z 1 [Note] A temporary password is generated for root@localhost: U%3epepqzVrk

[root@localhost ~]#

创建数据库并授权(安装主机运行)

(

cat <<EOF

set password for root@localhost = password(‘123456Aa.’);

grant all privileges on *.* to ‘root’@’%’ identified by ‘123456Aa.’;

flush privileges;

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE monitor DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON scm.* TO ‘scm’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON monitor.* TO ‘monitor’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON amon.* TO ‘amon’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON rman.* TO ‘rman’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON hue.* TO ‘hue’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON metastore.* TO ‘hive’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON sentry.* TO ‘sentry’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON nav.* TO ‘nav’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON navms.* TO ‘navms’@’%’ IDENTIFIED BY ‘123456Aa.’;

GRANT ALL ON oozie.* TO ‘oozie’@’%’ IDENTIFIED BY ‘123456Aa.’;

SHOW DATABASES;

EOF

) >> /root/cdhinit.sql

CREATE DATABASE cmf DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

运行脚本

source /root/cdhinit.sql

配置JDBC驱动包(安装主机运行)

mkdir -p /usr/share/java/

cp mysql-connector-java.jar /usr/share/java/

配置CM的数据库实例

1、CM数据库可选：mysql、postgresql和oracle，我们选择使用mysql

[root@node1]# cd /etc/cloudera-scm-server

2、修改：数据库类型，账号，密码，主机名称

[root@node1 cloudera-scm-server]# vi db.properties

3、拷贝mysql驱动jar包到CM server中

[root@node1 cloudera-scm-server]# cp mysql-connector-java-5.1.47.jar  /opt/cloudera/cm/lib/

4、启动CM服务

启动CM(安装主机运行)

systemctl start cloudera-scm-server

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注