centos6.8下CDH5.13.3安装与配置(详细部署)

centos 6.8下 CDH5.13.3安装与配置(详细部署)

创新互联是一家专业提供石龙企业网站建设,专注与成都做网站、成都网站制作、成都外贸网站建设html5、小程序制作等业务。10年已为石龙众多企业、政府机构等服务。创新互联专业的建站公司优惠进行中。

一、安装前准备

1.1、下载地址

wgethttp://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.3_x86_64.tar.gz

http://archive.cloudera.com/cdh6/parcels/5.13.3/

wgethttp://archive.cloudera.com/cdh6/parcels/5.13.3/manifest.json

wgethttp://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel

wgethttp://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

1.2、防火墙关闭

/etc/init.d/iptables stop

chkconfig iptables off

1.3、设置selinux

vi /etc/selinux/config

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

1.4、设置时间同步ntp(每台机器)

yum -y install ntpdate

crontab -e

*/5 * * * * ntpdate 0.rhel.pool.ntp.org

保存退出

vi /etc/hosts(hadoop01增加)

注意:Cloudera-Manager台需要增加127.0.0.1   hadoop01

127.0.0.1   hadoop01

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

vi /etc/hosts(其他两台hadoo02、hadoop03)

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

二、依赖包安装(每台机器)

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi  openssh-clients

三、jdk安装(每台机器)

tar zxvf jdk-8u111-linux-x64.gz

mv jdk1.8.0_111 /usr/local/jdk

配置环境变量

vi /etc/profile

##jdk

export JAVA_HOME=/usr/local/jdk

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$PATH

让环境变量生产

source /etc/profile

四、ssh免密码登录配置

ssh-keygen  -t rsa

ssh-copy-id 192.168.1.201

ssh-copy-id 192.168.1.202

ssh-copy-id 192.168.1.202

五、安装MySQL(在haoop01上安装)

5.1、安装依赖包

yum -y install libaio*

yum -y install libnuma*

5.2、创建mysql用户

useradd mysql

5.3、创建目录

mkdir -p /data/app

mkdir /data/app/mysql3306

mkdir /data/app/undolog

5.4、解压文件

tar zxvf mysql-5.6.41-linux-glibc2.12-x86_64.tar.gz

mv mysql-5.6.41-linux-glibc2.12-x86_64 /data/app/mysql5.6

mkdir /data/app/mysql5.6/logs

5.5、配置my.cnf

vi /data/app/mysql5.6/my.cnf

[client]

socket=/data/app/mysql5.6/mysqld.sock

port=3306

[mysql]

prompt=\\u@\\d \\r:\\m:\\s>

no-auto-rehash

default-character-set=utf8

[mysqld_safe]

log-error=/data/app/mysql3306/mysqld.error

[mysqld]

socket=/data/app/mysql5.6/mysqld.sock

pid-file=/data/app/mysql5.6/mysqld.pid

basedir=/data/app/mysql5.6

datadir=/data/app/mysql3306

port=3306

server-id=1201

character-set-server=utf8

skip-external-locking

#skip-name-resolve

max_connections=1000

max_connect_errors=1000

max_allowed_packet=128M

table_definition_cache=500

table_open_cache=500

sort_buffer_size = 1M

tmp_table_size = 64M

read_buffer_size = 1M

read_rnd_buffer_size = 1M

myisam_sort_buffer_size = 64M

thread_cache_size = 8

query_cache_type=0

query_cache_size=0

lower_case_table_names = 1

log_bin_trust_function_creators = 1

#################slow log####################

slow-query_log=1

slow-query_log_file=/data/app/mysql5.6/logs/mysql.slow

long_query_time=2

####################binlog######################

log-bin=mysql-bin

binlog-format=ROW

expire_logs_days=10

log-slave-updates=1

sync_binlog=0

auto_increment_increment=2

auto_increment_offset=1

###

###

###

innodb_undo_directory=/data/app/undolog

innodb_undo_logs = 128

innodb_undo_tablespaces = 3

################INNODB################

transaction-isolation=READ-COMMITTED

innodb_buffer_pool_size=2G

innodb_flush_log_at_trx_commit=2

innodb_strict_mode=1

innodb_flush_method=O_DIRECT

innodb_file_format=Barracuda

innodb_log_files_in_group=3

innodb_file_per_table=1

innodb_io_capacity=500

innodb_support_xa=1

[mysqldump]

quick

max_allowed_packet=128M

[mysqlhotcopy]

interactive_timeout     = 28400

5.6、增加权限

chown -R mysql:mysql /data/app

5.7、配置环境变量

vi /etc/profile

##mysql

export MYSQL_HOME=/data/app/mysql5.6

export MY_BASEDIR_VERSION=/data/app/mysql5.6

export PATH=/data/app/mysql5.6/bin:$PATH

export LD_LIBRARY_PATH=/data/app/mysql5.6/lib

环境生效

source /etc/profile

5.8、初始化数据库

/data/app/mysql5.6/scripts/mysql_install_db --user=mysql --basedir=/data/app/mysql5.6   --datadir=/data/app/mysql3306

5.9、配置启动文件

cp /data/app/mysql5.6/support-files/mysql.server /etc/init.d/mysqld

vi /etc/init.d/mysqld

basedir=/data/app/mysql5.6

datadir=/data/app/mysql3306

保存退出

chmod +x /etc/init.d/mysqld

chkconfig mysqld on

启动

/etc/init.d/mysqld start

5.10、初始化密码

/data/app/mysql5.6/bin/mysqladmin -u root password 'yb2018#08'

5.11、设置远程权限

# mysql -uroot -pyb2018#08

Warning: Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 2

Server version: 5.6.41-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

root@(none) 10:28:44>grant all on *.* to 'scm'@'localhost' identified by 'temp' with grantoption;

Query OK, 0 rows affected (0.00 sec)

root@(none) 10:28:58>grant all on *.* to 'cdh'@'%' identified by 'cdh#2018' with grant option;

Query OK, 0 rows affected (0.00 sec)

root@(none) 10:29:08>create database hive;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:16>create database ooz;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:27>create database hue;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:36>flush privileges;

Query OK, 0 rows affected (0.00 sec)

六、安装Cloudera-Manager

6.1、解压包到指定目录,所有服务器都要

mkdir -p /data/cloudera-manager

tar zxvf cloudera-manager-el6-cm5.13.3_x86_64.tar.gz -C /data/cloudera-manager

6.2创建cloudera-scm用户(所有节点)

useradd -r -d /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-server -M -c"Cloudera SCM User" cloudera-scm

id cloudera-scm

6.3、配置从节点cloudera-manger-agent指向主节点服务器(hadoop02,hadoop03配置)

vi /data/cloudera-manager/cm-5.13.3/etc/cloudera-scm-agent/config.ini

将server_host改为CMS所在的主机名即hadoop01

6.4、主节点中创建parcel-repo仓库目录(这个只在主节点hadoop01上需要)

mkdir -p /data/cloudera/parcel-repo

chown cloudera-scm:cloudera-scm /data/cloudera/parcel-repo

cp CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 manifest.json /data/cloudera/parcel-repo/

cd /data/cloudera/parcel-repo/

mv CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha

注意:其中CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 后缀要把1去掉

6.5、所有节点创建parcels目录(所有节点)

mkdir -p /data/cloudera/parcels

chown cloudera-scm:cloudera-scm /data/cloudera/parcels

解释:Clouder-Manager将CDHs从主节点的/data/cloudera/parcel-repo目录中抽取出来,分发解压激活到各个节点的/data/cloudera/parcels目录中

6.6、初始脚本配置数据库scm_prepare_database.sh(在主节点上)

/data/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh  mysql -uscm -ptemp scm scm temp

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

如果报错,操作

mkdir /usr/share/java

cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

上图说明成功

成功把记住把#127.0.0.1   localhost hadoop01 注释掉,不然后面就会有问题

vi /etc/hosts

#127.0.0.1  hadoop01

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

6.7、启动主节点cloudera-scm-server(hadoop01操作)

cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server

chmod +x /etc/init.d/cloudera-scm-server

启动

/etc/init.d/cloudera-scm-server start

如果报File not found: /usr/sbin/cmf-server错误

vi /etc/init.d/cloudera-scm-server

找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}

修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

启动

/etc/init.d/cloudera-scm-server start

6.8、启动cloudera-scm-agent所有节点(hadoop02,hadoop03操作)

mkdir /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-agent

cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent

chmod +x /etc/init.d/cloudera-scm-agent

/etc/init.d/cloudera-scm-agent start

vi /etc/init.d/cloudera-scm-agent

找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}

修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

启动

/etc/init.d/cloudera-scm-agent start

6.9、登录用户admin 密码admin

http://192.168.1.201:7180

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

选择免费,继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

选择当前管理主机选择hadoop02,hadoop03,继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

点击更多选项配置

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

修改好后,保存修改

重启后才后出下如下图

/etc/init.d/cloudera-scm-server restart

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

重启好后才能如上图继续,因为默认/opt下安装的

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

每台执行以下

echo 10 >/proc/sys/vm/swappiness

echo never >/sys/kernel/mm/transparent_hugepage/defrag

echo never >/sys/kernel/mm/transparent_hugepage/enabled

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

以图根据自己需求选择不一样点继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

以上Datanode修改两台

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

把连接主机名,数据库名称,用户名,密码输入进去,点击测试连接都Succesful成功后,点击继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

根据自己修改点击继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

继续

centos 6.8下 CDH5.13.3安装与配置(详细部署)centos 6.8下 CDH5.13.3安装与配置(详细部署)

安装完成

centos 6.8下 CDH5.13.3安装与配置(详细部署)

centos 6.8下 CDH5.13.3安装与配置(详细部署)


网站标题:centos6.8下CDH5.13.3安装与配置(详细部署)
网址分享:http://myzitong.com/article/pjopeh.html