(实际应用)ELK环境搭建-创新互联

今天给大家带来的是开源实时日志分析 ELK , ELK 由 ElasticSearch 、 Logstash 和 Kiabana 三个开源工具组成。官方网站:https://www.elastic.co

成都创新互联-专业网站定制、快速模板网站建设、高性价比温泉网站开发、企业建站全套包干低至880元,成熟完善的模板库,直接使用。一站式温泉网站制作公司更省心,省钱,快速模板网站建设找我们,业务覆盖温泉地区。费用合理售后完善,十余年实体公司更值得信赖。

其中的3个软件是:

Elasticsearch 是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制, restful 风格接口,多数据源,自动搜索负载等。

Logstash 是一个完全开源的工具,他可以对你的日志进行收集、分析,并将其存储供以后使用(如,搜索)。

kibana 也是一个开源和免费的工具,他 Kibana 可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以帮助您汇总、分析和搜索重要数据日志

系统系统需要安装的软件ip描述
centos6.5Elasticsearch/test5192.168.253.210搜索存储日志
centos6.5Elasticsearch/test4192.168.253.200搜索存储日志
centos6.5Logstash/nginx/test1192.168.253.150用来收集日志给上面
centos6.5Kibana/nginx/test2192.168.253.100用来后端的展示

架构原理图:

(实际应用)ELK环境搭建

一、先安装elasticsearch集群,并测试通过再进行其他软件安装。

在test5,test4上安装分别安装elasticsearch-2.3.3.rpm 前提要安装java1.8 步骤如下:

yum remove java-1.7.0-openjdk

rpm -ivh jdk-8u51-linux-x64.rpm

java -version

yum localinstall elasticsearch-2.3.3.rpm -y

service elasticsearch start

cd /etc/elasticsearch/

vim elasticsearch.yml

(实际应用)ELK环境搭建

(实际应用)ELK环境搭建

修改如下配置

cluster.name: myelk #设置集群的名称,在一个集群里面都是这个名称,必须相同

node.name: test5  #设置每一个节点的名,每个节点的名称必须不一样。

path.data: /path/to/data  #指定数据的存放位置,线上的机器这个要放到单一的大分区里面。

path.logs: /path/to/logs  #日志的目录

bootstrap.mlockall: true #启动最优内存配置,启动就分配了足够的内存,性能会好很多,测试我就不启动了。

network.host: 0.0.0.0 #监听的ip地址,这个表示所有的地址。

http.port: 9200   #监听的端口号

discovery.zen.ping.unicast.hosts: ["hostip", "hostip"] #知道集群的ip有那些,没有集群就会出现就一台工作

mkdir -pv /path/to/{data,logs}

chown elasticsearch.elasticsearch /path -R

启动服务器 service elasticsearch start 并查看监控端口启动

[root@test4 ~]# ss -tln

State    Recv-Q Send-Q             Local Address:Port              Peer Address:Port

LISTEN    0    128                    :::54411                                   :::*

LISTEN    0    128                    :::111                                      :::*

LISTEN    0    128                     *:111                                   *:*

LISTEN    0    50                         :::9200                                   :::*

LISTEN    0    50                     :::9300                                   :::*

LISTEN    0    128                    :::22                                        :::*

LISTEN    0    128                     *:22                                    *:*

LISTEN    0    128                     *:51574                                 *:*

LISTEN    0    128                         127.0.0.1:631                                   *:*

LISTEN    0    128                        ::1:631                                    :::*

LISTEN    0    100                        ::1:25                                    :::*

LISTEN    0    100                     127.0.0.1:25                                 *:*

(实际应用)ELK环境搭建

两台的配置都一样就是上面的IP和note名称要配置不一样就行

(实际应用)ELK环境搭建

(实际应用)ELK环境搭建

[root@test5 ~]# ss -tln

State    Recv-Q Send-Q             Local Address:Port              Peer Address:Port

LISTEN    0    128                     *:45822                   *:*

LISTEN    0    128                    :::39620                   :::*

LISTEN    0    128                    :::111                    :::*

LISTEN    0    128                     *:111                    *:*

LISTEN    0    50                     :::9200                   :::*

LISTEN    0    50                     :::9300                   :::*

LISTEN    0    128                    :::22                    :::*

LISTEN    0    128                     *:22                     *:*

LISTEN    0    128                 127.0.0.1:631                    *:*

LISTEN    0    128                    ::1:631                    :::*

LISTEN    0    100                    ::1:25                    :::*

LISTEN    0    100                 127.0.0.1:25                     *:*

安装插件 head和kopf 之后访问 ip:9200/_plugin/head 和ip:9200/_plugin/kopf (插件可以图形查看elasticsearch的状态和删除创建索引)

/usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf

/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head

[root@test5]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf

-> Installing lmenezes/elasticsearch-kopf...

Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...

Downloading ......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE

Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...

NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)

Installed kopf into /usr/share/elasticsearch/plugins/kopf

[root@test5 ]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head

-> Installing mobz/elasticsearch-head...

Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...

Downloading ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE

Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...

NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)

Installed head into /usr/share/elasticsearch/plugins/head

(实际应用)ELK环境搭建

(实际应用)ELK环境搭建

二、安装nginx和logstash软件

yum -y install zlib zlib-devel openssl openssl--devel pcre pcre-devel

nginx-1.8.1-1.el6.ngx.x86_64.rpm

logstash-2.3.3-1.noarch.rpm

jdk-8u51-linux-x64.rpm

在test1上安装好nginx服务 就是收集它的日志呢

日志在/var/log/nginx/access.log

然后在test1上安装logstash-2.3.3-1.noarch.rpm

yum remove java-1.7.0-openjdk

rpm -ivh jdk-8u91-linux-x64.rpm

rpm -ivh logstash-2.3.3-1.noarch.rpm

/etc/init.d/logstash start #启动服务

/opt/logstash/bin/logstash -e "input {stdin{}} output{stdout{ codec=>"rubydebug"}}" #检测环境 执行这个命令检测环境正常否,启动完成后 直接输入东西就会出现

Settings: Default pipeline workers: 1

Pipeline main started

hello world

{

"message" => "hello world",

"@version" => "1",

"@timestamp" => "2017-05-24T08:04:46.993Z",

"host" => "0.0.0.0"

}

之后输入/opt/logstash/bin/logstash -e 'input {stdin{}} output{ elasticsearch { hosts => ["192.168.253.200:9200"] index => "test"}}'

 就是输入东西到253.200的elasticsearch上 会在/path/to/data/myelk/nodes/0/indices 生成你名称test索引文件目录 可以多输入几个到253.200的目录看看有没有文件有就证明正常。

[root@test4 ~]# ls /path/to/data/myelk/nodes/0/indices/

test

之后在test1的/etc/logstash/conf.d 建立以.conf结尾的配置文件,我收集nginx就叫nginx.conf了内容如下

[root@test1 nginx]# cd /etc/logstash/conf.d/

[root@test1 conf.d]# ls

nginx.conf

[root@test1 conf.d]# cat nginx.conf

input {

file {

type => "accesslog"

path => "/var/log/nginx/access.log"

start_position => "beginning"

}

}

output {

if [type] == "accesslog" {

elasticsearch  {

hosts => ["192.168.253.200"]

index => "nginx-access-%{+YYYY.MM.dd}"

}

}

}

/etc/init.d/logstash configtest

ps -ef |grep java

/opt/logstash/bin/logstash -f nginx.conf

之后在elasticearch查看有没有索引生成。多访问下nginx服务

如果没有就修改这个文件

vi /etc/init.d/logstash

LS_USER=root  ###把这里换成root或者把访问的日志加个权限可以让logstash可以读取它 重启服务就会生成索引了

LS_GROUP=root

LS_HOME=/var/lib/logstash

LS_HEAP_SIZE="1g"

LS_LOG_DIR=/var/log/logstash

LS_LOG_FILE="${LS_LOG_DIR}/$name.log"

LS_CONF_DIR=/etc/logstash/conf.d

LS_OPEN_FILES=16384

LS_NICE=19

KILL_ON_STOP_TIMEOUT=${KILL_ON_STOP_TIMEOUT-0} #default value is zero to this variable but could be updated by user request

LS_OPTS=""

test4查看:

[root@test4 ~]# ls /path/to/data/myelk/nodes/0/indices/

nginx-access-2017.05.23  test

[root@test1 logstash]# cat logstash.log

{:timestamp=>"2017-05-24T16:05:19.659000+0800", :message=>"Pipeline main started"}

三、安装kibana软件

上面的都安装完成后在test2上面安装kibana

rpm -ivh kibana-4.5.1-1.x86_64.rpm

编辑配置文件在这里/opt/kibana/config/kibana.yml 就修改下面几项就行

server.port: 5601 端口

server.host: "0.0.0.0" 监听

elasticsearch.url: "http://192.168.48.200:9200" elasticsearch地址

(实际应用)ELK环境搭建

/etc/init.d/kibana start 启动服务

访问kibana http://ip:5601

(实际应用)ELK环境搭建

添加展示的索引,就是在上面定义的 nginx-access-2016.07.03

(实际应用)ELK环境搭建

(实际应用)ELK环境搭建

(实际应用)ELK环境搭建

(实际应用)ELK环境搭建

配置kibana上面的收集Nginx日志的logstash

在kibana那台服务器上面安装logstash(按照之前的步骤安装)

然后再logstash的/etc/logstash/conf.d/下面

写一个配置文件:

[root@test2 conf.d]# vim nginx.conf

input {

file {

type => "accesslog"

path => "/var/log/nginx/access.log"

start_position => "beginning"

}

}

output {

if [type] == "accesslog" {

elasticsearch  {

hosts => ["192.168.253.200"]

index => "nginx-access-%{+MM.dd.YYYY}"

}

}

}

/opt/logstash/bin/logstash -f nginx.conf

查看Elasticsearch中多出了一个以月-日-年的Nginx访问日志索引

[root@test4 ~]# ls /path/to/data/myelk/nodes/0/indices/

nginx-access-05.23.2017  nginx-access-2017.05.23  test

然后在kibana,浏览器上面按照之前的创建,生成一个新的日志文件

(实际应用)ELK环境搭建

四、其他的一些配置。

kibana是直接访问的比较不安全,我们需要用nginx访问代理,并设置权限用户名和密码访问

先在kibana服务器上安装nginx 不介绍了

在nginx里面配置

#############################################################################

server

  {

      listen      80;

      server_name  localhost;

  auth_basic "Restricted Access";

      auth_basic_user_file /usr/local/nginx/conf/htpasswd.users; #密码和用户

    location / {

      proxy_pass  http://localhost:5601; #代理kibana的5601之后就可以直接80访问了

      proxy_set_header Host $host;

      proxy_set_header X-Real-IP $remote_addr;

      proxy_set_header REMOTE-HOST $remote_addr;

      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      }

}

#############################################################################

创建密码和用户文件:htpasswd.users

需要安装httpd-tool包先安装它

 htpasswd -bc /usr/local/nginx/conf/htpasswd.users admin paswdadmin #前面是用户后面是密码

(实际应用)ELK环境搭建

之后通过访问需要密码和用户并且是80端口了

(实际应用)ELK环境搭建

另外有需要云服务器可以了解下创新互联cdcxhl.cn,海内外云服务器15元起步,三天无理由+7*72小时售后在线,公司持有idc许可证,提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案,具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势,专为企业上云打造定制,能够满足用户丰富、多元化的应用场景需求。


名称栏目:(实际应用)ELK环境搭建-创新互联
链接URL:http://myzitong.com/article/cocshh.html