Dokcer - 搭建 ELK Stack日志管理服务

背景简介

基于 Docker Compose 部署 ELK Stack（Elasticsearch、Logstash、Kibana），通过 Logstash 处理并转发到 Elasticsearch，最后通过 Kibana 进行可视化和分析。

前置信息

Docker 28.2.2

详细步骤

文件架构

/home/your-user/elk-docker/  # 你的项目目录
 |-- docker-compose.yml
 |-- .env
 |-- data
   |-- elasticsearch/
   |    |-- instance.yml
   |    |-- node1
   |    |-- node2
   |    |-- node3 
   |-- certs/
   |-- logstash/
        |-- config/
        |    |-- logstash.yml
        |-- pipeline/
             |-- logstash.conf

配置文件夹权限

配置 elasticsearch 用户权限（UID 1000）

sudo chown -R 1000:1000 ./data/elasticsearch ./data/certs

配置环境变量

编辑 .env 文件

# Elastic Stack 版本
ELK_VERSION=8.17.0

# 项目名称
COMPOSE_PROJECT_NAME=elk-prod

# Elasticsearch 集群名称
CLUSTER_NAME=es-prod-cluster

# 设置 elastic 超级用户的密码
ELASTIC_PASSWORD=your_strong_elastic_password

# 设置 kibana_system 用户的密码
KIBANA_PASSWORD=your_strong_kibana_password

# 设置 logstash_system 用户的密码 (用于监控)
LOGSTASH_SYSTEM_PASSWORD=your_strong_logstash_system_password

# 设置 logstash_internal 用户的密码 (用于写入数据)
LOGSTASH_INTERNAL_PASSWORD=your_strong_logstash_internal_password

# Elasticsearch JVM 堆内存设置
ES_JAVA_OPTS=-Xms2g -Xmx2g

配置 ELK Stack 服务

docker-compose.yaml 配置
- 【elasticsearch 参考文档】
- 【logstash 参考文档】
- 【kibana 参考文档】

services:
  # Elasticsearch Create Certs Service: 仅用于一次性创建证书
  create-certs:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
    container_name: create-certs
    volumes:
      - ./data/certs:/usr/share/elasticsearch/config/certs
      - ./data/elasticsearch/instances.yml:/usr/share/elasticsearch/instances.yml:ro
    command: >
      bash -c '
        if [ ! -f config/certs/ca/ca.crt ]; then
          echo "Generating CA and certificates...";
          bin/elasticsearch-certutil ca --silent --pem --out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
          bin/elasticsearch-certutil cert --silent --pem --in /usr/share/elasticsearch/instances.yml --out config/certs/certs.zip --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
          chown -R 1000:1000 config/certs;
          echo "Certificates generated successfully.";
        else
          echo "Certificates already exist.";
        fi;
      '
    user: "0"
    networks:
      - elastic-net

  # Elasticsearch Node 1
  elasticsearch-node1:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
    container_name: elasticsearch-node1
    restart: unless-stopped
    environment:
      - node.name=elasticsearch-node1
      - cluster.name=${CLUSTER_NAME}
      - discovery.seed_hosts=elasticsearch-node2,elasticsearch-node3
      - cluster.initial_master_nodes=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=${ES_JAVA_OPTS}"
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.certificate=certs/elasticsearch-node1/elasticsearch-node1.crt
      - xpack.security.http.ssl.key=certs/elasticsearch-node1/elasticsearch-node1.key
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-node1/elasticsearch-node1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-node1/elasticsearch-node1.key
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./data/elasticsearch/node1:/usr/share/elasticsearch/data
      - ./data/certs:/usr/share/elasticsearch/config/certs:ro
    ports:
      - "9200:9200"
    networks:
      - elastic-net
    depends_on:
      create-certs:
        condition: service_completed_successfully
    healthcheck:
      test: ["CMD-SHELL", "curl -k -u elastic:${ELASTIC_PASSWORD} -s https://localhost:9200/_cluster/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5

  # Elasticsearch Node 2 & 3 (配置类似，仅 node.name 和 ports 不同)
  elasticsearch-node2:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
    container_name: elasticsearch-node2
    restart: unless-stopped
    environment:
      - node.name=elasticsearch-node2
      - cluster.name=${CLUSTER_NAME}
      - discovery.seed_hosts=elasticsearch-node1,elasticsearch-node3
      - cluster.initial_master_nodes=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=${ES_JAVA_OPTS}"
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.certificate=certs/elasticsearch-node2/elasticsearch-node2.crt
      - xpack.security.http.ssl.key=certs/elasticsearch-node2/elasticsearch-node2.key
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-node2/elasticsearch-node2.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-node2/elasticsearch-node2.key
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./data/elasticsearch/node2:/usr/share/elasticsearch/data
      - ./data/certs:/usr/share/elasticsearch/config/certs:ro
    networks:
      - elastic-net
    depends_on:
      create-certs:
        condition: service_completed_successfully

  elasticsearch-node3:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
    container_name: elasticsearch-node3
    restart: unless-stopped
    environment:
      - node.name=elasticsearch-node3
      - cluster.name=${CLUSTER_NAME}
      - discovery.seed_hosts=elasticsearch-node1,elasticsearch-node2
      - cluster.initial_master_nodes=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=${ES_JAVA_OPTS}"
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.certificate=certs/elasticsearch-node3/elasticsearch-node3.crt
      - xpack.security.http.ssl.key=certs/elasticsearch-node3/elasticsearch-node3.key
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-node3/elasticsearch-node3.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-node3/elasticsearch-node3.key
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./data/elasticsearch/node3:/usr/share/elasticsearch/data
      - ./data/certs:/usr/share/elasticsearch/config/certs:ro
    networks:
      - elastic-net
    depends_on:
      create-certs:
        condition: service_completed_successfully
  
  # 一次性设置密码服务
  setup-passwords:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
    container_name: setup-passwords
    volumes:
      - ./data/certs:/usr/share/elasticsearch/config/certs:ro
    environment:
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - KIBANA_PASSWORD=${KIBANA_PASSWORD}
      - LOGSTASH_SYSTEM_PASSWORD=${LOGSTASH_SYSTEM_PASSWORD}
      - LOGSTASH_INTERNAL_PASSWORD=${LOGSTASH_INTERNAL_PASSWORD}
    command: >
      bash -c '
        echo "Waiting for Elasticsearch cluster to be green...";
        until curl -s --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} https://elasticsearch-node1:9200/_cluster/health | grep -q '"'"'status":"green'"'"'; do sleep 10; done;
        echo "Cluster is green. Setting passwords...";
        curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://elasticsearch-node1:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}";
        curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://elasticsearch-node1:9200/_security/user/logstash_system/_password -d "{\"password\":\"${LOGSTASH_SYSTEM_PASSWORD}\"}";
        echo "All passwords have been set!";
      '
    user: "0"
    networks:
      - elastic-net
    depends_on:
      elasticsearch-node1:
        condition: service_healthy # 等待 node1 健康检查通过

  # Logstash
  logstash:
    image: docker.elastic.co/logstash/logstash:${ELK_VERSION}
    container_name: logstash
    restart: unless-stopped
    volumes:
      - ./data/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./data/logstash/pipeline:/usr/share/logstash/pipeline:ro
      - ./data/certs:/usr/share/logstash/config/certs:ro
    ports:
      - "5044:5044"
    environment:
      LS_JAVA_OPTS: "-Xmx1g -Xms1g"
      LOGSTASH_SYSTEM_PASSWORD: ${LOGSTASH_SYSTEM_PASSWORD}
    networks:
      - elastic-net
    depends_on:
      setup-passwords:
        condition: service_completed_successfully # 等待密码设置成功

  # Kibana
  kibana:
    image: docker.elastic.co/kibana/kibana:${ELK_VERSION}
    container_name: kibana
    restart: unless-stopped
    ports:
      - "5601:5601"
    environment:
      ELASTICSEARCH_HOSTS: https://elasticsearch-node1:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: ${KIBANA_PASSWORD}
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: /usr/share/kibana/config/certs/ca/ca.crt
      ELASTICSEARCH_SSL_VERIFICATION_MODE: certificate
      SERVER_SSL_ENABLED: true
      SERVER_SSL_CERTIFICATE: /usr/share/kibana/config/certs/elasticsearch-node1/elasticsearch-node1.crt
      SERVER_SSL_KEY: /usr/share/kibana/config/certs/elasticsearch-node1/elasticsearch-node1.key
    volumes:
      - ./data/certs:/usr/share/kibana/config/certs:ro
    networks:
      - elastic-net
    depends_on:
      setup-passwords:
        condition: service_completed_successfully # 等待密码设置成功

networks:
  elastic-net:
    driver: bridge

配置 logstash.yml


api.http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "https://elasticsearch-node1:9200" ]
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "${LOGSTASH_SYSTEM_PASSWORD}"
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/certs/ca/ca.crt"
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
pipeline.buffer.type: heap

配置logstash.conf

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["https://elasticsearch-node1:9200"]
    # 启用数据流
    data_stream => true
    # 为数据流设置一个名称，它会自动生成索引名
    # 例如，数据流名为 "logs-logstash.generic"，索引名会像 ".ds-logs-logstash.generic-2023.10.21-000001"
    data_stream_type => "logs"
    data_stream_dataset => "logstash"
    data_stream_namespace => "generic"
    
    user => "logstash_system"
    password => "${LOGSTASH_SYSTEM_PASSWORD}"
    # 启用 ssl
    ssl => true
    cacert => "/usr/share/logstash/config/certs/ca/ca.crt"
  }
}

配置 instances.yml

instances:
  - name: 'elasticsearch-node1'
    dns:
      - 'elasticsearch-node1'
      - 'localhost'
  - name: 'elasticsearch-node2'
    dns:
      - 'elasticsearch-node2'
  - name: 'elasticsearch-node3'
    dns:
      - 'elasticsearch-node3'

启动服务

初始化 elasticsearch

docker compose up -d

[+] Running 8/8
 ✔ Network elk-prod_elastic-net   Created 
 ✔ Container create-certs         Exited         
 ✔ Container elasticsearch-node1  Healthy        
 ✔ Container elasticsearch-node2  Running      
 ✔ Container elasticsearch-node3  Started       
 ✔ Container setup-passwords      Exited          
 ✔ Container kibana               Started    
 ✔ Container logstash             Started

查看 setup 容器日志（可根据实际需要查看其它容器日志）

docker logs -f create-certs

Generating CA and certificates...
Archive:  config/certs/ca.zip
   creating: config/certs/ca/
  inflating: config/certs/ca/ca.crt  
  inflating: config/certs/ca/ca.key  
Archive:  config/certs/certs.zip
   creating: config/certs/elasticsearch-node1/
  inflating: config/certs/elasticsearch-node1/elasticsearch-node1.crt  
  inflating: config/certs/elasticsearch-node1/elasticsearch-node1.key  
   creating: config/certs/elasticsearch-node2/
  inflating: config/certs/elasticsearch-node2/elasticsearch-node2.crt  
  inflating: config/certs/elasticsearch-node2/elasticsearch-node2.key  
   creating: config/certs/elasticsearch-node3/
  inflating: config/certs/elasticsearch-node3/elasticsearch-node3.crt  
  inflating: config/certs/elasticsearch-node3/elasticsearch-node3.key  
Certificates generated successfully.

登录 kibana 查看信息

输入地址（例如：https://192.168.122.1:5601 ）并使用 elastic 账号密码登录

首次登录点击 Explore on my own 即可

接下来就可以根据自己的需求配置啦！

以上便是本文的全部内容，感谢您的阅读，如遇到任何问题，欢迎在评论区留言讨论。

菜单

分享

Dokcer - 搭建 ELK Stack日志管理服务

背景简介

前置信息

详细步骤

文件架构

配置文件夹权限

配置环境变量

配置 ELK Stack 服务

启动服务

登录 kibana 查看信息

评论

Dokcer - 搭建 ELK Stack日志管理服务

Rclone - 挂载 WebDAV

Rclone - Ubuntu安装

Bash - 创建可供其他脚本调用的方法脚本

Bash - 增量备份文件夹

Bash - 记录日志 - 带参数

Bash - 记录日志

Mac - 安装 xcode-select

Mac - 安装 Homebrew

AList - 添加网盘