[OpenSearch] docker-elk 에서 AWS ES 로 옮기기 - 2. docker-logstash

docker-elk 에서 Elasticsearch 와 Kibana 는 AWS 를 이용하기에, Logstash 만 따로 띄워주면 됐다.

AWS Elastic Container Service (ECS) 에 띄우는 걸 최종 목표로 하고,

우선 AWS EC2 인스턴스를 띄워 docker 로 띄우기로 결정했다.

 

※ 사전준비 작업
- AWS Elasticsearch Service 생성하기
- AWS Elastic Container Registry(ECR) 세팅하기
- Bitbucket Pipelines 세팅하기

---

docker-logstash를 AWS EC2 인스턴스에 띄우기

(근데 이제 삽질을 곁들인)

---

▷ bitbucket-pipelines.yml

options:
  docker: true            
  size: 2x    
definitions:
  services:
    docker:
      memory: 2048
pipelines:
  branches:
    master:
      - step:
          name: Docker Logstash with AWS Elasticsearch Service
          services:
            - docker
          image: atlassian/pipelines-awscli
          script:
            - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ECR_REPOSITORY
            - IMAGE=$AWS_ECR_REPOSITORY/$AWS_ECR_REPOSITORY_NAME
            - TAG=latest
            - docker build -t $IMAGE:$TAG . 
              --build-arg ES_HOST=$ES_HOST 
              --build-arg ES_USER=$ES_USER 
              --build-arg ES_PASSWORD=$ES_PASSWORD 
            - docker push $IMAGE:$TAG

 

▷ Dockerfile

FROM docker.elastic.co/logstash/logstash-oss:7.9.0

ARG ES_HOST=ES_HOST
ENV ES_HOST=$ES_HOST
ARG ES_USER=ES_USER
ENV ES_USER=$ES_USER
ARG ES_PASSWORD=ES_PASSWORD
ENV ES_PASSWORD=$ES_PASSWORD

RUN logstash-plugin install logstash-output-amazon_es

RUN rm -f /usr/share/logstash/pipeline/logstash.conf
ADD pipeline/ /usr/share/logstash/pipeline/
ADD config/ /usr/share/logstash/config/

WORKDIR /usr/share/logstash

EXPOSE 5000

 

▷ logstash.conf

input {
    beats {
        port => 5000
        codec => json
        type => "filebeat"
    }
}

filter {
    json {
        source => "message"
    }
    date {
        match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
        target => "@timestamp"
    }
}

output {
  amazon_es {
      hosts => ["${ES_HOST}"]
      ssl => true
      region => "${AWS_REGION}"
      index => "%{[fields][log_type]}-%{+YYYY.MM.dd}"
      user => "${ES_USER}"
      password => "${ES_PASSWORD}"
  }
 }

 

▷ AWS EC2 인스턴스에 접속

$ docker pull $AWS_ECR_REPOSITORY/$AWS_ECR_REPOSITORY_NAME:$TAG
$ docker run -p 5000:5000 -p 9600:9600 -d --name logstash $AWS_ECR_REPOSITORY/$AWS_ECR_REPOSITORY_NAME:$TAG

 

---

[2021-04-19T02:56:07,118][INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://root:xxxxxx@${ES_HOST}:443/, :path=>"/"} [2021-04-19T02:56:07,140][WARN ][logstash.outputs.amazonelasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://root:xxxxxx@${ES_HOST}:443/", :error_type=>LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL '${ES_HOST}:443/'"}

---

---

 

▷ logstash.conf

output {
  amazon_es {
      hosts => ["${ES_HOST}"]
      ssl => true
      region => "${AWS_REGION}"
      index => "%{[fields][log_type]}-%{+YYYY.MM.dd}"
      aws_access_key_id => "${AWS_ACCESS_KEY_ID}"
      aws_secret_access_key => "${AWS_SECRET_ACCESS_KEY}"
      user => "${ES_USER}"
      password => "${ES_PASSWORD}"
  }
 }

 

---

[2021-04-19T02:56:07,118][INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://root:xxxxxx@${ES_HOST}:443/, :path=>"/"} [2021-04-19T02:56:07,140][WARN ][logstash.outputs.amazonelasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://root:xxxxxx@${ES_HOST}:443/", :error_type=>LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL '${ES_HOST}:443/'"}

---

결론은 아래와 같이 작성하면 된다.

output {
    elasticsearch {
        hosts => ["${ES_HOST}"]
        ssl => true
        index => "%{[fields][log_type]}-%{+YYYY.MM.dd}"
        user => "${ES_USER}"
        password => "${ES_PASSWORD}"
        ilm_enabled => false
    }
}

---

찾아보니 Amazon ES 도메인에서 HTTP 기본 인증으로 세분화된 액세스 제어 를 사용하는 경우에는

다른 Elasticsearch 클러스터와 유사하기 때문에 기존대로 elasticsearch 플러그인을 사용하면 되고,

 

IAM 기반 도메인 액세스 정책 또는 IAM 마스터 사용자의 세분화된 액세스 제어 를 사용하는 경우에

logstash-output-amazon-es 플러그인을 사용하면 된다고 한다.

 

(출처: docs.aws.amazon.com/ko_kr/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html)

 

한글이 이렇게나 어렵습니다.