PangYeon`s IT Log

OpenSearch metrics 을 Prometheus에서 가져오는 방법이다.

현재기준 OpenSearch 가 2.19.1 이 나왔는데 아직 플러그인은 2.17.1 이므로 버전을 맞춰주어야한다.

Git Issue 에 누가 상세히 설명해주어서 따라한다.(https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch/issues/330)

1. 플러그인 신규 버전 생성

우선 플러그인 git url에 접속한다

https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch

최신 소스를 다운받고 gradle.properties 파일의 version부분을 OpenSearch 버전과 맞춰준다

#group = org.opensearch.plugin.prometheus

# An actual version of plugin
#### 이 부분을 변경해준다. ####
version = 2.19.1.0

# Leave this property empty, it is assigned during the gradle build execution (yes, it is a hack! see issue #324)
opensearch_version =

# A version of OpenSearch cluster to run BWC tests against
BWCversion = 2.17.1

# A version of plugin to deploy to BWC clusters
BWCPluginVersion = 2.17.1.0

pluginName = prometheus-exporter
pluginClassname = org.opensearch.plugin.prometheus.PrometheusExporterPlugin
pluginDescription = Prometheus exporter plugin for OpenSearch

#github_token = secret
#github_owner = TBD
#github_repo = prometheus-exporter

그 후 빌드해주면 prometheus-exporter-2.19.1.0.zip 플러그인이 나타난다.

# 빌드
./gradlew clean build

# 빌드된 파일
./build/distributions/prometheus-exporter-2.19.1.0.zip

2. OpenSearch 및 ServiceMonitor 를 위한 role 배포

## ServiceMonitor에 사용될 패스워드
apiVersion: v1
kind: Secret
metadata:
  name: opensearch-admin
  namespace: logging
type: Opaque
data:
  username: YWRtaW4=
  password: c2luZ2xlTm9kZTEh
  
  
## 생성한 플러그인
kubectl create configmap opensearch-plugin --from-file prometheus-exporter-2.19.1.0.zi -n logging

## ClusterRole and Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: opensearch-metrics
  namespace: logging
rules:
- apiGroups: [""]
  verbs: ["get", "list", "watch"]
  resources: ["services", "endpoints", "pods"]
- nonResourceURLs: ["/_prometheus/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRolebinding
metadata:
  name: opensearch-metrics
  namespace: logging
subjects:
- kind: ServiceAccount
  namespace: monitor
  name: prometheus-k8s
roleRef:
  kind: ClusterRole
  name: opensearch-metrics
  apiGroup: rbac.authorization.k8s.io


## opensearch 및 prometheus plugin, ServiceMontior 설치
helm install opensearch-single opensearch/opensearch --namespace logging \
--set singleNode=true \
--set persistence.enabled=false \
--set-json 'extraEnvs=[{"name": "OPENSEARCH_INITIAL_ADMIN_PASSWORD", "value": "singleNode1!"}]' \
--set plugins.enabled=true \
--set plugins.installList[0]='file:/usr/share/opensearch/prometheus-export-2.19.1.0.zip' \
--set-json 'extraVolumes=[{"name": "plugin", "configMap": {"name": "opensearch-plugin"}}]' \
--set-json 'extraVolumeMounts=[{"mountPath": "/usr/share/opensearch/prometheus-export-2.19.1.0.zip", "name": "plugin", "subPath": "prometheus-exporter-2.19.1.0.zip"}]' \
--set serviceMonitor.enabled=true \
--set serviceMonitor.basicAuth.enabled=true \
--set serviceMonitor.basicAuth.existingSecret=opensearch-admin \
--set serviceMonitor.path='/_prometheus/metrics' \
--set serviceMonitor.scheme=https \
--set serviceMonitor.tlsConfig.insecureSkipVerify=true

ConfigMap general file volume mount

kubectl create configmap test-file --from-file test.tar -n test

apiVersion: v1
kind: Pod
metadata:
  name: test
  namespace: test
spec:
  containers:
  - image: ...
    name: ...
    volumeMounts:
    - mountPath: /usr/share/test.tar
      subPath: test.tar
      name: filetest
  volumes:
  - name: filetest
    configMap:
      name: test-file

Docker 및 Ctr 로 하는 법 -

2024.04.10 - [Develop/기타 작업] - Docker architecture, save, load, tag 및 ctr tag, export, import, push

buildah 에서 docker save, ctr export 과 같은 이미지를 로컬에 저장하고 싶으면,

buildah push 명령어를 사용하면된다.

# buildah push <image-id> oci-archive:/경로/이미지.tar:tag
buildah push 5854065bab5a oci-archive:/root/test.tar:latest

이후 ctr image imports 나 docker load 등으로 이미지를 올린 후 조회하면, 아래와 import-yyyy-mm-dd같이 나온다.

docker.io/library/import-2025-03-18 

crictl images
IMAGE                                                TAG                 IMAGE ID            SIZE
docker.io/library/import-2025-03-18                  latest              5854065bab5af       62.9MB

적절히 변경하여 사용한다.

기타 : buildah 로 push 후 ctr 로 업로드하면 crictl에 조회는 되지만 태그 변경이 안되는데,

이 경우 ctr imge list 로 조회 후 정확한 이름을 찾아서 변경하면된다.

(기본적으로 변경시 Source가 docker.io/library/import-2025-03-18 이 아닌 import-2025-03-18 로 Source를 선택 후

태그를 변경하면된다.

Git 저장소 릴리즈 알림이다.

개인의 경우 Github의 Watch 기능을 이용하면 알림을 받을 수 있지만,

해당기능은 개인만 받는것이 아닌 다수가 메일 또는 슬랙으로 받을 수 있으며,

알림 받기를 원하는 저장소를 통합해서 관리할 수 있음.

Github actions 또는 k8s cronjob 등으로 동작 가능하다.

소스코드 및 사용법 : https://github.com/wonkwangyeon/git-release-notify

• 개발할 경우 root 경로에 _env 파일을 .env로 수정 후 아래 환경변수 세팅
• Github Actions로 사용시 repository secrets 및 env 설정
• k8s CronJob 일 경우 DockerBuild 하여 yaml 생성 후 사용

# GIT_URL은 반드시 release로 끝나는 url이여야 함. 여러Repo 입력시 아래와 같이 입력.
GIT_URL="https://github.com/wonkwangyeon/git-release-notify/releases
https://github.com/wonkwangyeon/Run-Multiple-Program/releases"
MAIL_ENABLED=true
SMTP_SERVER="smtp.test.com"
SMTP_PORT=123
SMTP_TLS=false
SMTP_USER="test@test.com"
SMTP_PASSWORD=""
# Receiver가 한명일 경우 test@test.com 하나만 입력. 여러명일 경우 ','로 구분해서 나열
RECEIVER="test@test.com,testuser2@test.com"
SLACK_ENABLED=false
SLACK_WEBHOOK_URL=""

k8s cronjob

## docker build
docker build -t git-release-notify:latest .

## cronjon.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: git-release-notify
spec:
  schedule: "0 9 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: git-release-notify
            image: git-release-notify:latest
            imagePullPolicy: IfNotPresent
            env:
            - name: GIT_URL
              value: ""
            - name: MAIL_ENABLED
              value: ""
            - name: SMTP_SERVER
              value: ""
            - name: SMTP_PORT
              value: ""
            - name: SMTP_TLS
              value: ""
            - name: SMTP_USER
              value: ""
            - name: SMTP_PASSWORD
              value: ""
            - name: RECEIVER
              value: ""
            - name: SLACK_ENABLED
              value: ""
            - name: SLACK_WEBHOOK_URL
              value: ""
          restartPolicy: OnFailure

OpenSearch URL을 직접 호출하는게 아닌, OpenSearch Dashboards URL을 호출하여 OpenSearch API를 사용할 수 있다.

/api/console/proxy 를 호출하여 사용하면된다.

curl -u admin:password -H "osd-xsrf: true" -H "Content-Type: application/json" -XPOST "http://os-dashboard.wky.kr:30080/api/console/proxy?path=_cluster/health&method=GET"
## 결과
{"cluster_name":"opensearch-cluster","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"discovered_master":true,"discovered_cluster_manager":true,"active_primary_shards":9,"active_shards":9,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":81.81818181818183}

kubernetes dashboard를 설치하게 되면 기본적으로 kong proxy pod가 생성되면서,

이 kong proxy를 통해 사용하게되는데, 이를 비활성화하고 따로 ingress 적용을 통해 kubernetes dashboard에 접속하고

사용할 수 있다.

일반 ingress 적용 가이드 -

2024.10.22 - [Develop/k8s] - kubernetes dashboard ingress 연결 방법

1. 설치시 kong proxy disabled

helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard \
--create-namespace --namespace kubernetes-dashboard \
--set kong.enabled=false

2. ingress 적용

아래 URL과 service 조회를 통해 ingress를 생성한다

https://github.com/kubernetes/dashboard/blob/master/hack/gateway/prod.kong.yml

kubectl get svc -n kubernetes-dashboard
NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes-dashboard-api               ClusterIP   10.106.25.34     <none>        8000/TCP   6m13s
kubernetes-dashboard-auth              ClusterIP   10.101.168.28    <none>        8000/TCP   6m13s
kubernetes-dashboard-metrics-scraper   ClusterIP   10.104.27.35     <none>        8000/TCP   6m13s
kubernetes-dashboard-web               ClusterIP   10.109.176.153   <none>        8000/TCP   6m13s

dashboard ui에 로그인하기위해선 tls 적용이 필요하다.

Kubernetes Dashboard 아래 소스를 보면 setTokenCookie 부분이 있는데, 이 부분 때문에 tls가 필요해보인다.
아님말고...?
dashboard-master/modules/web/src/common/services/global/authentication.ts
---
private setTokenCookie_(token: string): void { if (this.isCurrentProtocolSecure_()) { this.cookies_.set(this.config_.authTokenCookieName, token, null, null, null, true, 'Strict'); return; } if (this.isCurrentDomainSecure_()) { this.cookies_.set(this.config_.authTokenCookieName, token, null, null, location.hostname, false, 'Strict'); } }
---

tls 적용을 위한 인증서 및 secret 생성

# 인증서 생성
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=dashboard.wky.kr/O=Kubernetes" -addext "subjectAltName = DNS:dashboard.wky.kr"

# secret 생성
kubectl create secret tls tls-dashboard --key tls.key --cert tls.crt -n kubernetes-dashboard

ingress 생성

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ingressClassName: nginx
  rules:
  - host: dashboard.wky.kr
    http:
      paths:
      - path: /api/v1/login
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-auth
            port:
              number: 8000
      - path: /api/v1/csrftoken/login
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-auth
            port:
              number: 8000
      - path: /api/v1/me
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-auth
            port:
              number: 8000
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-api
            port:
              number: 8000
      - path: /metrics
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-api
            port:
              number: 8000
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-web
            port:
              number: 8000
  tls:
  - hosts:
    - dashboard.wky.kr
    secretName: tls-dashboard

3. 로그인테스트를 위한 계정 및 토큰 발급

계정생성

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

토큰발급

kubectl -n kubernetes-dashboard create token admin-user

4. 로그인 및 확인

tls 적용을 했으므로 반드시 https로 접속 후 테스트한다.

이러다 설치만 장인되겠다

해당 글은 helm chart로 설치하는 정리 가이드이다.

가이드 작성시 상세 설명은 따로 하지 않는다. 

OpenSearch Operator로 설치는 가이드보면 설치자체는 어려운것은 없으나, 인증서 부분에서 상당히 복잡하기 때문에 아래 글 참고한다.

2024.11.28 - [Develop/OpenSearch ／ FluentBit] - OpenSearch Operator Certificate 설정

Single Node 로 설치는 아래 글 참고한다.

2025.03.06 - [Develop/OpenSearch ／ FluentBit] - OpenSearch SingleNode

OpenSearch FluentBit 연동은 아래 글 참고한다.

2024.11.03 - [Develop/OpenSearch ／ FluentBit] - OpenSearch, Dashboard 설치 및 Fluent-Bit OpenSeach 연동

OpenSearch Operator Coordinator 설치는 아래 글 참고한다.

2025.02.18 - [Develop/OpenSearch ／ FluentBit] - OpenSearch Operator Coordinator

1. Demo 인증서가아닌 self 인증서를 생성한다. 인증서가 있으면 건너 뛴다.

# root ca 생성
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/CN=opensearch" -out root-ca.pem -days 3650

# OpenSearch Operator가 OpenSearch Cluster를 관리하는 관리자 인증서
openssl genrsa -out admin-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
openssl req -new -key admin-key.pem -subj "/CN=opensearch-admin" -out admin.csr
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 3650

# Node간 DNS 통신을 위해 subjectAltName 생성
echo 'subjectAltName=DNS:opensearch-cluster-master,DNS:opensearch-cluster-master.logging,DNS:opensearch-cluster-master.logging.svc,DNS:opensearch-cluster-master.logging.svc.cluster.local' > san.ext

# OpenSearch Node간 인증서
openssl genrsa -out node-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node-key.pem
openssl req -new -key node-key.pem -subj "/CN=opensearch-cluster-master" -out node.csr
openssl x509 -req -in node.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node.pem -days 3650 -extfile san.ext

# OpenSearch yaml에 등록할 인증서 파일들 secret으로 생성
kubectl create ns logging
kubectl create secret generic opensearch-cert --from-file=root-ca.pem=root-ca.pem --from-file=admin.pem=admin.pem --from-file=admin-key.pem=admin-key.pem \
--from-file=root-ca.pem=root-ca.pem --from-file=node.pem=node.pem --from-file=node-key.pem=node-key.pem -n logging

2. helm repo 추가

공식 가이드에 나온대로 진행한다.

helm repo add opensearch https://opensearch-project.github.io/helm-charts
helm repo upadte

3. OpenSearch에 사용할 관리자 계정 secret 생성

아래 링크에서 가져온 샘플이다. 필요에 따라 수정해서 사용하면 됨.

https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/opensearch-operator/examples/securityconfig-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
stringData:
      action_groups.yml: |-
         _meta:
           type: "actiongroups"
           config_version: 2
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2y$12$lJsHWchewGVcGlYgE3js/O4bkTZynETyXChAITarCHLz8cuaueIyq"
          reserved: true
          backend_roles:
          - "admin"
          description: "admin user"
        dashboarduser:
          hash: "$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."
          reserved: true
          description: "OpenSearch Dashboards user"
      nodes_dn.yml: |-
        _meta:
          type: "nodesdn"
          config_version: 2
      whitelist.yml: |-
        _meta:
          type: "whitelist"
          config_version: 2
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
        own_index:
          reserved: false
          users:
          - "*"
          description: "Allow full access to an index named like the username"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        manage_snapshots:
          reserved: false
          backend_roles:
          - "snapshotrestore"
        dashboard_server:
          reserved: true
          users:
          - "dashboarduser"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"
      config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "4"
                http_authenticator:
                  type: basic
                  challenge: true
                authentication_backend:
                  type: intern

4. OpenSearch Dashboards에 등록할 관리자 계정 Secret생성

vi admin-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: admin-credentials
  namespace: logging
type: Opaque
data:
  username: YWRtaW4= # admin
  password: YWRtaW4xMjM=  # admin123
  
kubectl apply -f admin-secret.yaml

5. OpenSearch 설치

기본적으로 helm chart에 values.yaml 을 수정해서 적용하는 것을 추천하나, 이번엔 바로 적용을 해본다.

인증서를 mount하고, opensearch.yml을 정의하고 securityConfigSecret을 추가하여 관리자 계정을 생성한다.

helm install opensearch-cluster opensearch/opensearch \
--namespace logging --create-namespace
--set clusterName='opensearch-cluster'
--set persistence.storageClass=nfs \ # StorgeClass 가 있으면 설정 없으면 삭제
--set persistence.enabled=false \ # EmptyDir로 실행하고 싶으면 설정 그게아니라면 따로 PV설정필요
# 기본적으로 아래로 적용
--set-json 'extraEnvs=[{"name": "DISABLE_INSTALL_DEMO_CONFIG", "value": "true"}] \
# SingleNode로 설치시 lifecycle 이용할려면 아래로 적용
--set-json 'extraEnvs=[{"name": "DISABLE_INSTALL_DEMO_CONFIG", "value": "true"}
{"name": "OPENSEARCH_USERNAME", "valueFrom": {"secretKeyRef": {"key": "username",
"name": "admin-credentials-secret"}}},
{"name": "OPENSEARCH_PASSWORD", "valueFrom": {"secretKeyRef": {"key": "password",
"name": "admin-credentials-secret"}}}]' \
--set-json 'extraVolumes=[{"name": "certs-secret" "secret": {"defaultMode": 384,
"secretName": "opensearch-cert"}}] \
--set-json 'extraVolumeMounts=[{"mountPath": "/usr/share/opensearch/config/certs",
"name": "certs-secret"}]' \
--set config.opensearch.yml="cluster.name: opensearch-cluster
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem
plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
plugins.security.ssl.http.pemcert_filepath: certs/node.pem
plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authz.admin_dn: ['CN=opensearch-admin']
plugins.security.nodes_dn: ['CN=opensearch*']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privilege: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: ['.opendistro-alerting-config', '.opendistro-alerting-alert*', '.opendistro-anomaly-results*', '.opendistro-anomaly-detector*', '.opendistro-anomaly-checkpoints', '.opendistro-anomaly-detection-state', '.opendistro-reports-*", ".opendistro-notifications-*', '.opendistro-notebooks', '.opendistro-asynchronous-search-response*']
node.max_local_storage_nodes: 3' # Single Node 이면 삭제하고 따옴표 주의
--set securityConfig.config.securityConfigSecret=securityconfig-secret"

기타.

https://opensearch.org/docs/latest/security/configuration/system-indices/

를 확인하면 관리자 계정으로도 사용할 수 없는 api들이 있음.

5. OpenSearch Dashboards 설치

4번에서 생성한 secret을 넣어준다.

helm install opensearch-dashboards opensearch/opensearch-dashboards \
-n logging --create-namespace \
--set-json 'extraEnvs=[{"name": "OPENSEARCH_USERNAME", "valueFrom": {"secretKeyRef": 
{"key": "username", "name": "admin-credentials"}}},
{"name": "OPENSEARCH_PASSWORD", "valueFrom": {"secretKeyRef": 
{"key": "password", "name": "admin-credentials"}}}]'

Ingress 설정은 맨위에 OpenSearch FluentBit 연동글을 참고한다.

1. OpenSearch Operator로 설치할 경우 지원하지않는다. 일반 helm으로 설치시 가능하다.

helm install opensearch-single opensearch/opensearch \
--namespace logging \
--set singleNode=true \
--set persistence.enabled=false \
--set-json 'extraEnvs=[{"name": "OPENSEARCH_INITIAL_ADMIN_PASSWORD", "value": "singleNode1!"}]'

2. 설치 완료 후 접속하여 Cluster 상태를 확인한다.

상태를 확인하면 yellow로 나타는데 이는 Single Node임에도 불구하고 replicas가 1로 설정되어있어서 그렇다

curl -k -u admin:singleNode1! https://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "opensearch-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 83.33333333333334
}

# 아래 명령어를 통해 대충 왜 yellow 인지 알 수 있다.
curl -k -u admin:singleNode1! https://localhost:9200/_cluster/allocation/explain?pretty

3. health Status 변경

_all/_settings 명령어를 통해 모든 index 의 replicas 0으로 변경

curl -k -XPUT -u admin:singleNode1! https://localhost:9200/_all/_settings \
-H 'Content-Type: application/json' -d'
{
  "index": {
     "number_of_replicas": 0
  }
}' --cert ./config/kirk.pem --key ./config/kirk-key.pem


# 다시 status 확인
curl -k -u admin:singleNode1! https://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "opensearch-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

--cert 와 --key 의 경우 현재 데모인증서를 사용해서 kirk.pem 과 kirk-key.pem을 사용하였다.

데모인증서가아닌 직접 인증서를 설정했으면 설정한대로 해야한다.

또한, 해당 API는 만약 Dashbaord를 사용하면 Dashboard의 devtool로도 적용되지 않는다.

적용되게 할려면 따로 opensearch.yml 에 설정해주어야 함. 아래 url 참고

https://opensearch.org/docs/1.3/security/configuration/system-indexes/

4. 또한 설치시 replicas 0으로 설정할 수 있다. 

lifecycle 에 index가 replicas 0으로 생성 되도록 template을 추가하는 postStart 를 추가하면된다.

lifecycle:
  postStart:
    exec:
      command:
      - bash
      - -c
      - |
        #!/bin/bash
        until curl -s "https://localhost:9200/_cluster/heatlh" -k -u admin:singleNode1! | grep -q "green\|yellow; do
        sleep 5
        done
        curl -k -u admin:singleNode1! -XPUT "https://localhost:9200/_index_template/single_init_template"
        -H 'Content-Type: application/json' -d'{
          "index_patterns": ["*"],
          "template": {
            "settings": {
              "number_of_replicas": 0
             }
           }
         }'

5. 기타

2.19.0 버전부터 Query Insights 라는 plugin이 설치되어 replicas가 1인 top_queries 인덱스가 자동으로 하나 생성되는데,

이것 때문에 health yellow로 되는데 반드시 0으로 수동으로 변경해주어야 green으로 돌아온다

helm chart harbor upload이다

harbor내에 프로젝트 생성 후 진행

helm registry login harbor.wky.kr --insecure
# Input id/password

# harbor에 chart 라는 프로젝트 생성했음
helm push <helm-chart-name>.tgz oci://harbor.wky.kr/chart --plain-http

OpenSearch Operator Coordinator 설정 방법

만약 아래 글을 통해 인증서를 생성했으면, subjectAltName 수정으로 인해 인증서 또한 수정해야한다.

2024.11.28 - [Develop/k8s] - OpenSearch Operator Certificate 설정

# root ca 생성
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/CN=opensearch" -out root-ca.pem -days 3650

# OpenSearch Operator가 OpenSearch Cluster를 관리하는 관리자 인증서
openssl genrsa -out admin-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
openssl req -new -key admin-key.pem -subj "/CN=opensearch-admin" -out admin.csr
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 3650

# Node간 DNS 통신을 위해 subjectAltName 생성
echo 'subjectAltName=DNS:opensearch-cluster-masters,DNS:opensearch-cluster-masters.opensearch,DNS:opensearch-cluster-masters.opensearch.svc,DNS:opensearch-cluster-masters.opensearch.svc.cluster.local,DNS:opensearch-cluster-coordinator,DNS:opensearch-cluster-coordinator.opensearch.svc.cluster.local,DNS:opensearch-cluster-coordinator.logging,DNS:opensearch-cluster-coordinator.logging.svc' > san.ext

# OpenSearch Node간 인증서
openssl genrsa -out node-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node-key.pem
openssl req -new -key node-key.pem -subj "/CN=opensearch-cluster-*" -out node.csr
openssl x509 -req -in node.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node.pem -days 3650 -extfile san.ext

# OpenSearch yaml에 등록할 인증서 파일들 secret으로 생성
kubectl create secret generic opensearch-cert-admin --from-file=ca.crt=root-ca.pem --from-file=tls.crt=admin.pem --from-file=tls.key=admin-key.pem -n opensearch
kubectl create secret generic opensearch-cert-node --from-file=ca.crt=root-ca.pem --from-file=tls.crt=node.pem --from-file=tls.key=node-key.pem -n opensearch

OpenSearchCluster yaml 파일 수정

Dashboard가 coordinator 를 보도록 수정하였고 Coordinator를 설정하였다

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch-cluster
  namespace: opensearch
spec:
  security:
    config:
      adminSecret:
        name: opensearch-cert-admin
    tls:
       http:
         generate: false
         secret:
           name: opensearch-cert-node
       transport:
         generate: false
         perNode: false
         secret:
           name: opensearch-cert-node
         nodesDn: ["CN=opensearch-cluster-*"]
         adminDn: ["CN=opensearch-admin"]
  general:
    httpPort: 9200
    serviceName: my-first-cluster
    version: 2.19.0
    pluginsList: ["repository-s3"]
    drainDataNodes: true
  dashboards:
    addtionalConfig: ## hosts 를 coordinator 로 변경
      opensearch.hosts: '["http://opensearch-cluster-coordinator.logging.svc.cluster.local:9200"]'
    tls:
      enable: true
      generate: false
      secret:
        name: opensearch-cert-node
    version: 2.19.0
    enable: true
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: masters
      replicas: 3
      resources:
         requests:
            memory: "4Gi"
            cpu: "1000m"
         limits:
            memory: "4Gi"
            cpu: "1000m"
      roles:
        - "data"
        - "cluster_manager"
      persistence:
         emptyDir: {}
    - component: coordinator # coordinator 추가
      replicas:1
      roles:
      - ""
      additionalConfig:
        node.master: "false"
        node.data: "false"
        node.ingest: "false"
        node.remote_cluster_client: "false"
      resources:
         requests:
            memory: "2Gi"
            cpu: "1000m"
         limits:
            memory: "2Gi"
            cpu: "1000m"
      persistence:
         emptyDir: {}

참고로 Opensearch에서 제공하는 ingest기능을 사용하기 위해서는 ingest node를 생성해야한다.