Develop/ElasticSearch
ElasticSearch Custom Opentelemetry Collector
팡연
2026. 1. 10. 18:54
반응형
ElasticAPM 및 ElasticAPM Agent 사용하지않고,
Opentelemetry Agent -> Opentelemetry Collector -> ElasticSearch 로 전달하게되면,
Kibana의 Observability 메뉴에서 UI 로 조회가 되지않는다.
(물론 Opentelemetry Collector -> ElasticAPM -> ElasticSearch 또는 EDOT를 사용하면 되긴 함.)
EDOT를 쓰기엔 너무 무겁고 ElasticAPM을 사용하기에도 조금 그렇다면,
일반 Opentelemetry Collector에서 ElasticAPM 을 Custom 해야한다.
1. OCB 설치
https://opentelemetry.io/docs/collector/extend/ocb/
공식가이드에 따라 설치하면 되나, ElasticSearch Custom Collector 버전 때문인지, 조금 낮은버전 설치한다.
현재 가이드 기준 0.143.0 이지만 0.138.0 설치했다.
curl --proto '=https' --tlsv1.2 -fL -o ocb https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.138.0/ocb_0.138.0_linux_arm64
실행파일로 변경
chmod +x ocb
2. Go언어 버전확인 및 builder-config 설정
ElasticSearch Custom Collector 버전이 현재 날짜 기준 1.25.5 에서는 동작하지않아 1.24.11 버전을 설치하였다.
https://www.elastic.co/docs/reference/edot-collector/custom-collector
위 가이드를 통해 builder-config를 설정하면되는데, 가이드가 최신인듯하면서 구식이라 설정을 아래와 같이 변경해주어야한다.
dist.name과 output_path의 경우 내가 사용할 이름으로 설정하였다.
참고로 가이드에는 basicauthextension이 누락되어있으나, 필요하여 따로 추가해주었다.
basicatuhextension이 없으면 collector에서 아래 옵션 사용해서 사용하면됨.
headers:
authorization: Basic base64
dist:
name: otelcol-edot
description: Elastic Distribution of OpenTelemetry Collectors
output_path: ./otelcol-edot
receivers:
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/apachereceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver v0.139.0
- gomod:
github.com/elastic/opentelemetry-collector-components/receiver/elasticapmintakereceiver v0.21.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/httpcheckreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/iisreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jmxreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8seventsreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sobjectsreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/mysqlreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/nginxreceiver v0.139.0
- gomod:
go.opentelemetry.io/collector/receiver/nopreceiver v0.139.0
- gomod:
go.opentelemetry.io/collector/receiver/otlpreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/postgresqlreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/redisreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sqlserverreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowseventlogreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/windowsperfcountersreceiver v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.139.0
processors:
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.139.0
- gomod:
go.opentelemetry.io/collector/processor/batchprocessor v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor v0.139.0
- gomod:
github.com/elastic/opentelemetry-collector-components/processor/elasticapmprocessor v0.21.0
- gomod:
github.com/elastic/opentelemetry-collector-components/processor/elasticinframetricsprocessor v0.20.0
- gomod:
github.com/elastic/opentelemetry-collector-components/processor/elastictraceprocessor v0.20.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/geoipprocessor v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor v0.139.0
- gomod:
go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.139.0
exporters:
- gomod:
go.opentelemetry.io/collector/exporter/debugexporter v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/kafkaexporter v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter v0.139.0
- gomod:
go.opentelemetry.io/collector/exporter/nopexporter v0.139.0
- gomod:
go.opentelemetry.io/collector/exporter/otlpexporter v0.139.0
- gomod:
go.opentelemetry.io/collector/exporter/otlphttpexporter v0.139.0
connectors:
- gomod:
github.com/elastic/opentelemetry-collector-components/connector/elasticapmconnector v0.20.0
- gomod:
go.opentelemetry.io/collector/connector/forwardconnector v0.139.0
- gomod:
github.com/elastic/opentelemetry-collector-components/connector/profilingmetricsconnector v0.20.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/connector/routingconnector v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.139.0
extensions:
- gomod:
github.com/elastic/opentelemetry-collector-components/extension/apikeyauthextension v0.22.0
- gomod:
github.com/elastic/opentelemetry-collector-components/extension/apmconfigextension v0.20.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/headerssetterextension v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckv2extension v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/k8sleaderelector v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/k8sobserver v0.139.0
- gomod:
go.opentelemetry.io/collector/extension/memorylimiterextension v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.139.0
- gomod:
github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.139.0 ## 추가
providers:
- gomod:
go.opentelemetry.io/collector/confmap/provider/envprovider v1.45.0
- gomod:
go.opentelemetry.io/collector/confmap/provider/fileprovider v1.45.0
- gomod:
go.opentelemetry.io/collector/confmap/provider/httpprovider v1.45.0
- gomod:
go.opentelemetry.io/collector/confmap/provider/httpsprovider v1.45.0
- gomod:
go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.45.0
3. 빌드 테스트
아래 명령어를 통해 빌드가 잘되는지 확인한다.
(웬만하면 안될 경우 go 언어 버전 또는 ocb 버전문제이다.)
./ocb --config builder-config.yaml
성공하면 아래와 같이 나온다.
$ ls -al
builder-config.yaml ocb otelcol-edo
4. 이미지 빌드
이제 collector-config.yaml 과 Dockerfile을 생성해준다.
collector-config는 아래와 같이 공식가이드대로 생성해주었다.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
exporters: [debug]
metrics:
receivers: [otlp]
exporters: [debug]
logs:
receivers: [otlp]
exporters: [debug]
Dockerfile도 공식가이드에서 golang 버전과 ocb 버전, ENTRYPOINT를 변경해주었다.
FROM alpine:3.19 AS certs
RUN apk --update add ca-certificates
FROM golang:1.24.11 AS build-stage
WORKDIR /build
COPY ./builder-config.yaml builder-config.yaml
RUN --mount=type=cache,target=/root/.cache/go-build GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@v0.138.0
RUN --mount=type=cache,target=/root/.cache/go-build builder --config builder-config.yaml
FROM gcr.io/distroless/base:latest
ARG USER_UID=10001
USER ${USER_UID}
COPY ./collector-config.yaml /otelcol/collector-config.yaml
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --chmod=755 --from=build-stage /build/otelcol-edot /otelcol
ENTRYPOINT ["/otelcol/otelcol-edot"]
CMD ["--config", "/otelcol/collector-config.yaml"]
EXPOSE 4317 4318 12001
다 만들면 아래와 같이 되어있다.
.
├── builder-config.yaml
├── collector-config.yaml
└── Dockerfile
└── ocb # 삭제해도 됨
└── otelcol-edot # 삭제해도 됨
가이드대로 그대로 빌드한다.
# Enable Docker multi-arch builds
docker run --rm --privileged tonistiigi/binfmt --install all
docker buildx create --name mybuilder --use
# Build the Docker image as Linux AMD and ARM
# and load the result to "docker images"
docker buildx build --load \
-t <collector_distribution_image_name>:<version> \
--platform=linux/amd64,linux/arm64 .
# Test the newly built image
docker run -it --rm -p 4317:4317 -p 4318:4318 \
--name otelcol <collector_distribution_image_name>:<version>
otel-custom:1.0 이라고 빌드하여서 아래와 같이 되어있다.
IMAGE ID DISK USAGE CONTENT SIZE
otel-custom:1.0 c6bd6908f678 318MB 64.1M
5. Opentelemetry Collector 설치
이제 해당 이미지를 이미지 서버에 추가하여 배포한다.
테스트를 위해 operator, collecto 따로 설치하였다.
operator
helm install opentelemetry-operator open-telemetry/opentelemetry-operator --set admissionWebhooks.certManager.enabled=false --set admissionWebhooks.autoGenerateCert.enabled=true -n tracing
collector
helm install opentelemetry-collector open-telemetry/opentelemetry-collector --set image.repository=docker.io/library/otel-custom --set image.tag=1.0 --set image.pullPolicy=IfNotPresent --set mode=deployment -n tracing
collector 설치 후 configmap수정
exporters에 elasticsearch와 connector에 elasticapm을 추가하였고
recevier에도 추가했다
apiVersion: v1
data:
relay: |
exporters:
debug: {}
elasticsearch: ## 추가
endpoint: https://127.0.0.1:9200
headers:
authorization: Basic
tls:
insecure: false
insecure_skip_verify: true
extensions:
health_check:
endpoint: ${env:MY_POD_IP}:13133
processors:
batch: {}
elasticapm: {}
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
connectors: ## 추가
elasticapm: {} ## 추가
receivers:
jaeger:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:14250
thrift_compact:
endpoint: ${env:MY_POD_IP}:6831
thrift_http:
endpoint: ${env:MY_POD_IP}:14268
otlp:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:4317
http:
endpoint: ${env:MY_POD_IP}:4318
prometheus:
config:
scrape_configs:
- job_name: opentelemetry-collector
scrape_interval: 10s
static_configs:
- targets:
- ${env:MY_POD_IP}:8888
zipkin:
endpoint: ${env:MY_POD_IP}:9411
service:
extensions:
- health_check
pipelines:
logs:
exporters:
- debug
- elasticsearch ## 추가
processors:
- memory_limiter
- batch
receivers:
- otlp
metrics:
exporters:
- debug
processors:
- memory_limiter
- batch
receivers:
- otlp
- prometheus
traces:
exporters:
- debug
- elasticsearch ## 추가
processors:
- memory_limiter
- batch
- elasticapm ## 추가
receivers:
- otlp
- jaeger
- zipkin
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: ${env:MY_POD_IP}
port: 8888
6. 또한, 테스트용으로 aws-sample 도 설치하였다
테스트용 aws-sample
kubectl apply -f https://github.com/aws-containers/retail-store-sample-app/releases/latest/download/kubernetes.yaml -n aws
auto instrumentation 추가
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: demo-instrumentation
spec:
exporter:
endpoint: http://opentelemetry-collector.tracing.svc.cluster.local:4318
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "1"
inject-java
## 환경변수에 활성화 설정
OTEL_JAVAAGENT_ENABLED: "true"
## Annotation에 추가
instrumentation.opentelemetry.io/inject-java: "true"
7. aws-sample 에 접속해서 테스트하면 원래 Opentelemetry Collector만 사용하면 안나오던 UI가 나온다.
반응형