[k8s] 프로메테우스, 그라파나 설치
Posted by Albert 103Day 2Hour 30Min 51Sec ago [2025-04-25]
1. 프로메테우스 와 그라파나 개념
프로메테우스 는 쿠버네티스 환경의 모니터링 표준에 해당하는 Application
(웹서버 요청횟수와 같이 숫자로 나타낼수 있는 성능지표를 통하여 서버 상태를 알수있다. 그라파나는 프로메테우스에서 제공하는 data를 쉽게 그래프로 확인할수있는 응용프로그램이다.)
2. 프로메테우스 와 그라파나 설치
(프로메테우스 스택을 설치하면 프로메테우스 뿐만아니라 모니터링에 필요한 여러 프로그램도 함께 설치된다.)
albert@k8s-master:~/work/app/prometheus$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
albert@k8s-master:~/work/app/prometheus$ helm repo list
NAME URL
bitnami https://charts.bitnami.com/bitnami
metalib https://metallb.github.io/metallb
argo https://argoproj.github.io/argo-helm
metrics-server https://kubernetes-sigs.github.io/metrics-server/
prometheus-community https://prometheus-community.github.io/helm-charts
albert@k8s-master:~/work/app/prometheus$ helm search repo prometheus
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/kube-prometheus 11.1.6 0.82.0 Prometheus Operator provides easy monitoring de...
bitnami/prometheus 1.4.10 2.55.1 Prometheus is an open source monitoring and ale...
bitnami/wavefront-prometheus-storage-adapter 2.3.3 1.0.7 DEPRECATED Wavefront Storage Adapter is a Prome...
prometheus-community/kube-prometheus-stack 70.8.0 v0.81.0 kube-prometheus-stack collects Kubernetes manif...
albert@k8s-master:~/work/app/prometheus$ helm pull prometheus-community/kube-prometheus-stack
albert@k8s-master:~/work/app/prometheus$ tar xvfz kube-prometheus-stack-70.8.0.tgz
albert@k8s-master:~/work/app/prometheus$ ls
kube-prometheus-stack kube-prometheus-stack-70.8.0.tgz
albert@k8s-master:~/work/app/prometheus$ cd kube-prometheus-stack/
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ cp values.yaml my-values.yaml
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ ls
Chart.lock charts Chart.yaml my-values.yaml README.md templates values.yaml
설치전 my-values.yaml파일을 편집하여 관련 설정값 변경한다.
1) prometheus 접속타입을 ClusterIP -> NodePort 변경
2705 '' Service type
2706 '' NodePort, ClusterIP, LoadBalancer
2707 '' ClusterIP -> NodePort
2708 type: NodePort
2) serviceMonitorSelectorNilUsesHelmValues 항목을 ture-> false 로 변경
4055 '' If true, a nil or {} value for prometheus.prometheusSpec.serviceMonitorSelector will cause the
4056 '' prometheus resource to be created with selectors based on values in the helm deployment,
4057 '' which will also match the servicemonitors created
4058 ''
4059 serviceMonitorSelectorNilUsesHelmValues: false
3) 데이터 유지가간 10일 그대로 놔두고 최대 매트릭 용량은 1기가로 설정
4148 '' How long to retain metrics
4149 ''
4150 retention: 10d
4151
4152 '' Maximum size of metrics
4153 ''
4154 retentionSize: "1GiB"
4) pvc사용시 해당설정도 주석해제하여 하면됨
4255 storageSpec: {}
4256 '' Using PersistentVolumeClaim
4257 ''
4258 ' volumeClaimTemplate:
4259 ' spec:
4260 ' storageClassName: gluster
4261 ' accessModes: ["ReadWriteOnce"]
4262 ' resources:
4263 ' requests:
4264 ' storage: 50Gi
4265 ' selector: {}
5) 그라파나 설정
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ cd charts/grafana/
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack/charts/grafana$ vi values.yaml
기본설정
226 '' Expose the grafana service to be accessed from outside the cluster (LoadBalancer service).
227 '' or access it from within the cluster (ClusterIP service). Set the service type and the port to serve it.
228 '' ref: http://kubernetes.io/docs/user-guide/services/
229 ''
230 service:
231 enabled: true
232 type: NodePort
233 ' Set the ip family policy to configure dual-stack see [Configure dual-stack](https://kubernetes.io/docs/concepts/services-networking/dual-stack/'services)
234 ipFamilyPolicy: ""
235 ' Sets the families that should be supported and the order in which they should be applied to ClusterIP as well. Can be IPv4 and/or IPv6.
236 ipFamilies: []
237 loadBalancerIP: ""
238 loadBalancerClass: ""
239 loadBalancerSourceRanges: []
240 port: 80
241 targetPort: 3000
pvc 설정
412 '' Enable persistence using Persistent Volume Claims
413 '' ref: https://kubernetes.io/docs/user-guide/persistent-volumes/
414 ''
415 persistence:
416 type: pvc
417 enabled: false
418 ' storageClassName: default
419 accessModes:
420 - ReadWriteOnce
421 size: 10Gi
422 ' annotations: {}
423 finalizers:
424 - kubernetes.io/pvc-protection
425 ' selectorLabels: {}
426 '' Sub-directory of the PV to mount. Can be templated.
427 ' subPath: ""
428 '' Name of an existing PVC. Can be templated.
429 ' existingClaim:
430 '' Extra labels to apply to a PVC.
431 extraPvcLabels: {}
432 disableWarning: false
prometheus-stack 설치
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack/charts/grafana$ cd ../../abt-monitoring namespace 생성
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl create namespace abt-monitoring
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl get namespace
NAME STATUS AGE
abt-monitoring Active 88m
설치
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ helm install --namespace abt-monitoring --generate-name prometheus-community/kube-prometheus-stack -f my-values.yaml
NAME: kube-prometheus-stack-1745546644
LAST DEPLOYED: Fri Apr 25 02:04:06 2025
NAMESPACE: abt-monitoring
STATUS: deployed
REVISION: 1
NOTES:
...
3. 설치 내용확인
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl get all --namespace abt-monitoring
NAME READY STATUS RESTARTS AGE
pod/alertmanager-kube-prometheus-stack-1745-alertmanager-0 2/2 Running 0 2m21s
pod/kube-prometheus-stack-1745-operator-66b7f6bb89-xw796 1/1 Running 0 2m35s
pod/kube-prometheus-stack-1745546644-grafana-76cff5d579-ngg78 3/3 Running 0 2m35s
pod/kube-prometheus-stack-1745546644-kube-state-metrics-66d6c9ndqxj 1/1 Running 0 2m35s
pod/kube-prometheus-stack-1745546644-prometheus-node-exporter-5nnst 1/1 Running 0 2m35s
pod/kube-prometheus-stack-1745546644-prometheus-node-exporter-jvrlv 1/1 Running 0 2m35s
pod/kube-prometheus-stack-1745546644-prometheus-node-exporter-szg7t 1/1 Running 0 2m35s
pod/prometheus-kube-prometheus-stack-1745-prometheus-0 2/2 Running 0 2m21s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 2m21s
service/kube-prometheus-stack-1745-alertmanager ClusterIP 10.97.30.105 <none> 9093/TCP,8080/TCP 2m35s
service/kube-prometheus-stack-1745-operator ClusterIP 10.108.75.9 <none> 443/TCP 2m35s
service/kube-prometheus-stack-1745-prometheus NodePort 10.98.4.182 <none> 9090:32071/TCP,8080:31213/TCP2m35s
service/kube-prometheus-stack-1745546644-grafana ClusterIP 10.103.13.178 <none> 80/TCP 2m35s
service/kube-prometheus-stack-1745546644-kube-state-metrics ClusterIP 10.101.23.141 <none> 8080/TCP 2m35s
service/kube-prometheus-stack-1745546644-prometheus-node-exporter ClusterIP 10.102.146.228 <none> 9100/TCP 2m35s
service/prometheus-operated ClusterIP None <none> 9090/TCP 2m21s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-prometheus-stack-1745546644-prometheus-node-exporter 3 3 3 3 3 kubernetes.io/os=linux 2m35s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kube-prometheus-stack-1745-operator 1/1 1 1 2m35s
deployment.apps/kube-prometheus-stack-1745546644-grafana 1/1 1 1 2m35s
deployment.apps/kube-prometheus-stack-1745546644-kube-state-metrics 1/1 1 1 2m35s
NAME DESIRED CURRENT READY AGE
replicaset.apps/kube-prometheus-stack-1745-operator-66b7f6bb89 1 1 1 2m35s
replicaset.apps/kube-prometheus-stack-1745546644-grafana-76cff5d579 1 1 1 2m35s
replicaset.apps/kube-prometheus-stack-1745546644-kube-state-metrics-66d6c96b68 1 1 1 2m35s
NAME READY AGE
statefulset.apps/alertmanager-kube-prometheus-stack-1745-alertmanager 1/1 2m21s
statefulset.apps/prometheus-kube-prometheus-stack-1745-prometheus 1/1 2m21s
1) 매트릭정보를 확인할수있는 node-export 확인
(포트포워딩은 기록사마 등록한거고 실제로는 9100포트로 브라우저상 바로 확인가능하다)
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl get svc --namespace abt-monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 58m
kube-prometheus-stack-1745-alertmanager ClusterIP 10.97.30.105 <none> 9093/TCP,8080/TCP 58m
kube-prometheus-stack-1745-operator ClusterIP 10.108.75.9 <none> 443/TCP 58m
kube-prometheus-stack-1745-prometheus NodePort 10.98.4.182 <none> 9090:32071/TCP,8080:31213/TCP 58m
kube-prometheus-stack-1745546644-grafana ClusterIP 10.103.13.178 <none> 80/TCP 58m
kube-prometheus-stack-1745546644-kube-state-metrics ClusterIP 10.101.23.141 <none> 8080/TCP 58m
kube-prometheus-stack-1745546644-prometheus-node-exporter ClusterIP 10.102.146.228 <none> 9100/TCP 58m
prometheus-operated ClusterIP None <none> 9090/TCP 58m
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl port-forward --address 0.0.0.0 svc/kube-prometheus-stack-1745546644-prometheus-node-exporter 7777:9100 --namespace abt-monitoring
Forwarding from 0.0.0.0:7777 -> 9100
Handling connection for 7777
2) 프로메테우스 화면 확인 (3071포트로 접속하여 확인가능)
3) 그라파나 화면 접속
최초 그라파나는 접속타입이 ClusterIP로 설정되어 외부접속되지않아 접속타입을 LoadBalancer로 변경한다.
변경후 공통으로 접속할수있는 Loadbalancer ip: 192.168.56.22 가 추가된걸 확인할 수있다.
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl patch svc kube-prometheus-stack-1745546644-grafana -n abt-monitoring -p '{"spec": {"type": "LoadBalancer"}}'
service/kube-prometheus-stack-1745546644-grafana patched
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl get svc --namespace abt-monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 6m39s
kube-prometheus-stack-1745-alertmanager ClusterIP 10.97.30.105 <none> 9093/TCP,8080/TCP 6m53s
kube-prometheus-stack-1745-operator ClusterIP 10.108.75.9 <none> 443/TCP 6m53s
kube-prometheus-stack-1745-prometheus NodePort 10.98.4.182 <none> 9090:32071/TCP,8080:31213/TCP 6m53s
kube-prometheus-stack-1745546644-grafana LoadBalancer 10.103.13.178 192.168.56.22 80:30837/TCP 6m53s
kube-prometheus-stack-1745546644-kube-state-metrics ClusterIP 10.101.23.141 <none> 8080/TCP 6m53s
kube-prometheus-stack-1745546644-prometheus-node-exporter ClusterIP 10.102.146.228 <none> 9100/TCP 6m53s
prometheus-operated ClusterIP None <none> 9090/TCP 6m39s
최초 그라파나 설치시 아이디: admin 비번: 아래 명령어로확인
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl get secrets kube-prometheus-stack-1745546644-grafana --namespace abt-monitoring -o yaml
apiVersion: v1
data:
admin-password: cHJvbS1vcGVyYXRvcg==
admin-user: YWRtaW4=
ldap-toml: ""
kind: Secret
...
type: Opaque
albert@k8s-master:~/work/app/prometheus/kube-prometheus-stack$ kubectl get secrets kube-prometheus-stack-1745546644-grafana --namespace abt-monitoring -o jsonpath="{.data.admin-password}" | base64 -d
prom-operator
브라우저상 접속(admin/prom-operator)
끝