[k8s] ingress + nginx ingress controller + metalLB를 통한 베어메탈 LoadBalacer 구성
Posted by Albert 107Day 14Hour 41Min 36Sec ago [2025-04-21]
1. 인그레스 개념
인그레스란 구버네티스 클러스터 외부에서 내부에 존재하는 구버네티스 서비스에 접근하기 위해 HTTP/HTTPS를 활용한 라우팅 규칙을 제공하는 오브젝트이다.
인그레스를 활용하면 클러스터 내부에 존재하는 여러 서비스를 다수의 LoadBalancer없이도 외부에 노출시킬수 있으므로 프로덕션 환경에서 유용하게 사용할 수있다.
2. nginx ingress controller 설치
nginx ingress controller 은 쿠버네티스 환경에서 트래픽을 관리할 수 있도록 도와주는 애플리케이션이다.
albert@k8s-master:~/work/app/nginx-ingress-controller$ helm pull bitnami/nginx-ingress-controller
albert@k8s-master:~/work/app/nginx-ingress-controller$ ls
nginx-ingress-controller-11.6.13.tgz
albert@k8s-master:~/work/app/nginx-ingress-controller$ tar xvfz nginx-ingress-controller-11.6.13.tgz
albert@k8s-master:~/work/app/nginx-ingress-controller$ mv nginx-ingress-controller nginx-ingress-controller-11.6.13
albert@k8s-master:~/work/app/nginx-ingress-controller$ cd nginx-ingress-controller-11.6.13
albert@k8s-master:~/work/app/nginx-ingress-controller/nginx-ingress-controller-11.6.13$ cp values.yaml my-values.yaml
mynginx namespace 생성
albert@k8s-master:~/work/app/nginx-ingress-controller/nginx-ingress-controller-11.6.13$ kubectl create namespace mynginx
namespace/mynginx created
nginx ingress controller 설치
albert@k8s-master:~/work/app/nginx-ingress-controller/nginx-ingress-controller-11.6.13$ helm install --namespace mynginx --generate-name bitnami/nginx-ingress-controller -f my-values.yaml
albert@k8s-master:~/work/app/nginx-ingress-controller/nginx-ingress-controller-11.6.13$ helm ls --namespace mynginx
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
nginx-ingress-controller-1745216421 mynginx 1 2025-04-21 06:20:25.241311321 +0000 UTC deployed nginx-ingress-controller-11.6.13 1.12.1
하지만 최초 설치시 EXTERNAL-IP가 pending 상태로 확인됨(아직 ip 할당하지않아서 나타나는 현상임)
albert@k8s-master:~/work/app/nginx-ingress-controller/nginx-ingress-controller-11.6.13$ kubectl get all --namespace mynginx
NAME READY STATUS RESTARTS AGE
pod/nginx-ingress-controller-1745216421-7584794c74-jghzc 1/1 Running 0 113s
pod/nginx-ingress-controller-1745216421-default-backend-8cb5f59ljk5 1/1 Running 0 113s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx-ingress-controller-1745216421 LoadBalancer 10.100.32.34 <pending> 80:30236/TCP,443:31844/TCP 113s
service/nginx-ingress-controller-1745216421-default-backend ClusterIP 10.105.141.213 <none> 80/TCP
3. metalLB를 통한 베어메탈 LoadBalancer구성
kube-proxy의 strictARP확인
albert@k8s-master:~/work/app/nginx-ingress-controller/nginx-ingress-controller-11.6.13$ kubectl get configmap kube-proxy -n kube-system -o yaml | grep strictARP
strictARP: false
쿠버네티스 v1.14.2버전이후부터는 strictARP사용하여야하므로 false -> true로 변경필요
albert@k8s-master:~/work/app/nginx-ingress-controller/nginx-ingress-controller-11.6.13$ kubectl get configmap kube-proxy -n kube-system -o yaml | \sed -e "s/strictARP: false/strictARP: true/" | kubectl apply -f - -n kube-system
configmap/kube-proxy configured
albert@k8s-master:~/work/app/nginx-ingress-controller/nginx-ingress-controller-11.6.13$ kubectl get configmap kube-proxy -n kube-system -o yaml | grep strictARP
strictARP: true
metallb repo 추가
albert@k8s-master:~/work/app/metalib$ helm repo add metallb https://metallb.github.io/metallb
"metallb" has been added to your repositories
albert@k8s-master:~/work/app/metallb$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "metallb" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
albert@k8s-master:~/work/app/metallb$ helm repo list
NAME URL
bitnami https://charts.bitnami.com/bitnami
metallb https://metallb.github.io/metallb
albert@k8s-master:~/work/app/metallb$ helm search repo metallb
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/metallb 6.4.9 0.14.9 MetalLB is a load-balancer implementation for b...
metallb/metallb 0.14.9 v0.14.9 A network load-balancer implementation for Kube...
mymetallb namespace 추가
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ kubectl create namespace mymetallb
namespace/mymetallb created
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ kubectl get namespace
NAME STATUS AGE
calico-apiserver Active 4d1h
calico-system Active 4d1h
default Active 4d2h
kube-node-lease Active 4d2h
kube-public Active 4d2h
kube-system Active 4d2h
mymetallb Active 5s
mynginx Active 27m
tigera-operator Active 4d1h
metallb pull 및 설치
albert@k8s-master:~/work/app/metallb$ helm pull metallb/metallb
albert@k8s-master:~/work/app/metallb$ ls
metallb-0.14.9.tgz
albert@k8s-master:~/work/app/metallb$ tar xvfz metallb-0.14.9.tgz
albert@k8s-master:~/work/app/metallb$ mv metallb metallb-0.14.9
albert@k8s-master:~/work/app/metallb$ cd metallb-0.14.9/
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ ls
Chart.lock charts Chart.yaml policy README.md templates values.schema.json values.yaml
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ cp values.yaml my-values.yaml
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ ls
Chart.lock charts Chart.yaml my-values.yaml policy README.md templates values.schema.json values.yaml
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ helm install --namespace mymetallb --generate-name metalib/metallb -f my-values.yaml
NAME: metallb-1745217547
LAST DEPLOYED: Mon Apr 21 06:39:08 2025...Now you can configure it via its CRs. Please refer to the metallb official docs on how to use the CRs.
metallb 외부에서 접근가능한 IP 설정하는 config 파일 생성
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ vi my-config.yaml
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: my-metallb-config
namespace: mymetallb
spec:
addresses:
- 192.168.56.20-192.168.56.40
autoAssign: true
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: my-metallb-config
namespace: mymetallb
spec:
ipAddressPools:
- my-metallb-config
config 내용 적용
(적용후 webhook ip가 192.168.56.20으로 세팅된부분확인가능하다. 이후 인그레스로 접속시 192.168.56.20/xx로 접속하면 된다.)
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ kubectl apply -f my-config.yaml
ipaddresspool.metallb.io/my-metallb-config created
l2advertisement.metallb.io/my-metallb-config created
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ kubectl get validatingwebhookconfigurations
NAME WEBHOOKS AGE
metallb-webhook-configuration 6 10m
albert@k8s-master:~/work/app/metallb/metallb-0.14.9$ kubectl describe service/nginx-ingress-controller-1745216421 --namespace mynginx
Name: nginx-ingress-controller-1745216421
Namespace: mynginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=nginx-ingress-controller-1745216421
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=nginx-ingress-controller
app.kubernetes.io/version=1.12.1
helm.sh/chart=nginx-ingress-controller-11.6.13
Annotations: meta.helm.sh/release-name: nginx-ingress-controller-1745216421
meta.helm.sh/release-namespace: mynginx
metallb.io/ip-allocated-from-pool: my-metallb-config
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress-controller-1745216421,app.kubernetes.io/name=nginx-ingress-controller
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.100.32.34
IPs: 10.100.32.34
LoadBalancer Ingress: 192.168.56.20
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30236/TCP
Endpoints: 192.168.182.231:8080
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31844/TCP
Endpoints: 192.168.182.231:8443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 6m7s metallb-controller Assigned IP ["192.168.56.20"]
Normal nodeAssigned 6m7s metallb-speaker announcing from node "k8s-work1" with protocol "layer2"
4. 서비스 배포
Deploy: ingress01-deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-deploy-test01
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: web-deploy01
template:
metadata:
labels:
app.kubernetes.io/name: web-deploy01
spec:
containers:
- name: nginx
image: nginx:1.25
Service: ingress01-service.yml
apiVersion: v1
kind: Service
metadata:
name: ingress-service-test01
spec:
selector:
app.kubernetes.io/name: web-deploy01
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 80
Ingress: ingress01-ingress.yml
(http://192.168.56.20/test01로 접속시 ingress-service-test01서비스로 접속하도록 설정)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-test01
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /test01
pathType: Prefix
backend:
service:
name: ingress-service-test01
port:
number: 80
멀티서비스 일때 참고(/test01 로 접속시 ingress-service-test01 서비스 실행, /test02 로 접속시 ingress-service-test02 서비스 실행)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-test02
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /test01
pathType: Prefix
backend:
service:
name: ingress-service-test01
port:
number: 80
- path: /test02
pathType: Prefix
backend:
service:
name: ingress-service-test02
port:
number: 80
배포 실행
albert@k8s-master:~/work/ch09/ex13$ kubectl apply -f ingress01-deploy.yml
deployment.apps/ingress-deploy-test01 created
albert@k8s-master:~/work/ch09/ex13$ kubectl apply -f ingress01-service.yml
service/ingress-service-test01 created
albert@k8s-master:~/work/ch09/ex13$ kubectl apply -f ingress01-ingress.yml
ingress.networking.k8s.io/ingress-test01 created
albert@k8s-master:~/work/ch09/ex13$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-deploy-test01-68d47df476-d89dv 1/1 Running 0 15s 192.168.84.171 k8s-work2<none>
ingress-deploy-test01-68d47df476-jfc6l 1/1 Running 0 15s 192.168.84.172 k8s-work2<none>
ingress-deploy-test01-68d47df476-m9ctr 1/1 Running 0 15s 192.168.182.236 k8s-work1<none>
5. 웹상에서 확인(http://192.168.56.20/test01)
끝 ^^