Kubernetes Private Cluster Ingress Deployment, Configuration Guide & Automation Using Azure DevOps Pipeline
Overview
This document describes the process for deploying and configuring the NGINX Ingress Controller within the Kubernetes cluster, and for automating the deployment. The setup is divided into three main stages:
Ingress Controller Setup
Ingress Route Configuration
Using Azure DevOps Pipeline for automation.
Stage 1: Ingress Controller Setup
1. Identify Target Environment
The first step is to identify the target environment or Kubernetes cluster where the Ingress Controller will be deployed.
Examples:
Development Cluster
Staging Cluster
Production Cluster
Ensure:
The correct Kubernetes context is selected.
Appropriate access permissions are available.
The private Azure Container Registry (ACR) is accessible from the cluster.
2. Import NGINX Images into Private ACR
The required NGINX Ingress images are pulled from a publicly available container registry and pushed into the organization’s private Azure Container Registry (ACR).
+----------------------------------+
| Public Container Registry |
| (NGINX Ingress Images) |
+----------------+------------------+
|
| ingress.sh
v
+----------------------------------+
| Azure Container Registry (ACR) |
| (Private Images) |
+----------------+------------------+
|
| Helm (ingress-deployment.sh)
v
+----------------------------------------------------------------------------------+
| Kubernetes Cluster (Private Network) |
| |
| +----------------------+ +--------------------------------------+ |
| | Ingress Controller | | Kubernetes Service (Private IP) | |
| | Pods (NGINX) |-------->| Internal Load Balancer | |
| +----------------------+ +--------------------------------------+ |
| |
+----------------------------------------------------------------------------------+
This is executed using the script:
REGISTRY_NAME=your-registry.azurecr.io
SOURCE_REGISTRY=registry.k8s.io
CONTROLLER_IMAGE=ingress-nginx/controller
CONTROLLER_TAG=v1.8.1
PATCH_IMAGE=ingress-nginx/kube-webhook-certgen
PATCH_TAG=v20230407
DEFAULTBACKEND_IMAGE=defaultbackend-amd64
DEFAULTBACKEND_TAG=1.5
az acr import --name \(REGISTRY_NAME --source \)SOURCE_REGISTRY/\(CONTROLLER_IMAGE:\)CONTROLLER_TAG --image \(CONTROLLER_IMAGE:\)CONTROLLER_TAG
az acr import --name \(REGISTRY_NAME --source \)SOURCE_REGISTRY/\(PATCH_IMAGE:\)PATCH_TAG --image \(PATCH_IMAGE:\)PATCH_TAG
az acr import --name \(REGISTRY_NAME --source \)SOURCE_REGISTRY/\(DEFAULTBACKEND_IMAGE:\)DEFAULTBACKEND_TAG --image \(DEFAULTBACKEND_IMAGE:\)DEFAULTBACKEND_TAG
Purpose
Ensures all ingress images are stored internally.
Avoids direct dependency on public registries.
Improves security and compliance.
⏱ Estimated duration: A few minutes.
3. Deploy the Ingress Controller
The Ingress Controller is deployed using Helm commands defined in:
# Add the ingress-nginx repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
# Set variable for ACR location to use for pulling images
ACR_LOGIN_SERVER=your-registry.azurecr.io
SOURCE_REGISTRY=registry.k8s.io
CONTROLLER_IMAGE=ingress-nginx/controller
CONTROLLER_TAG=v1.8.1
PATCH_IMAGE=ingress-nginx/kube-webhook-certgen
PATCH_TAG=v20230407
DEFAULTBACKEND_IMAGE=defaultbackend-amd64
DEFAULTBACKEND_TAG=1.5
# Use Helm to deploy an NGINX ingress controller with required security policies
helm install ingress-nginx ingress-nginx/ingress-nginx \
--version 4.7.1 \
--namespace ingress-main \
--create-namespace \
--set controller.replicaCount=2 \
--set controller.nodeSelector."kubernetes\.io/os"=linux \
--set controller.image.registry=$ACR_LOGIN_SERVER \
--set controller.image.image=$CONTROLLER_IMAGE \
--set controller.image.tag=$CONTROLLER_TAG \
--set controller.image.digest="" \
--set controller.admissionWebhooks.patch.nodeSelector."kubernetes\.io/os"=linux \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
--set controller.service.externalTrafficPolicy=Local \
--set controller.admissionWebhooks.patch.image.registry=$ACR_LOGIN_SERVER \
--set controller.admissionWebhooks.patch.image.image=$PATCH_IMAGE \
--set controller.admissionWebhooks.patch.image.tag=$PATCH_TAG \
--set controller.admissionWebhooks.patch.image.digest="" \
--set defaultBackend.nodeSelector."kubernetes\.io/os"=linux \
--set defaultBackend.image.registry=$ACR_LOGIN_SERVER \
--set defaultBackend.image.image=$DEFAULTBACKEND_IMAGE \
--set defaultBackend.image.tag=$DEFAULTBACKEND_TAG \
--set defaultBackend.image.digest="" \
`# --- Security Policy Additions Start Here. This is Optional ---` \
--set controller.allowPrivilegeEscalation=false \
--set controller.runAsUser=101 \
--set controller.runAsGroup=101 \
--set controller.containerSecurityContext.allowPrivilegeEscalation=false \
--set controller.containerSecurityContext.readOnlyRootFilesystem=true \
--set controller.containerSecurityContext.runAsNonRoot=true \
--set defaultBackend.runAsUser=101 \
--set defaultBackend.runAsGroup=101 \
--set defaultBackend.containerSecurityContext.allowPrivilegeEscalation=false \
--set defaultBackend.containerSecurityContext.readOnlyRootFilesystem=true \
--set defaultBackend.containerSecurityContext.runAsNonRoot=true \
--set controller.containerSecurityContext.runAsUser=101 \
--set controller.containerSecurityContext.runAsGroup=101 \
--set controller.containerSecurityContext.readOnlyRootFilesystem=false
What This Step Does
- Pulls ingress images from the private ACR.
Deploys:
Ingress Controller pods
Associated Kubernetes Service
Configures the ingress with a private IP address
Important Notes
The ingress is configured for private access only.
It is accessible only within the internal network.
No public exposure is enabled unless explicitly configured.
Stage 2: Ingress Route Configuration
Once the Ingress Controller is deployed, routing rules must be configured.
There are two supported approaches:
Option 1: General (Shared) Ingress Route
In this approach:
A central ingress route file is used.
Services are manually added after deployment.
The configuration is maintained in:
ingress-route-file.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: your-ingress-name
namespace: ingress-main
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
kubernetes.io/ingress.class: "nginx"
# secrets-store.csi.k8s.io/used: "true"
# cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
ingressClassName: nginx
# REQUIRED BY AZURE POLICY: Defines the hostname(s) and the Kubernetes Secret for the TLS certificate
tls:
- hosts:
- "your-company.com"
secretName: your-tls-secretname
rules:
- host: "your-company.com"
http:
paths:
- path: /app1-dev(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: app1-dev
port:
number: 80
- path: /app2-web(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: app2-web
port:
number: 80
- path: /hello-world-service(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: hello-world-service
port:
number: 80
Process
Deploy application/service.
Update
ingress-route-file.yaml.Add routing rules for the new service.
Apply the updated configuration.
Use Case
Suitable for centralized route management.
Recommended for environments where routing is controlled by a platform or DevOps team.
Option 2: Service-Specific Ingress Route
In this approach:
Each service defines its own ingress configuration.
Routing rules are created per deployment.
Configuration can be included directly within the Helm chart of the service.
values.yaml
# -- Ingress
ingress:
enabled: false
className: "nginx"
annotations:
kubernetes.io/`ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/use-regex: "true"
path: ServiceName
hosts:
- host: ingress_host
paths:
- path: /
pathType: Prefix
tls:
enabled: true
hosts:
- ingress_host
- secretName: ingress_secretName
ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "myapp.fullname" . }}
namespace: {{ .Values.namespace.name | default .Release.Namespace }}
labels:
{{- include "myapp.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
pathType: {{ .pathType }}
backend:
service:
name: {{ include "myapp.fullname" $ }}
port:
number: {{ $.Values.service.port }}
{{- end }}
{{- end }}
{{- with .Values.ingress.tls }}
tls:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
Process
Define ingress configuration inside the service helm chartdeployment.
Add the code snippet to the Helm Chart “values.yaml” file.
Create a file “ingress.yaml” and add it to the Helm Chart template folder
Deploy via Helm.
Ingress route is automatically created as part of the deployment.
Use Case
Suitable for microservices architecture.
Enables teams to manage their own routing.
Improves deployment autonomy.
Stage 3: Automating ingress deployment via Azure DevOps Pipeline
The task below can be added to your pipeline to enable the automation of the ingress deployment
ADO Pipeline Task
Variables
$(subscription_id)
$(aks_rg_name)
$(kubernetes_name)
- task: AzureCLI@2
displayName: AKS CLI login
inputs:
azureSubscription: 'Hanger-AKS-DEV'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
echo "login succesful"
echo "setting subscription"
az account set --subscription $(subscription_id)
echo "Getting AKS Credentials"
az aks get-credentials --resource-group \((aks_rg_name) --name \)(kubernetes_name) --overwrite-existing
echo "Installing kubelogin"
curl -LO https://github.com/Azure/kubelogin/releases/latest/download/kubelogin-linux-amd64.zip
unzip kubelogin-linux-amd64.zip
sudo mv bin/linux_amd64/kubelogin /usr/local/bin/
echo "Converting kubeconfig to use non-interactive login"
kubelogin convert-kubeconfig -l azurecli
cd into your folder
cat Ingress-Controller-file.sh
./Ingress-Controller-file.sh
Summary
Internal Client
(VM / App / VPN User)
|
|
v
+----------------------------------+
| Internal Load Balancer (Private IP)
+----------------+------------------+
|
v
+----------------------+
| NGINX Ingress |
| Controller Pods |
+----------+-----------+
|
-----------------------------------------
| | |
v v v
+--------------+ +--------------+ +--------------+
| Service A | | Service B | | Service C |
| (ClusterIP) | | (ClusterIP) | | (ClusterIP) |
+--------------+ +--------------+ +--------------+
| | |
v v v
Pods Pods Pods
The ingress implementation consists of:
Importing NGINX images into a private Azure Container Registry.
Deploying the Ingress Controller using Helm.
Configuring routing either:
Centrally (shared ingress file), or
Per service (Helm-based configuration).
This setup ensures:
Secure internal-only access (private IP).
Controlled image sourcing via private ACR.
Flexible routing configuration strategies.
