Azure Kubernetes Service — Handling the custom Private DNS Zone for private clusters

Azure has been enhancing many PaaS services to use private endpoints. A private endpoint is a way of ensuring all the traffic to a service occurs through your network fabric and not go over the internet.

In the past, most services, when accessed from an on-premise network would route from the companies location over the internet to a publicly accessible URL. While it is possible to restrict access using IP rules, this method raises eyebrows in security conscious organizations.

When using Private Endpoints, Azure typically requires an Azure Private DNS Zone to insert the DNS records for the endpoint and have your DNS infrastructure resolve the record. For most services, a zone specific to the service is required as documented here.

For example, when using a Private Endpoint for an Azure Container Registry (ACR), a Private DNS zone needs to be created by the customer using the zone name: privatelink.azurecr.io. Then this zone needs to be linked to the Virtual Network where your DNS servers are; you can resolve the hostname for your endpoint. Refer to this document for more details about hybrid DNS topology. At a high-level, it looks like this:

Hybrid DNS Infrastructure with forwarders

When working with AKS, the approach is different and more challenging to integrate into your environment. When creating a private cluster, the API masters are turned into a private endpoint and placed on the same subnet as the nodes & pods.

Azure will automatically create a managed Private DNS Zone under the covers in the managed resource group. This Zone name is system generated and non-deterministic.

The non-deterministic nature of the zone name makes it more difficult to integrate it to your DNS infrastructure and link the Virtual Network. The zone name looks like this:

1565c3fb-ddf7–4310-ab3e-b5070dxx1b10.privatelink.canadacentral.azmk8s.io

With some inspiration from Pavel Tuzov, I decided to get creative to ensure the Private DNS Zone is linked to the Virtual Network where my DNS forwarders are during the cluster creation.

Solution

The solution requires dissecting the private_fqdn member of the azurerm_kubernetes_cluster resource. Using locals, it is easy to perform the logic and then use outputs to export the required data elements to support a virtual network link

# Magic to generate the Private DNS zone information that is automatically created by Microsoft.
# Used for creation of a Private DNS zone link
locals {
  private_dns_zone_name = join(".", slice(split(".", azurerm_kubernetes_cluster.aks_cluster.private_fqdn), 1, length(split(".", azurerm_kubernetes_cluster.aks_cluster.private_fqdn))))
  private_dns_zone_id   = "${data.azurerm_subscription.current.id}/resourceGroups/${azurerm_kubernetes_cluster.aks_cluster.node_resource_group}/providers/Microsoft.Network/privateDnsZones/${local.private_dns_zone_name}"
}
resource "azurerm_kubernetes_cluster" "aks_cluster" {
  # Content omitted for brevity
}
# Supports the configuration of private dns zone links.
# May perform the linking as an opinion in this module in the future.
output "managed_rg_name" {
  value       = azurerm_kubernetes_cluster.aks_cluster.node_resource_group
  description = "Resource Group name managed by Microsoft when creating the cluster"
}
output "private_dns_zone_name" {
  value       = local.private_dns_zone_name
  description = "Zone Name for the Private DNS Zone generated by private AKS"
}
output "private_dns_zone_id" {
  value       = local.private_dns_zone_id
  description = "Resource Id for the Private DNS Zone generated by private AKS"
}
# This is to perform the network link from the module
resource "azurerm_private_dns_zone_virtual_network_link" "zone_link_cac_cluster" {
  name                  = "hub_link"
  resource_group_name   = module.cac_aks.managed_rg_name
  private_dns_zone_name = module.cac_aks.private_dns_zone_name
  virtual_network_id    = azurerm_virtual_network.hub.id // The Hub needs to reach the zone
}

I guess one could modify the Terraform module to export this data. In the interim, this is a pretty good workaround.

I hope Microsoft will have AKS follow the same methodology as the other services supporting private endpoints.