Azure Resource Manager templates – Deploy an AKS cluster with multiple node pool...

This week the multiple node pool feature for Azure Kubernetes Service went GA.

-> https://azure.microsoft.com/en-us/updates/support-for-multiple-node-pools-in-azure-kubernetes-service-is-now-available/

Today we walk through the steps to deploy multiple AKS node pools with Azure Resource Manager templates.

Looking at the ARM template reference for AKS, node pools get defined in the managedClusters resource and additional node pools with an agentPools resource.

-> https://docs.microsoft.com/en-us/azure/templates/microsoft.containerservice/2019-08-01/managedclusters

-> https://docs.microsoft.com/en-us/azure/templates/microsoft.containerservice/2019-08-01/managedclusters/agentpools

We could define multiple node pools statically, but we are using the copy object in the managedClusters resource to define them dynamically and deploying the AKS cluster with multiple node pools.

It is the same approach I already talked about using Terraform.

-> https://www.danielstechblog.io/terraform-deploy-an-azure-kubernetes-service-cluster-with-multiple-node-pools/

...
  "resources": [
    {
      "apiVersion": "[variables('apiVersion').aks]",
      "type": "Microsoft.ContainerService/managedClusters",
      "name": "[parameters('name')]",
      "location": "[resourceGroup().location]",
      "properties": {
        ...
        "copy": [
          {
            "name": "agentPoolProfiles",
            "count": "[length(parameters('agentPoolProfiles'))]",
            "input": {
              "name": "[concat('nodepool',add(copyIndex('agentPoolProfiles'),1))]",
              "maxPods": 250,
              "osDiskSizeGB": 128,
              "count": "[parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].nodeCount]",
              "vmSize": "[parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].nodeVmSize]",
              "osType": "Linux",
              "vnetSubnetID": "[variables('agentPoolProfiles').vnetSubnetId]",
              "enableAutoScaling": "[if(parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].enableAutoScaling, parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].enableAutoScaling, json('null'))]",
              "maxCount": "[if(parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].enableAutoScaling, parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].maxCount, json('null'))]",
              "minCount": "[if(parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].enableAutoScaling, parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].minCount, json('null'))]",
              "type": "VirtualMachineScaleSets",
              "availabilityZones": "[parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].availabilityZones]"
            }
          }
        ],
        ...
  ],
...

The node pool configuration is specified with an array parameter.

...
    "agentPoolProfiles": {
      "value": [
        {
          "nodeCount": 3,
          "nodeVmSize": "Standard_D2_v3",
          "availabilityZones": [
            "1",
            "2",
            "3"
          ],
          "enableAutoScaling": true,
          "maxCount": 6,
          "minCount": 3
        },
        {
          "nodeCount": 2,
          "nodeVmSize": "Standard_D2_v3",
          "availabilityZones": null,
          "enableAutoScaling": false
        }
      ]
    },
...

If we want to add additional node pools after our AKS cluster deployment happened, we cannot do that with the managedClusters resource leveraging the copy object. We would then receive the following error message.

Azure Error: InvalidTemplateDeployment
Message: The template deployment 'aks' is not valid according to the validation procedure. The tracking id is '7302060e-9acf-430f-944b-5f0b4c9fb0d3'. See inner errors for details.
Exception Details:
  Error Code: BadRequest
  Message: Provisioning of resource(s) for container service azst-aks-demo in resource group azst-aks-demo failed. Message: {
  "code": "BadRequest",
  "message": "A new agent pool was introduced. Adding agent pools to an existing cluster is not allowed through managed cluster operations. For agent pool specific change, please use per agent pool operations: https://aka.ms/agent-pool-rest-api",
  "target": "agentPoolProfiles"
 }. Details:
...

We need a second ARM template with the agentPools resource.

...
  "resources": [
    {
      "apiVersion": "[variables('apiVersion').aks]",
      "type": "Microsoft.ContainerService/managedClusters/agentPools",
      "name": "[concat(parameters('name'),'/nodepool',add(copyIndex('agentPoolProfiles'),1))]",
      "location": "[resourceGroup().location]",
      "copy": {
        "name": "agentPoolProfiles",
        "count": "[length(parameters('agentPoolProfiles'))]"
      },
      "properties": {
        "maxPods": 250,
        "osDiskSizeGB": 128,
        "count": "[parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].nodeCount]",
        "vmSize": "[parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].nodeVmSize]",
        "osType": "Linux",
        "vnetSubnetID": "[variables('agentPoolProfiles').vnetSubnetId]",
        "enableAutoScaling": "[if(parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].enableAutoScaling, parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].enableAutoScaling, json('null'))]",
        "maxCount": "[if(parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].enableAutoScaling, parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].maxCount, json('null'))]",
        "minCount": "[if(parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].enableAutoScaling, parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].minCount, json('null'))]",
        "type": "VirtualMachineScaleSets",
        "availabilityZones": "[parameters('agentPoolProfiles')[copyIndex('agentPoolProfiles')].availabilityZones]"
      }
    }
  ],
...

Even in this template we are using the copy object. That requires the specification of the configuration of the already existing node pools and the new ones.

...
    "agentPoolProfiles": {
      "value": [
        {
          "nodeCount": 3,
          "nodeVmSize": "Standard_D2_v3",
          "availabilityZones": [
            "1",
            "2",
            "3"
          ],
          "enableAutoScaling": true,
          "maxCount": 6,
          "minCount": 3
        },
        {
          "nodeCount": 2,
          "nodeVmSize": "Standard_D2_v3",
          "availabilityZones": null,
          "enableAutoScaling": false
        },
        {
          "nodeCount": 1,
          "nodeVmSize": "Standard_D2_v3",
          "availabilityZones": [
            "1",
            "2",
            "3"
          ],
          "enableAutoScaling": true,
          "maxCount": 3,
          "minCount": 1
        }
      ]
    },
...

Finally, we can deploy the AKS cluster with two node pools in Azure.

az group deployment create -g azst-aks-demo --template-file aks.json --parameters aks.parameters.json --verbose

As seen in the screenshot we got two node pools.

Now we add a third node pool with the second Azure Resource Manager template to the AKS cluster.

az group deployment create -g azst-aks-demo --template-file aks-nodepool.json --parameters aks-nodepool.parameters.json --verbose

You can find my ARM templates in my GitHub repository.

-> https://github.com/neumanndaniel/armtemplates/tree/master/container