Azure Data Factory Deployment Mistake

Recently, I was working with a release pipeline to add our Azure Data Factory solution. This was the first attempt to release Azure Data Factory components. Of course, there were many issues to resolve. The release pipeline handles both databases from multiple Git repositories and now ADF code, so it’s no longer an out of the box solution.

Microsoft provides documentation for AzureResourceManagerTemplateDeployment@3, which is helpful, if you read it thoroughly. Of course, I did not read it thoroughly, so I didn’t see the note about the DeploymentMode options. With the different issues, I decided to set parameter to ‘Complete’, instead of the default value, ‘Incremental’. The task looked a lot like the example below.

   - task: AzureResourceManagerTemplateDeployment@3
      displayName: 'Deploy ADF Solution'
      inputs:
        deploymentScope: 'Resource Group'
        azureResourceManagerConnection: '$(azureSubscription)'
        subscriptionId: '<subscriptionID>'
        action: 'Create Or Update Resource Group'
        location: '<Azure region>'
        resourceGroupName: '<resource group>'
        templateLocation: 'Linked artifact'
        csmFile: '$(System.ArtifactsDirectory)/<SolutionPath>/ARMTemplateForFactory.json'
        csmParametersFile: '$(System.ArtifactsDirectory)/<SolutionPath>/ARMTemplateParametersForFactory.json'
        overrideParameters: -factoryName xxxxxxxxxxxxx
                            -KeyVault_properties_typeProperties_baseUrl $(KeyVault-AutomatedTesting)
                            -ADLS_Storage_properties_typeProperties_url $(ADLSLocation-AutomatedTesting)
        deploymentMode: Complete

Oops! That was a mistake. The deployment failed, with a new error message about the data factory not being found. Excuse me?! I just saw it a couple minutes ago. Going back to Azure Data Factory, the instance I had been working with was no longer listed. What happened in the last couple minutes? The only thing I did was change the DeploymentMode to ‘Complete’.

Referring back to the documentation, I found this little nugget.

Complete mode deletes resources that are not in your template. Complete mode takes relatively more time than incremental mode. If the task times out, consider increasing the timeout or changing to the Incremental mode.

My ARM template only included the the code contained within Azure Data Factory. It did not contain the definition of the data factory itself. As a result, the Complete deployment mode deleted the data factory (along with a storage account) and then tried to deploy the code. Of course this didn’t work.

Fortunately, a proof-of-concept environment was created to develop the release pipeline, so I was the only one using the instances that were deleted. This mistake is a good reminder why resource groups should be resource instance specific and not encompass a large number (or even a modest number) of resources. It’s also a reminder to read the documentation, when new features are being used. ADO Pipeline tasks (and jobs) are concisely defined in YAML, but they perform a lot of work. It is easy to run into unexpected outcomes.