Azure Data Factory Integration Templates: ADF-to-Databricks Integration Patterns

ADF-to-Databricks Integration Patterns Datanest Digital — datanest.dev Overview This guide covers production patterns for orchestrating Databricks notebooks from Azure Data Factory. It addresses cluster strategy, parameter passing, error handling, medallion architecture orchestration, and operational considerations. 1. Linked Service Authentication Patterns Managed Identity (Recommended) Managed identity is the most secure approach — no tokens or credentials to manage. { "type" : "AzureDatabricks" , "typeProperties" : { "domain" : "https://adb-<id>.azuredatabricks.net" , "authentication" : "MSI" , "workspaceResourceId" : "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Databricks/workspaces/<ws>" } } Requirements: ADF managed identity must have Contributor role on the Databricks workspace The workspace must have Unity Catalog or legacy ACLs configured for the service principal Access Token via Key Vault For environments where MSI is not supported (e.g., cross-tenant): { "t