Effortlessly Orchestrate Your ZenML Pipelines with the Power of AzureML
Seamlessly integrate ZenML with AzureML Pipelines to leverage the robustness and scalability of Microsoft's cloud-based orchestration service. This integration enables you to efficiently build, train, deploy, and manage your machine learning models, streamlining your MLOps workflow.
Features with ZenML
- Seamless execution of ZenML pipelines on AzureML infrastructure
- Easy configuration of compute resources for optimal performance
- Secure access and authentication using Azure credentials or service connectors
- Integrate with other Azure-based stack components for end-to-end MLOps
- Support for scheduled pipeline runs using cron expressions or intervals
Main Features
- A comprehensive environment for the entire ML lifecycle
- Scalable computing resources for training and inference
- Integrated model management and deployment capabilities
- Collaborative workspace for team productivity
How to use ZenML with
AzureML Pipelines
# 1. Install the ZenML `azure` integration
# zenml integration install azure
# 2. Register an AzureML container registry
# zenml orchestrator register <NAME> -f azureml \
# --subscription_id=<YOUR_AZUREML_SUBSCRIPTION_ID> \
# --resource_group=<NAME_OF_YOUR_RESOURCE_GROUP> \
# --workspace=<NAME_OF_YOUR_AZUREML_WORKSPACE> \
# --connector azure_connector
# 3. Update your stack with your new container registry
# zenml stack update -o <NAME>
from zenml import pipeline, step
from zenml.integrations.azure.flavors import AzureMLOrchestratorSettings
azureml_settings = AzureMLOrchestratorSettings(
mode="compute-cluster",
compute_name="my-gpu-cluster", # Will fetch or create this instance
size="Standard_NC6s_v3", # Using a NVIDIA Tesla V100 GPU
tier="Dedicated", # Can be set to either "Dedicated" or "LowPriority"
min_instances=2,
max_instances=10,
idle_time_before_scaledown_down=60,
)
@step
def hello_world() -> str:
return "Hello World!"
@pipeline
def my_pipeline():
_ = hello_world()
if __name__ == "__main__":
my_pipeline()
This code snippet demonstrates how to configure an AzureML orchestrator in ZenML using the AzureMLOrchestratorSettings class. It sets up a compute cluster with GPU instances and defines a simple pipeline with a single step. Running the pipeline triggers execution on the configured AzureML infrastructure.
The code example demonstrates how to set up and use an AzureML Orchestrator with ZenML:
- Install the Azure integration
- Register an orchestrator with the
azureml flavor - Update your active stack with your orchestrator.
- You will need a remote artifact store and a container registry)
- Use the
@step and @pipeline decorators to define a pipeline. Once executed, ZenML will create the corresponding AzureML job.
Additional Resources
Azure Machine Learning Documentation
Read the ZenML AzureML Orchestrator Documentation
Read the guide for setting up a full Azure stack
