Streamline ML Workloads on any cloud with SkyPilot VM Orchestration in ZenML

Simplify the process of running machine learning pipelines on the cloud by integrating SkyPilot VM orchestration with ZenML. This integration empowers you to provision and manage virtual machines (VMs) on AWS, GCP, Azure, or Lambda Labs, offering cost savings, high GPU availability, and managed execution for your ML workloads.

Features with ZenML

Seamless provisioning and scaling of VMs for ZenML pipelines
Automatic selection of cost-optimized VM/zone/region/cloud configurations
Fine-grained resource allocation for each pipeline step
Autostop feature to clean up idle clusters and prevent unnecessary costs
Compatibility with ZenML's Dashboard for pipeline and artifact monitoring

‍

Main Features

Support for AWS, GCP, Azure, and Lambda Labs cloud platforms
Automatic provisioning of spot and on-demand VMs
Built-in cost optimization for selecting VM/zone/region/cloud
Configurable VM types and resources for each workload
Autostop functionality to terminate idle clusters

‍

How to use ZenML with

Skypilot VM


 # Install the necessary requirements locally
 # For AWS
  pip install "zenml[connectors-aws]"
  zenml integration install aws skypilot_aws 

  # for GCP
  pip install "zenml[connectors-gcp]"
  zenml integration install gcp skypilot_gcp # for GCP

  # for Azure
  pip install "zenml[connectors-azure]"
  zenml integration install azure skypilot_azure # for Azure
  
  # for Azure
  pip install "zenml[connectors-azure]"
  zenml integration install skypilot_lambda # for Lambda Labs


# easily configure your runtime
skypilot_settings = {
    cpus="2",
    memory="16",
    accelerators="V100:2",
    ...
}


@pipeline(
    settings={
        "orchestrator.vm_aws": skypilot_settings,
        # "orchestrator.vm_gcp": skypilot_settings,
        # "orchestrator.vm_azure": skypilot_settings,
        # "orchestrator.vm_lambda": skypilot_settings
    }
)

This code snippet demonstrates how to configure the SkyPilot VM orchestrator for a ZenML pipeline. The Skypilot settings allows you to specify the desired resources (CPUs, memory, accelerators) and other parameters like spot instances, region, disk size, and auto-stop behavior. These settings can be applied at the pipeline level or individual step level for fine-grained resource allocation.

Additional Resources

SkyPilot VM Orchestrator Documentation

SkyPilot blog