ZenML 0.63.0 brings with it another feature that makes connecting your existing AWS, GCP, and Azure cloud resources into an MLOps stack super simple. We’re calling it the easy stack registration wizard. This wizard allows you can utilize existing cloud infrastructure and register it back to ZenML.
Here is Baris Can Durak with a 2-minute video showcasing the new feature in action:
☁️ Recalling the challenges of registering MLOps cloud stacks
In order to run a machine learning pipeline on one of the major cloud providers, users have to register a ZenML stack. The stack consists of, at minimum:
- an
orchestrator, the configuration of the service that executes the pipeline (e.g. GCP Vertex AI Pipelines/Kubeflow Pipelines) - an
artifact store, the place where the output data of the pipeline is stored (e.g. a S3 bucket) - a
container registry, the place where the Docker images that encapsulate the environment of the pipeline are stored (e.g. Azure Container Storage)
As we recently discussed, connecting your cloud provider to ZenML can be a hassle. You have to individually register all the stack components, and then connect them in a stack. You have to figure out permissions and accounts, which can be a bit cumbersome.
In principle, registering the stack is a few lines of CLI code with ZenML (See this step-by-step guide in the ZenML docs). However, we heard again and again from our community that registering stacks and components was one of the most difficult parts of the journey in the ZenML experience.
- Registration is a multi-faceted process, with the service connector registration followed by stack components, and finally the stacks themselves.
- Along the way, things tend to fail, largely with local CLI setups and cloud permissions.
- It can be overwhelming to dive into cloud concepts, especially for teams who do not have the necessary resources/time in-house.
A few weeks ago, we made deploying stack infrastructure a lot easier with the 1-click deployment experience in version 0.61.0 onwards. However, the feature was for users who wanted to deploy infrastructure from scratch. What if you already have the existing infrastructure (e.g. A Google Cloud Storage Bucket, or an existing GCP artifact registry)? Enter the stack registration wizard.
🧙♂️ Easy Stack Registration Wizard
This feature is for users who have existing infrastructure deployed, and who simply need an easier way to connect that infrastructure to ZenML.
Here is how it works:
This flow makes it super trivial to just point to the right services and defaults to sane configuration options. You can simply select what sort of orchestration service you want to use, what bucket to utilize for the artifacts and the container registry you’d like to use. All the rest is created automatically by ZenML.
How to use it
📢 Note that this feature is only available in a deployed version of ZenML (not in a local scenario via zenml up). Learn how to deploy ZenML here.
To register a stack with the wizard, you can deploy and connect to ZenML, and then use:
This will launch a CLI flow that will guide you through registering a stack.
If you’d rather use the dashboard, you can do so by clicking on the Stacks section, and then clicking on “Create Stack”:
🤸What’s next?
We think this feature will make life a lot easier for new and existing users of ZenML. All three cloud providers are supported. If you’d like even more stack options, just let us know on Slack so we can prioritize.
🦫 Try it now
Try these features today! In order to get access, simply update to the 0.63.0 version:
The full docs are available here. As always, feedback is appreciated!
❓FAQ
Why should I use ZenML with AWS/Azure/GCP Cloud MLOps cloud platforms?
An MLOps pipeline is an end-to-end workflow that orchestrates the steps involved in building, deploying, and monitoring machine learning models. ZenML provides an ML platform that makes it easy to define MLOps pipelines locally and deploy them to various cloud services like AWS Sagemaker Pipelines, Azure ML pipelines, and GCP Vertex AI Pipelines with just a few clicks, as described above. Using ZenML and the Cloud services together, you can significantly accelerate ML development.
How can ZenML's stack wizard accelerate ML platform setup?
Setting up the right infrastructure for MLOps development can be tricky for those with limited infrastructure resources internally. ZenML’s stack wizard is one of the easiest ways to register a simple MLOps stack without having to worry about the underlying infrastructure. This allows machine learning teams to quickly set up an MLOps pipeline and iterate fast.
Which cloud providers are supported by the stack wizard?
ZenML provides a cloud-agnostic MLOps platform that allows you to easily deploy and run ML pipelines on AWS, Azure, and GCP. With ZenML, you can define your pipeline once and then deploy it to your cloud provider of choice with minimal configuration. You can also deploy in a completely cloud-native way using the Kubernetes Orchestrator.
