Autoscaling your ML workloads

What it is and how we do it at UbiOps

Autoscaling is an essential MLOps tool for allocating resources automatically to different MLOps workloads. A lot of MLOps tasks have very variable workloads and traffic. If your use case falls into that category, you need autoscaling. Because if you don’t use autoscaling you either:

Having autoscaling features that optimize resources while saving time and money sounds a lot better right? However, there are still two distinct types of autoscaling which you can choose from, and it depends on your particular use case which one fits best.

The two types of autoscaling are scheduled scaling and dynamic scaling. Scheduled scaling is exactly what it sounds like, you have a predefined schedule that dictates when more resources need to be spun up, and when those resources need to be spun down. This is handy when you have variable, but very predictable traffic to your models. An example use case where this would be useful is when you have an ML model that only needs to run on the weekends, and you know exactly how much data is going to pass through the model. In that case, scheduled scaling would suffice.

Dynamic scaling on the other hand, works well when you have very unpredictable traffic to your models. With dynamic scaling, compute resources are scaled up and down dynamically based on the incoming traffic. If there are suddenly a lot of requests made to your model, it will be scaled up, but when it’s not used for a certain amount of time it will be scaled back down again. This optimizes your use of compute resources, which also minimizes your costs related to those resources. At the same time, it also ensures high availability of your models, as there will always be enough resources to handle the incoming traffic. Unless you have a perfectly constant level of traffic to your model 24/7, dynamic scalability is always worth looking into. However, it is typically a bit harder to set up, depending on what type of tools you use.

Which of these factors are important will always differ on a case-by-case basis. No tool will score a perfect ten in all of these categories, so evaluate how you should make the trade-off to best fit your needs.

Related posts:

The Hillary Step was the final obstacle on a climb to the summit of Everest. We’re at a similar moment in U.S. healthcare. In 2015, a revelation shocked the climbing world: an earthquake shook the…

There was a King in ancient India who needed a wise man as his minister. This minister is supposed to be responsible for providing strategic advisory to King in all his matters. Given the level of…

Sometimes it is difficult when you like someone and have been hanging out with him or her for a while to know if he or she is interested in you as well or if you are just a friend. This kind of…