I'm looking to scale some compute in a python application.
I'm running this on a multi-core machine, and it scales more or less linearly with the number of processes I spawn.
I'd like to spawn additional processes in a remote machine/machines - using either Azure Functions or Containers as a service.
I came across Ray as a framework to do this, though it requires a cluster of VMs, and for my usecase a simple scaling across serverless compute to do a one-off calculation would be enough.
Any clues how I could achieve this?
I considered deploying a fastAPI app to auto-scale on Azure Container Apps that accepts a python function to execute and executes it, returning the result as a pickle-serialized object.
This kind of works, but it feels sub-optimal (not to mention the obvious security issues with a server meant to execute arbitrary python code).