We want to build a simplified job/task processing system based on Kubernetes. We thought about using Knative and its eventing features. However, one requirement is to execute each task/job isolated in a separate pod. Afterwards, we wanna destroy the pod. Every other task/job is processed by new pods, etc. Further, the jobs/tasks can be long-running, i.e., multiple hours or even days.
We are wondering if we can use and configure Knative to achieve this. I'm actually a bit sceptical due to the scale to zero feature, which would destroy long-running jobs (learned from here: https://stackoverflow.com/a/65881346/7065173). Further, our jobs/tasks shouldn't necessarily listen to an HTTP(S) port. These jobs/tasks are basically pre-packaged into a container and the respective action is executed using Docker CMD.
What do you guys think, is Knative a good baseline for our endeavour? ... Even more, do you have any tip/suggestion what baseline to use instead (we also have an eye on Tekton btw.)?
If you have tasks you want to run for days, then Knative is probably not a good baseline for the effort. Knative assumes that your application is only active as long as there is at least one HTTP request in flight to your application. As you intuit, leaving an HTTP connection open for days is probably not a good design practice.
For your use case, it seems like Kubernetes Jobs might be the best approach. If you need something to react to the "there is work to spin up a job" signal and create the Job, you could use a Knative Service to talk to Kubernetes to create the Job; I've seen that work successfully in other cases.
Knative also doesn't provide a hard mechanism for "exit after processing one request"; for users that want this level of isolation, I've suggested putting an
exit(1)call in their application after they handle one request, but I agree that it's not an ideal workaround.