We are trying to monitor an argo workflow using the Kubernetes client for C#. The Kubernetes server is hosted on AWS EKS cluster. The method we are using is IKubernetes.CustomObjects.GetNamespacedCustomObjectAsync(). The issue is that the first 15 minutes or so all calls go through just fine. After that we start getting the above stated exception.
The kubeconfig file was created using the aws eks update-kubeconfig so I'm assuming no errors are present there (also the calls going through initially).
Sample of code used:
public static async System.Threading.Tasks.Task MonitorWorkflowAsync(this IKubernetes kubernetes, string? workflowName)
{
string? state = null;
do
{
try
{
dynamic response = await kubernetes.CustomObjects.GetNamespacedCustomObjectAsync(KubeConstants.Group, KubeConstants.Version, KubeConstants.NamespaceParameter, KubeConstants.Plural, workflowName);
state = response?.GetProperty("status")?.GetProperty("phase").ToString();
}
catch (Exception ex)
{
Log.Error(ex.Message + "/n" + ex.StackTrace);
}
Log.Information("Pipeline: {0}\t Status: {1} ", workflowName,
state);
if (state?.Equals(KubeConstants.Running,
StringComparison.InvariantCultureIgnoreCase) ?? false)
Thread.Sleep(30000);
if (!(state?.Equals(KubeConstants.Failure,
StringComparison.InvariantCultureIgnoreCase) ?? true)) continue;
Log.Warning(
"The workflow {0} has failed to execute properly. Please access the Kubernetes cluster and troubleshoot OR check your yaml file used to create the workflow!",
workflowName);
break;
}
while (state?.Equals(KubeConstants.Running, StringComparison.InvariantCultureIgnoreCase) ?? false);
}
}
Kubernetes Client version : 8.0.6
Kubernetes Version: 1.22
.NET version: 6.0