kubernetes.client.exceptions.ApiException: (0) Reason: Handshake status 500 Internal Server Error

Question

• getting the above exception while deploying dags in the pipeline.

  The log is as follows:
  

    ************************************************************************************************************
    *                                                                                                          *
    *                                                                                                          *
    *                                           Deploying 'dags'...                                            *
    *                                                                                                          *
    *                                                                                                          *
    ************************************************************************************************************
    [2022-12-16 14:09:48,076] {io} INFO - Current directory: /artifacts/dags
    [2022-12-16 14:09:48,076] {copy_deploy_tool} INFO - Deploy 'dags' by copying files...
    [2022-12-16 14:09:48,083] {deploy_tool} INFO -  saving values.yaml...
    [2022-12-16 14:09:48,162] {copy_deploy_tool} INFO - Removing files from 'development:airflow-5db795dd7c-d586h:/root/airflow/dags'
    [2022-12-16 14:09:48,264] {deploy} ERROR - Execution failed for project: dags
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/stream/ws_client.py", line 296, in websocket_call
        client = WSClient(configuration, get_websocket_url(url), headers, capture_all)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/stream/ws_client.py", line 94, in __init__
        self.sock.connect(url, header=header)
      File "/usr/local/lib/python3.6/dist-packages/websocket/_core.py", line 253, in connect
        self.handshake_response = handshake(self.sock, *addrs, **options)
      File "/usr/local/lib/python3.6/dist-packages/websocket/_handshake.py", line 57, in handshake
        status, resp = _get_resp_headers(sock)
      File "/usr/local/lib/python3.6/dist-packages/websocket/_handshake.py", line 143, in _get_resp_headers
        raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
    websocket._exceptions.WebSocketBadStatusException: Handshake status 500 Internal Server Error
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/commands/deploy/deploy.py", line 65, in __try_execute
        deployment=project[DEPLOYMENT],
      File "/usr/local/lib/python3.6/dist-packages/tools/deploy/copy_deploy_tool.py", line 50, in run
        namespace, container, pod_name, command, api_client=api_client
      File "/usr/local/lib/python3.6/dist-packages/helpers/kubernetes.py", line 133, in run_pod_command
        stdout=True,
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/stream/stream.py", line 35, in stream
        return func(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api/core_v1_api.py", line 841, in connect_get_namespaced_pod_exec
        (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)  # noqa: E501
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api/core_v1_api.py", line 941, in connect_get_namespaced_pod_exec_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 345, in call_api
        _preload_content, _request_timeout)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 176, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/stream/stream.py", line 30, in _intercept_request_call
        return ws_client.websocket_call(config, *args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/stream/ws_client.py", line 302, in websocket_call
        raise ApiException(status=0, reason=str(e))
    kubernetes.client.rest.ApiException: (0)
    Reason: Handshake status 500 Internal Server Error
    
    [2022-12-16 14:09:49,631] {shell} INFO - doi: Deployment record uploaded successfully
    [2022-12-16 14:09:49,631] {shell} INFO - OK
    [2022-12-16 14:09:49,635] {io} INFO - Current directory: /artifacts
    [2022-12-16 14:09:49,635] {pretty_info} INFO - ```

Answer 1

Usually, it happens when pod becomes into the Running state, but no running containers on it (0/1). And if you run execute command to this pod and container you will get 500 Internal server error instead of errors related to the real issue(the container is not running).

Check that all containers are running.

if  all([ p.status.phase == "Running" for p in my_pods]) \
    and all([c.state.running for p in my_pods for c in p.status.container_statuses]):

Also refer to this Stackpost and Github issue.