How to read from last position when logstream is interrputed

30 Views Asked by At

I have a logstream uri to access the pod logs in K8s, but the stream always interrupt, how to request again from lastposition until the pod finished. sample code:

response_log_streaming = requests.get(
                log_streaming_endpoint,
                headers=headers,
                stream=True)
if not response_log_streaming.ok:
   raise ValidationError(f"Error when streaming the logs, request exited with {response_log_streaming.status_code}")
response_log_streaming_lines = response_log_streaming.iter_lines()
for line in response_log_streaming_lines:
    print(line)

because it always interrupt, so requests.get(log_streaming_endpoint,headers=headers,stream=True) can't really get the full logs. I hope to reconnect it again to get the whole logs

one method is that I can pass headers['Range'] = f'bytes={last_byte + 1}-' to request header, but the url response doesn't return Content-Range, below code snap is not works

def stream_logs(log_streaming_endpoint, headers):
    while True:
        try:
            response = requests.get(log_streaming_endpoint, headers=headers, stream=True)
            response_log_streaming_lines = response.iter_lines(chunk_size=1024)
            for line in response_log_streaming_lines:
                log_line = remove_ansi_characters(line.decode("utf-8"))
                print(log_line)
        except requests.exceptions.RequestException as e:
            print("Error fetching logs:", e)
        
        # Check if the stream was interrupted
        if not response.ok:
            break
        else:
            # If response is successful, clear the 'Range' header
           
            content_range = response.headers.get('Content-Range')
            if content_range:
                last_byte = int(content_range.split('/')[-1])
                # Set the 'Range' header to resume streaming from the last byte received
                headers['Range'] = f'bytes={last_byte + 1}-'
            else:
                # If 'Content-Range' header is not present, retry from the beginning
                try:
                    del headers['Range']
                except KeyError:
                    pass  # 'Range' header does not exist, nothing to delete
                print("Content-Range header not found. Retrying from the beginning.")

        time.sleep(1)


0

There are 0 best solutions below