Python_Gitlab How to extract string from the output of ProjectJob.trace API call

Question

I use python-gitlab API call ProjectJob.trace (https://python-gitlab.readthedocs.io/en/stable/api/gitlab.v4.html#gitlab.v4.objects.ProjectJob.trace) to get the log of a GitlabCI pipeline Terraform job.
I want to search in the jobs' logs for "Warning" messages and the output the project id, pipeline id, job id, job name and the warning message to a text file. The issue I face is how to obtain the warning message from the output of ProjectJob.trace API call.
The output of ProjectJob.trace API call is (a single line):

 "TaggingVersion"  = "V1.4"\n        }\n      ~ tags_all               
 = {\n          ~ "Customer"        = "XXX" -> "Assurance"\n          ~ "Platform"        = "XXXX" -> "XXXX"\n            
 # (7 unchanged elements hidden)\n        }\n        # (7 unchanged attributes hidden)\n    }\n\nPlan: 0 to add, 2 to change, 0 to XXXXXXXXXXXXX\n\nWarning: Argument is 
 deprecated\n\n  with aws_msk_cluster.default[0],\n  on main.tf line 70, in resource "aws_msk_cluster" "default":\n  70:     ebs_volume_size = var.broker_volume_size\n\nuse 
\'storage_info\' argument instead\n\n\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94
\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\
x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\x
e2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x8
0\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94
\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\xe2\x94\x80\n\nSaved the plan to: plan\n\nTo perform exactly these actions, run the following command to 
apply:\n    terraform apply "plan"\n\x1b[0Ksection_end:XXXX:section_script_step_10\r\x1b[0K\nsection_end:XXX:step_script\r\x1b[0Ksection_start:XXXX:archive_cache\r\x1b[0K\x1b[
0K\x1b[36;1mSaving cache for successful job\x1b[0;m\x1b[0;m\n\x1b[32;1mCreating cache development-eu-west-1-workload-assurance-msk-cluster-non_protected...\x1b[0;m\ndevelopment/
eu-west-1/workload/assurance/msk/

I would like to get for each job trace (could be 100+ jobs) the warning message like "Warning: Argument is deprecated\n\n with aws_msk_cluster.default[0],\n on main.tf line 70, in resource "aws_msk_cluster" "default":\n 70: ebs_volume_size = var.broker_volume_size\n\nuse 'storage_info' argument instead".

I used:

substring2 = re.search("Warning(.*?)argument", my_string).group(1) 
print(substring)
#Output:
: Argument is deprecated\n\n  with aws_msk_cluster.default[0],\n  on main.tf line 70, in resource "aws_msk_cluster" "default":\n  70:     ebs_volume_size = var.broker_volume_size\n\nuse \'storage_info

and I got the desired result for this case but it may be the warning does not contain at the end "argument" or "instead" or "\n\n\xe2". Also the warning message is not a fixed number of characters.

My questions:

1. Could the output of the ProjectJob.trace be changed to be more friendly for string or line searching?
2. How to extract the warning message if at the end of it there is no regular pattern (it can change for any job trace)?
   Thank you.

Answer 1

If you're just working with Terraform, most commands, including plan, support the -json flag which can be used to show output in the machine-readable UI format.

The output will include a diagnostics key, which includes all the warning messages:

• diagnostics (array of objects): A JSON array of nested objects that each describe an error or warning from Terraform.

This way you don't have to parse unstructured output. I would additionally recommend saving the JSON output in a file (-out) and exposing it as an artifact, rather than keeping it in the job trace.