how to create a curl req to create a vertex pipeline with a google cloud storage pipeline spec?

Question

I'm following this doc:

I have the following curl request (changed some naming to not expose information):

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json; charset=utf-8" \
    -d '{
        "display_name":"training",
        "max_concurrent_run_count": 10,
        "create_pipeline_job_request":  {
                "parent": "projects/project_name/locations/us-west2",
                "pipelineJob": {
                    "displayName": "training",
                    "pipelineSpec": "gs://bucket_name/artifacts/pipeline.yaml",
                }
            }
    }' \
    "https://us-west2-aiplatform.googleapis.com/v1/projects/project_name/locations/us-west2/schedules"

I'm getting the error:

{
  "error": {
    "code": 400,
    "message": "Invalid value at 'schedule.create_pipeline_job_request.pipeline_job.pipeline_spec' (type.googleapis.com/google.protobuf.Struct), \"gs://project_name/artifacts/pipeline.yaml\"",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "field": "schedule.create_pipeline_job_request.pipeline_job.pipeline_spec",
            "description": "Invalid value at 'schedule.create_pipeline_job_request.pipeline_job.pipeline_spec' (type.googleapis.com/google.protobuf.Struct), \"gs://project_name/artifacts/pipeline.yaml\""
          }
        ]
      }
    ]
  }
}

Looks like the curl request MUST have a pipeline spec specified in JSON format. Is this possible at all to specify a GCS path in a curl request?

My use case is to use Cloud Scheduler to create a pipeline run weekly, pulling from a GCS file. I don't want to use the built-in recurring pipeline creation because it needs to be created again if the pipeline spec changes.

Answer 1

It can't work. The API spec require a struct (a JSON) format.

You can imagine doing something like that with yq

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json; charset=utf-8" \
    -d "{
        \"display_name\":\"training\",
        \"max_concurrent_run_count\": 10,
        \"create_pipeline_job_request\":  {
                \"parent\": \"projects/project_name/locations/us-west2\",
                \"pipelineJob\": {
                    \"displayName\": \"training\",
                    \"pipelineSpec\": $(gcloud storage cat gs://bucket_name/artifacts/pipeline.yaml | yq -p yaml -o json),
                }
            }
    }" \
    "https://us-west2-aiplatform.googleapis.com/v1/projects/project_name/locations/us-west2/schedules"