AWS SageMaker pipeline xcom alternative

Question

I'm currently working with SageMaker Pipelines and have a question about inter-step communication. In systems like Apache Airflow, there's a concept of XCom, which allows tasks to exchange small data artifacts or messages. I'm wondering if there's an equivalent feature in SageMaker Pipelines that would allow us to pass variables or small data artifacts directly between steps, without writing to and reading from an S3 bucket? Any help would be greatly appreciated.

Answer 1

With different steps you can take the output of the previous step and pass it in as an input for the next step. For example if you have a training step that you have defined like the following:

training_step = TrainingStep(
    name="Training",
    step_args=train_args,
)

You can grab the model artifacts generated by this training job with the following code.

model_data=training_step.properties.ModelArtifacts.S3ModelArtifacts