I have a PySpark Transform that makes an API call, causing external side-effects. Because Preview runs on a sample of the input rows, previewing the transform causes bad API calls to be made.
Is there a way to detect whether my transform is running in Preview, so that I can avoid making the API calls in that case?
My code so far:
from transforms.api import transform_df, Input, incremental
@incremental()
@transform_df(
df=Input("my_input")
)
def compute(df):
return make_api_call(df)
You can check something like this from within your transform code: