dbt Set custom condition for external source tables

73 Views Asked by At

I have a csv file in a GCS bucket. I want to source it but with a condition like "rows before a certain date".

I have a source.yml file like this:

  - name: my_ref_name
    database: my_database
    schema: my_schema
    tables:
      - name: my_table
        external:
          location: "gs://my-gcs-bucket/my-path/my-file.csv"
          options:
            format: csv
            ignore_unknown_values: true
        source_config:
          where: "created_at < '2024-01-01'"

I've also tried the key: config: source_config.

I've checked dbt docs but it doesn't seem like there's a config property like that. How can I achieve that?

0

There are 0 best solutions below