Is there a way to pull only metadata and specific range in grib?

Question

There is an increasing huge abundance of weather data available on cloud buckets. Awesome! However they are not stored in cloud optimized formats. I was wondering if there was a way to only pull metadata from grib2 files stored on AWS, and subsequently only pull single points from those files. Same question for netcdf4. I know Netcdf4 support libraries allow you to do so for files on disk, but I have no idea how to do it on the cloud.

I'm at a loss for which resources I should be looking into in order to explore this. Any help would be really appreciated.

Answer 1

You could parse the GRIB2 file on-the-fly and drop everything you don't need right away. Each GRIB2 file contains one or more GRIB2 messages which has the following structure:

• Section 0: Indicator Section
• Section 1: Identification Section
• Section 2: Local Use Section (optional)
• Section 3: Grid Definition Section (can be repeated)
• Section 4: Product Definition Section (can be repeated)
• Section 5: Data Representation Section (can be repeated)
• Section 6: Bit-Map Section (can be repeated)
• Section 7: Data Section (can be repeated)
• Section 8: End Section

Section 0 has in GRIB2 always 16bytes, Section 8 always 4 bytes. The rest has always starts with length of the section (4 bytes) and section number (1 byte). Therefore it should be easy to skip all section you don't need fast. You can then read only section 1, 3 or 5, depending on what meta-data you want.

There is a drawback however. If I understand it correctly you want to do that on online resources. In this case you will download the whole file while skipping over some or most of its parts.

If you are trying to build some kind of index of available GRIB data this will be probably one options. Kind of a GRIB crawler.

Note that GRIB1 has a bit different structure

More details about GRIB2 sections: https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/