parquet.net ParquetOptions.UseDeltaBinaryPackedEncoding how to disable

67 Views Asked by At

I am using Parquet.Net library to generate parquet files from C# code. 4.15.0 version works fine, but after switching to latest 4.16.4 the files generated are not readable using ParquetViewer. It throws error "DELTA_BINARY_PACKED" encoding is not support. I found on the net that this can be disabled using ParquetOptions. However, I am not able to find this option as part of the ParquetWriter method.

This is the code snippet

private async Task ParquetWriteRowGroup(ParquetSchema schema, List<dynamic> data)
{
    Console.WriteLine($"output file =>  {OutputFilePath}");
    using (Stream fs = File.Open(OutputFilePath, FileMode.CreateNew))
    {
        using (ParquetWriter writer = await ParquetWriter.CreateAsync(schema, fs))
        {
            writer.CompressionMethod = CompressionMethod.Snappy;


            using (ParquetRowGroupWriter groupWriter = writer.CreateRowGroup())
            {
                for (int i = 0; i < data.Count; i++)
                {
                    await groupWriter.WriteColumnAsync(new DataColumn(schema.DataFields[i], data[i]));
                }
            }
        }
    }
}
1

There are 1 best solutions below

0
On

Got caught out with this also try this:

ParquetOptions options = new ParquetOptions();
options.UseDeltaBinaryPackedEncoding = false;
ParquetSerializerOptions serializerOptions = new ParquetSerializerOptions();
serializerOptions.ParquetOptions = options;
await ParquetSerializer.SerializeAsync(stuff, destination, 
serializerOptions);