Using Event trigger Azure function, how to convert csv file to parquet file using c#

524 Views Asked by At

I want to get a file in azure blob storage and using event trigger function, I want to convert this csv file into parquet format and upload it to another blob storage. I tried to use choparquet reader, but it takes a lot of time to convert the file to parquet format and when a large files comes, few events are missed from the queue and not all files are this converted in to parquet format. I am using c#

Can anyone help in simplified approach to this problem? The code I am working is

using (Stream blobStream = await blockBlob.OpenWriteAsync(accessCondition, null, null))

                    {
                        using (var reader = new StreamReader(longestFile.Open(), Encoding.UTF8))
                        using (var r = ChoCSVReader.LoadText(reader.ReadToEnd()).WithMaxScanRows(2).WithDelimiter("\t"))
                        {
                            log.LogInformation($"Inside csv reader");
                            using (var w = new ChoParquetWriter(blobStream))
                            {
                                log.LogInformation($"Inside parquet writer");
                                w.Write(r);
                                w.Serialize();
                                w.Close();
                            }
                        }
                    }
0

There are 0 best solutions below