How to handle large amount of data from array of pointer struct faster in Golang?

50 Views Asked by At
  • I want to create a CSV file from []*dto.Customer.
  • Getting all values and convert each struct values into a single string & then push into an array.
  • Finally write into string builder then convert into []byte.

Here is the function,

func generateCSV(customers []*dto.Customer) []byte {

    var csvData strings.Builder
    var srcFileArray []string

    header := []string{"created Time", "Reference Number", "Customer Name", "Mobile Number", "Area", "Payment Mode"}
    csvData.WriteString(strings.Join(header, ",") + "\n")

    for _, cust := range customers {
        record := formatTimeZone(cust.CreatedDate).Format("2006-01-02 15:04:05") + "," + cust.ReferenceID + "," + cust.Name + "," + cust.number + "," + cust.Area + "," + cust.Payment
        srcFileArray = append(srcFileArray, record)
    }

    csvData.WriteString(strings.Join(srcFileArray, "\n"))
    byteArray := []byte(csvData.String())

    return byteArray
}

The struct I'm using,

type Customer struct {
    Name          string      `json:"name,omitempty" bson:"name,omitempty"`
    Number        string      `json:"number,omitempty" bson:"number,omitempty"`
    ReferenceID   string      `json:"referenceID,omitempty" bson:"referenceId,omitempty"`
    CreatedDate   time.Time   `json:"createdDate,omitempty" bson:"createdDate,omitempty"`
    Area          string      `json:"area,omitempty" bson:"area,omitempty"`
    Payment       string      `json:"payment,omitempty" bson:"payment,omitempty"`
}
  • The problem is when am handling hundreds of thousands of data in customer array, it takes 2 minutes for forming the string. so, my request gets failed and showing 502 error.

How to do that in minimal time period?

  • I tried to split the array into two half and then run both array in Go routines, but the data given from this is not in order.
  • I want the data in same order from the array.
0

There are 0 best solutions below