Invoke-Sqlcmd and Export-Csv consume huge amounts of memory?

494 Views Asked by At

Currently, we're exporting csv file from SQL Server by using Invoke-Sqlcmd and Export-Csv commands in PowerShell.

To upload to Redshift, we needs to enclose the data by double-quotations and convert to UTF8 encoding.

So we considered the combination of Invoke-Sqlcmd and Export-Csv easier and simple to be developed.

Here's the sample command:

Invoke-Sqlcmd –Username $StrUserName –Password $StrPassword 
              -Query $StrQry -ServerInstance $StrSvrName 
              -IgnoreProviderContext -QueryTimeOut 3000 | Export-Csv -path $StrExpFile -Encoding UTF8

There's no problem for normal amount export operations, but when the target table data exceeds over a million rows, it can consume huge amounts of memory.

If you have any god idea to reduce the cost or any alternative way to export data, enclosing by double-quotations with encoding UTF-8, please advise me.

Thank you.

0

There are 0 best solutions below