Currently, we're exporting csv file from SQL Server by using Invoke-Sqlcmd
and Export-Csv
commands in PowerShell.
To upload to Redshift, we needs to enclose the data by double-quotations and convert to UTF8 encoding.
So we considered the combination of Invoke-Sqlcmd
and Export-Csv
easier and simple to be developed.
Here's the sample command:
Invoke-Sqlcmd –Username $StrUserName –Password $StrPassword
-Query $StrQry -ServerInstance $StrSvrName
-IgnoreProviderContext -QueryTimeOut 3000 | Export-Csv -path $StrExpFile -Encoding UTF8
There's no problem for normal amount export operations, but when the target table data exceeds over a million rows, it can consume huge amounts of memory.
If you have any god idea to reduce the cost or any alternative way to export data, enclosing by double-quotations with encoding UTF-8, please advise me.
Thank you.