Transfer data onto aws s3 snowball using c#

Question

I'm trying to import all my data onto s3 using a snowball, but I need to keep track of some metadata for each file. I have some c# code that uploads the files directly to the bucket with this metadata, which is working fine. I thought I would be able to use the snowball s3 adapter to connect to my snowball and upload data onto from my c# code, because this link says:

You can use the S3 Adapter with existing Amazon S3 interfaces like the AWS SDKs, the AWS CLI, or your own custom Amazon S3 REST client.

I have the adapter running on http://localhost:8090, so I tried setting that as the endpoint for the Amazon3Client by passing in an AmazonS3Config object with that endpoint as the ServiceUrl, but when I try to upload the file using either a PutObjectRequest or the TransferUtility, I get the following exception:

 Amazon.Runtime.AmazonServiceException: A WebException with status
 NameResolutionFailure was thrown. ---> System.Net.WebException: The
 remote name could not be resolved: 'BUCKET.localhost' at
 System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
 at System.Net.HttpWebRequest.GetRequestStream() at
 Amazon.Runtime.Internal.HttpRequest.GetRequestContent() at
 Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext
 executionContext)
 --- End of inner exception stack trace --- at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext
 executionContext, WebException exception) at
 Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext
 executionContext, Exception exception) at
 Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext
 executionContext, Exception exception) at
 Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.Signer.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.CredentialsRetriever.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3KmsHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.EndpointResolver.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3PostMarshallHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.Marshaller.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3PreMarshallHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.AmazonServiceClient.Invoke(TRequest request,
 IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller) at
 Amazon.S3.AmazonS3Client.PutObject(PutObjectRequest request)

I tried using my local ip address, as well as 127.0.0.1, but all got the same error. When I use the amazon cli to upload the document to the snowball, using --endpoint to change the endpoint, it worked (with both localhost or my local ip), so apparently it's just something about the .net sdk. But I would really strongly prefer to use my existing c# code than to write up a command line program to do this - it won't be so simple from the cli.

Edit: here is the c# code that isn't working:

var config = new AmazonS3Config
{
  ServiceURL = "http://localhost:8090"
};

var file = new FileInfo(FILENAME);

var pRequest = new PutObjectRequest
{
  BucketName = BUCKET,
  Key = S3FILENAME,
  StorageClass = S3StorageClass.StandardInfrequentAccess,
  FilePath = file.FullName
};

pRequest.Metadata.Add("x-amz-meta-local-date-modified", file.CreationTime.ToFileTimeUtc().ToString());

using (var client = new AmazonS3Client(config))
{
  client.PutObject(pRequest);
}

This is the command line statement that works:

aws s3 cp FILENAME s3://BUCKET/S3FILENAME --endpoint http://localhost:8090

Can anyone help me?

Thanks

Answer 1

The problem is, the AWS SDK prepends your BucketName to the ServiceURL by default, so you end up with an invalid URL: http://BUCKET.localhost:8090.

You need to use the ForcePathStyle param, which prevents the BucketName from being prepended to the ServiceURL:

var config = new AmazonS3Config
{
    ServiceURL = "http://localhost:8090",
    ForcePathStyle = true
};