java.nio.file implementation for AWS

Question

Is there any official java.nio.file implementation for AWS?

I found one for GoogleCloudStorage here, and need similar for AWS and Azure.

Answer 1

General

For anyone interested, the Upplication/Amazon-S3-FileSystem-NIO2 was abandoned a very long time ago. Pull requests and fixes are no longer accepted. Many developers forked it and made their fixes for their own purposes and then those forks died as well.

We decided to make a new new spin-off project -- not as a fork, but as a new project called s3fs-nio that is based on Upplication/Amazon-S3-FileSystem-NIO2's master. You can find out more about the history and reasoning here. We have tried to collect some of the more important fixes from forks and have added them to our project (with the authors' permissions).

Our spin-off uses AWS SDK for Java version 2.x and is under an Apache 2.0 license.

The documentation for configuring things can be found here.

Code examples are located here.

We've now cut a release for org.carlspring.cloud.aws:s3fs-nio:1.0.0 and this is available via Maven Central (https://repo.maven.apache.org/maven2/). Here are our Release Notes.

We are also working on improving our documentation and contributions would be highly appreciated.

We would like to welcome you to test and report back any findings!

For those of you interested in contributing, there is plenty yet to be done and we'd be more than happy to have you aboard!

Basic Example

Here's a basic configuration:

• Add the respective dependency (you can check the available versions here):
  • For Maven:

<dependency>
    <groupId>org.carlspring.cloud.aws</groupId>
    <artifactId>s3fs-nio</artifactId>
    <version>1.0.0</version>
</dependency>

• For Gradle:

implementation 'org.carlspring.cloud.aws:s3fs-nio:1.0.0'

• Use a Service Locator and an amazon.properties file In your src/main/resources/amazon.properties, add the following settings:

s3fs.access.key=access-key
s3fs.secret.key=secret-key

Use the following code to create the FileSystem and set it to a specific end-point:

FileSystems.newFileSystem(URI.create("s3:///"),
                          new HashMap<>(),
                          Thread.currentThread()
                                .getContextClassLoader());

Answer 2

An implementation of the FileSystemProvider SPI that uses asynchronous read-ahead buffering is available from github and from Maven Central. This library supports reads and writes.

You can include it in a Maven project using:

<dependency>
    <groupId>software.amazon.nio.s3</groupId>
    <artifactId>aws-java-nio-spi-for-s3</artifactId>
    <version>1.2.1</version>
</dependency>

Or you can just place the JAR file on your classpath and the JVM will route all URIs with a scheme of s3 to this provider.

Answer 3

According to a comment on S3: provide java.nio.FileSystem implementation #1388, someone just released v1.0.0 (now v1.0.1) of S3FS NIO at carlspring / s3fs-nio on GitHub. I haven't had a chance to try it out. If anyone does, please leave your experience here in the comments.

Answer 4

You can try using Amazon AWS S3 FileSystem Provider JSR-203 for Java 7 (NIO2)

Download from Maven Central

<dependency>
    <groupId>com.upplication</groupId>
    <artifactId>s3fs</artifactId>
    <version>2.2.2</version>
</dependency>

add to your META-INF/services/java.nio.file.spi.FileSystemProvider (create if not exists yet) a new line like this: com.upplication.s3fs.S3FileSystemProvider.

Use this code to create the fileSystem and set to a concrete endpoint.

FileSystems.newFileSystem("s3:///", new HashMap<String,Object>(), Thread.currentThread().getContextClassLoader());

How to use in Apache MINA

public FileSystemFactory createFileSystemFactory(String bucketName) throws IOException, URISyntaxException {
    FileSystem fileSystem = FileSystems.newFileSystem(new URI("s3:///"), env, Thread.currentThread().getContextClassLoader());
    String bucketPath = fileSystem.getPath("/" + bucketName);

    return new VirtualFileSystemFactory(bucketPath);
}

How to use in Spring

Add to classpath and configure:

@Configuration
public class AwsConfig {

    @Value("${upplication.aws.accessKey}")
    private String accessKey;

    @Value("${upplication.aws.secretKey}")
    private String secretKey;

    @Bean
    public FileSystem s3FileSystem() throws IOException {
        Map<String, String> env = new HashMap<>();
        env.put(com.upplication.s3fs.AmazonS3Factory.ACCESS_KEY, accessKey);
        env.put(com.upplication.s3fs.AmazonS3Factory.SECRET_KEY, secretKey);

        return FileSystems.newFileSystem(URI.create("s3:///"), env, Thread.currentThread().getContextClassLoader());
    }
}

Inject in any spring component:

@Autowired
private FileSystem s3FileSystem;