How to simulate hdfs operations using spring data

605 Views Asked by At

I'm new to spring data-hadoop and would like to ask one general question. I have files in different format and would like to extract the useful content with Apache Tika and store as text files in HDFS. I've gone through the reference documentation of spring data-hadoop(http://docs.spring.io/spring-hadoop/docs/2.0.0.RELEASE/reference/html/store.html) but didn't understand how to do it. And I didn't find any other useful resources for this.

Is there any sample projects or sources for writing data to HDFS using spring data-hadoop ?

1

There are 1 best solutions below

0
Sachin On BEST ANSWER

From Risberg's comment one useful example :-

https://github.com/trisberg/springone-2015/tree/master/boot-ingest

Another code snippet with TextFileWriter implementation of DataWriter interface :-

   //build naming strategy
             ChainedFileNamingStrategy namingStrategy = 
      new ChainedFileNamingStrategy( 
            Arrays.asList(new FileNamingStrategy[] { 
                new StaticFileNamingStrategy("document"),
                         new UuidFileNamingStrategy(someUUID),
                          new StaticFileNamingStrategy("txt", ".") }));
        //set the naming strategy 
            textFileWriter.setFileNamingStrategy(namingStrategy);
            textFileWriter.write("this is a test content");
       //flush and close the writer
            textFileWriter.flush();
            textFileWriter.close();