How to write same number of files as Spark partitions

Question

I'm trying to write from Spark into a single file on S3. Doing something like this

dataframe.repartition(1)
      .write
      .option("header", "true")
      .option("timestampFormat", "yyyy/MM/dd HH:mm:ss ZZ")
      .option("maxRecordsPerFile", batchSize)
      .option("delimiter", delimiter)
      .option("quote", quote)
      .format(format)
      .mode(SaveMode.Append)
      .save(tempDir)

Now as I'm forcing the partition to be 1 before writing (I also tried coalesce), I expected that one output file to be written. But it is not, this I have as many files as the number of partitions before writing.

How can I make sure that there is a single output file on S3?

Answer 1

It turns out the maxRecordsPerFile was causing this, removing it I got one final file.