bug using golang io.pipe to tar files

Question

I have been testing code using io.Pipe to tar and gunzip files into a tar ball and then unzipping using the tar utility. The follow code passes, however the untaring process keeps getting

error: tar: Truncated input file (needed 1050624 bytes, only 0 available) tar: Error exit delayed from previous errors.

This issue is really driving me crazy. It has been two weeks. I really need help debugging.

Thanks.

Development enviroment: go version go1.9 darwin/amd64

package main

import (
    "archive/tar"
    "compress/gzip"
    "fmt"
    "io"
    "log"
    "os"
    "path/filepath"
    "testing"
)

func testTarGzipPipe2(t *testing.T) {
    src := "/path/to/file/folder"

    pr, pw := io.Pipe()
    gzipWriter := gzip.NewWriter(pw)
    defer gzipWriter.Close()
    tarWriter := tar.NewWriter(gzipWriter)
    defer tarWriter.Close()

    status := make(chan bool)

    go func() {
        defer pr.Close()
        // tar to local disk
        tarFile, err := os.OpenFile("/path/to/tar/ball/test.tar.gz", os.O_RDWR|os.O_CREATE, 0755)
        if err != nil {
            log.Fatal(err)
        }
        defer tarFile.Close()
        if _, err := io.Copy(tarFile, pr); err != nil {
            log.Fatal(err)
        }

        status <- true
    }()

    err := filepath.Walk(src, func(path string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }

        header, err := tar.FileInfoHeader(info, info.Name())
        if err != nil {
            return err
        }

        // header.Name = strings.TrimPrefix(strings.Replace(path, src, "", -1), string(filepath.Separator))

        if err := tarWriter.WriteHeader(header); err != nil {
            return err
        }

        if info.Mode().IsDir() {
            return nil
        }

        fmt.Println(path)
        f, err := os.Open(path)
        if err != nil {
            return err
        }
        defer f.Close()

        if _, err := io.Copy(tarWriter, f); err != nil {
            return err
        }

        return nil
    })

    if err != nil {
        log.Fatal(err)
    }

    pw.Close()
    <-status
}

Answer 1

You are closing the pipe before the deferred Close calls on the gzipWriter and tarWriter. There's no error, because you're not checking the error on either of those close calls. You need to close the tarWriter, then the gzipWriter, then the PipeWriter, in that order.

However, there's no reason for the pipe at all in this code, and you can remove the goroutine and the associated coordination altogether if you write directly to the file.

tarFile, err := os.OpenFile("/tmp/test.tar.gz", os.O_RDWR|os.O_CREATE, 0644)
if err != nil {
    log.Fatal(err)
}
defer tarFile.Close()

gzipWriter := gzip.NewWriter(tarFile)
defer gzipWriter.Close()
tarWriter := tar.NewWriter(gzipWriter)
defer tarWriter.Close()