Log Jenkins Console Output to S3

Question

So i have to Log my Jenkins Job Console Output to S3 Bucket and this involves many jobs to be configured in a way such that the console output log of the Jenkins job is pushed to S3 post build.

Answer 1

One way is to simply upload logs file to s3 using AWS CLI. You can try below approach.

echo "hello from job"
echo "job logs to s3"

cat ${JENKINS_HOME}/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log >> ${BUILD_NUMBER}.log

/usr/bin/aws s3 cp  ${BUILD_NUMBER}.log s3://s3_bucket_name/${BUILD_NUMBER}.log

Or you can set cron job to sync local logs folder with s3.

for example, you can set cron job that will run every minutes and sync local log file to s3 or copy and have versioning on s3 side, or the other option is to trigger cronjob script after completion of each job.

for logs_file in $(ls /var/lib/jenkins/jobs/*/builds/lastSuccessfulBuild/log); 
#now logs_file contain all job logs file path


do  
   echo "copy logs file to s3 $logs_file
   # to identify file unique
   s3_file_name=$(date +'%Y%-m%d%s')
   aws s3 cp  $logs_file s3://s3_bucket_name/"${s3_file_name}".log
done

#Note there are different directories you can copy accordingly
#lastFailedBuild
#lastSuccessfulBuild
# lastUnstableBuild
#lastUnsuccessfulBuild
# and logs by build numbers

Answer 2

See tee in Utility steps:

script {
    tee("$WORKSPACE/my_output.txt") {
       sh "echo Hello"
    }
}

You may then upload the results to s3:

def path = env.JOB_NAME.replace('%2F','-').replace('/', "-").replace('_', "-").toLowerCase()

withAWS(endpointUrl:"...", credentials:"...") {
  s3Upload bucket: "jenkins",
    path:"${path}/${env.BUILD_NUMBER}/",
    file: "my_output.txt",
    workingDir: "${env.WORKSPACE}"
}