How to use Log4perl to rotate log files in multithread perl application

Question

Below is the sample code that I have tried to rotate a log file in multithread application using log4perl. But it is working fine unless it is a multithread application. Logs are not rotated and Log file grows in size. Can anyone guide me on where I am going wrong?

use strict;
use warnings;
use Log::Log4perl;
use POSIX;
use threads;
use threads::shared;

my @InputFiles;

my $InputDirectory=$ARGV[0];
my $LogName=$ARGV[1];
opendir(DIR,$InputDirectory) or die "could not open the input directory";
@InputFiles=readdir(DIR);
close(DIR);
my $file;

    #logger_configuration
my $log_conf ="
   log4perl.rootLogger              = DEBUG, LOG1

   log4perl.appender.LOG1           = Log::Dispatch::FileRotate
   log4perl.appender.LOG1.filename  = $LogName
   log4perl.appender.LOG1.mode      = append
   log4perl.appender.LOG1.autoflush = 1
   log4perl.appender.LOG1.size      = 10000
   log4perl.appender.LOG1.max       = 20
   log4perl.appender.LOG1.layout    = Log::Log4perl::Layout::PatternLayout
   log4perl.appender.LOG1.layout.ConversionPattern = \%d{yyyy-MM-dd HH:mm:ss}|\%P|\%m|\%n
";

#loading the configuration file
Log::Log4perl::init(\$log_conf);

#creating logger instance
my $logger = Log::Log4perl->get_logger();

my $thread_count=5;
my $file_total= scalar @InputFiles;
#print STDERR "$file_total\n";

#dividing total files among the no of given threads
my $div = $file_total/$thread_count;
$div = ceil($div);
my $start = 0;
my $end = $div;
my @threads;
for (my $count = 1; $count <=$thread_count ; $count++) 
{
    my $thread = threads->new(\&process,$start,$end);
    push(@threads,$thread);        
    $start = $end;
    $end = $end + $div;
    if ($end > $file_total)
    {
        $end = $file_total;
    }
}

foreach (@threads) 
{
   $_->join;
}

sub process
{
    my $lstart = shift;
    my $lend = shift;
    my $id = threads->tid();
    for (my $index = $lstart; $index < $lend; ++$index) 
    {   
      $logger->info($InputFiles[$index]);
    }
}

Answer 1

OK, pretty fundamentally your problem is this - your 'logger' is created before your threads start. This means that all your threads will have the same file handles.

This will inherently cause you problems, unless you've got some sort of arbitration mechanism for file IO. Think of your threads as separate programs, all trying to open and write to the same file - and you can see how messy it might get.

I would suggest instead that what you need to do is create another thread for the logger, and send the IO through something like Thread::Queue

use Thread::Queue;
my $log_q = Thread::Queue -> new(); 

sub logger_thread {
     #init logger here 

    while ( my $log_item = $log_q -> dequeue() ) {
        $logger -> info ( $log_item );
    }
}

my $log_thread = threads -> create ( \&logger_thread ); 

And then replace the $logger -> info (....) with:

$log_q -> enqueue($message_to_log);

Then, once you've joined all your 'process' threads (e.g. as you are right now) close off the logger thread:

$log_q -> end();
$log_thread -> join(); 

This will cause each of the threads to queue log messages, and once they're finished (and joined) you close the queue so the logger knows it's 'done' - and so will exit once the queue is empty and can be joined.

Multithreading file IO is messy, so it's better to avoid as much as possible.