bash: /usr/bin/find: Argument list too long

Question

I have 27 000 logs file and need to parse them in one time in GoAccess. every time i run this command

goaccess /home/goaccess/part/2/*.log --log-format='%^ %dT%t.%^ %v %h:%^ %^ %^ %T %^ %s %^ %^ %b "%r" "%u" %k %K %^' -o /var/www/html/index.html --date-format=%Y-%m-%d --time-format=%T

it says

bash: /usr/bin/find: Argument list too long

i have tried all the things suggested by chatgpt and bard still same problem.

Answer 1

Try this instead:

find /home/goaccess/part/2/ -name \*\.log -print0 | xargs -0 -I goaccess {} --log-format='%^ %dT%t.%^ %v %h:%^ %^ %^ %T %^ %s %^ %^ %b "%r" "%u" %k %K %^' -o /var/www/html/index.html --date-format=%Y-%m-%d --time-format=%T

This should batch the goaccess commands into groups that fit under the arg list restriction amount, using xargs (which was specifically designed for this type of issue), and allow an arbitrary number of files to be processed. See man xargs for more details and examples.

Answer 2

You could try a few things.

Change into the log directory first:

cd /home/goaccess/part/2 && goaccess *.log --log-format='%^ %dT%t.%^ %v %h:%^ %^ %^ %T %^ %s %^ %^ %b "%r" "%u" %k %K %^' -o /var/www/html/index.html --date-format=%Y-%m-%d --time-format=%T

but for 27k files it is probably not enough.

If the filename is not important for your goaccess invocation (and I don't know it well enough to say if it is important) piping the log content would work:

find /home/goaccess/part/2/ -name '*.log' -exec cat {} + | goaccess - --log-format='%^ %dT%t.%^ %v %h:%^ %^ %^ %T %^ %s %^ %^ %b "%r" "%u" %k %K %^' -o /var/www/html/index.html --date-format=%Y-%m-%d --time-format=%T

As last resort: goaccess seem to support setting the input log file in the configuration file but it is not clear if it can be more than one. If so you could generate a config file which includes all 28k log files.