Why is this non-recursive "find" much slower than searching output of "ls"?

132 Views Asked by At

I wrote two different script as below:

# Script 1:
current_date=$(date +"%m-%d-%Y")
logsPath="/hadoop_common/smallsite/realtime/current/spark/logs"

find $logsPath -maxdepth 1 -name "*$current_date*" -print > tmp

and

# Script 2:
current_date=$(date +"%m-%d-%Y")
logsPath="/hadoop_common/smallsite/realtime/current/spark/logs"

ls $logsPath | grep "$current_date" > tmp
sed -i "s|^|$logsPath|" tmp

First script took 24 minutes to list 2367 filePath and second script took 16 seconds to list same number of filePath.

Why this much difference? I ran them on different days, multiple times and in any order and always first script took above 20 minutes, and second script took below 20 seconds.

OS: Red Hat


UPDATE 2

find (GNU findutils) 4.4.2

I ran this script

#!/bin/bash

current_date=$(date +"%m-%d-%Y")

logsPath="/hadoop_common/smallsite/realtime/current/spark/logs"

touch resultStat

date >> resultStat

ls $logsPath | grep "$current_date" > tmp1
sed -i "s|^|$logsPath|" tmp1

date >> resultStat

find $logsPath -maxdepth 1 -type f -name "*$current_date*" -print > tmp2

date >> resultStat

strace ls "$logsPath" 2>&1 | grep -c stat >> resultStat

date >> resultStat

strace find "$logsPath" -maxdepth 1 -name "*$current_date*" -print 2>&1 | grep -c stat >> resultStat

date >> resultStat

ls -1q $logsPath | wc -l >> resultStat

date >> resultStat

files=(/hadoop_common/smallsite/realtime/current/spark/logs/*"$(date +%m-%d-%Y)"* )

date >> resultStat

resultStat content:

Thu Mar 29 19:14:28 UTC 2018
Thu Mar 29 19:14:47 UTC 2018
Thu Mar 29 19:41:26 UTC 2018
14
Thu Mar 29 19:41:44 UTC 2018
189805
Thu Mar 29 20:08:30 UTC 2018
190348
Thu Mar 29 20:08:48 UTC 2018
Thu Mar 29 20:09:06 UTC 2018
0

There are 0 best solutions below