What happens if my stripe count is set to more than my number of stripes

Question

I have a doubt regarding Lustre file system. If I have a file of size 64 GB and I set stripe size to 1GB, my number of stripes become 64. But if I set my stripe count as 128, what does the Lustre do in that case?

Answer 1

Probably you are missing considering the details like number of OSTs while thinking about this. I will be more elaborated on this because I have seen confusion about striping in many. Please bear with me.

So in case of file of 64GB with stripe_size=1GB; stripes of 1GB are 64 but not the stripe_count. We can have any no. of stripes depending upon the file size and stripe_size, but not the stripe_count which is dependent on OSTs. Here is small experiment, I have 2 OSTs and creating 64 MB file with stripe_size=1M(default)...

/dev/loop0 on /mnt/mds1 type lustre (rw,loop=/dev/loop0)
/dev/loop1 on /mnt/ost1 type lustre (rw,loop=/dev/loop1)
/dev/loop2 on /mnt/ost2 type lustre (rw,loop=/dev/loop2)
ashish-203@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,flock)
[root@ashish-203 tests]# lfs df -h
UUID                       bytes        Used   Available Use% Mounted on
lustre-MDT0000_UUID       146.4M       17.5M      119.4M  13% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       183.1M       25.2M      147.7M  15% /mnt/lustre[OST:0]
lustre-OST0001_UUID       183.1M       25.2M      147.7M  15% /mnt/lustre[OST:1]

filesystem summary:       366.1M       50.3M      295.5M  15% /mnt/lustre

Now I create a file foo with -c -1(stripe on all OSTs)...

[root@ashish-203 tests]# lfs setstripe -c -1 /mnt/lustre/foo
[root@ashish-203 tests]# lfs getstripe /mnt/lustre/foo
/mnt/lustre/foo
lmm_stripe_count:   2
lmm_stripe_size:    1048576
lmm_pattern:        1
lmm_layout_gen:     0
lmm_stripe_offset:  1
    obdidx           objid           objid           group
         1               2            0x2                0
         0               2            0x2                0
[root@ashish-203 tests]# dd if=/dev/urandom of=/mnt/lustre/foo bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 7.465 s, 9.0 MB/s
[root@ashish-203 tests]# du -h /mnt/lustre/foo
64M     /mnt/lustre/foo
[root@ashish-203 tests]# lfs df -h /mnt/lustre/foo
UUID                       bytes        Used   Available Use% Mounted on
lustre-MDT0000_UUID       146.4M       17.5M      119.4M  13% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       183.1M       57.2M      115.8M  33% /mnt/lustre[OST:0]
lustre-OST0001_UUID       183.1M       57.2M      115.8M  33% /mnt/lustre[OST:1]

filesystem summary:       366.1M      114.3M      231.6M  33% /mnt/lustre
[root@ashish-203 tests]# lfs getstripe -c /mnt/lustre/foo
2

So here we can see file 64MB is created and stripe_count is 2 which means data is written to all the 2 OSTs equally.

stripe_count is equal to the no. of objects in a single file. and each object of the file is stored on different OST. Hence it is the number of OSTs which is responsible for no. of stripe_count.

Now when you say if I change stripe_count to 128 then you should have the 128 OSTs if you don't then file will be striped only on available OSTs and that will be your stripe_count(if file is created with "-c -1" option).

But if I set my stripe count as 128, what does the Lustre do in that case?

So if you have suppose 64 OSTs then lustre will stripe file only on 64 OSTs

Here is small experiment for above theory...

[root@ashish-203 tests]# lfs setstripe -c 4 /mnt/lustre/bar
[root@ashish-203 tests]# dd if=/dev/urandom of=/mnt/lustre/bar bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 7.31459 s, 9.2 MB/s
[root@ashish-203 tests]# du -h /mnt/lustre/bar
64M     /mnt/lustre/bar
[root@ashish-203 tests]# lfs df -h /mnt/lustre
UUID                       bytes        Used   Available Use% Mounted on
lustre-MDT0000_UUID       146.4M       17.5M      119.4M  13% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       183.1M       89.2M       83.9M  52% /mnt/lustre[OST:0]
lustre-OST0001_UUID       183.1M       89.2M       83.9M  52% /mnt/lustre[OST:1]

filesystem summary:       366.1M      178.3M      167.7M  52% /mnt/lustre
[root@ashish-203 tests]# lfs getstripe -c /mnt/lustre/bar
2

You can see in spite of setting the stripe_count=4 stripe_count is 2 and data was written only 2 OSTs.

Summary:- Don't confuse stripe_count with no. of stripes. stripe_count is on how many OSTs you want to stripe your data if they are available and stripes = (file size / stripe_size).

Hope that answers you question...

Answer 2

If the stripe count is set to 128, 64 stripes are utilized out of 128 and rest 64 stripes are left out. Lustre filesystem writes data in round-robin fashion which is necessary while striping. Also to make sure that the rest of the stripes are not left out imbalanced we need to set a property so that the write starts from 65th stripe.