Today we've received an error in nginx Pod:
pread() returned only 126 bytes instead of 796 in /etc/nginx/nginx.conf:1
It is running nginx Pod in EKS with image
nginx@sha256:aa338f686356e1b0a4287cbed06b86d460c22928c4989865a01b7aac321ca14c
and EKS nodes are using OS image
ami-0c2a649f9d523b271
(on older AMI it works well)
Quick search for the error message in nginx source https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/nginx/nginx%24+%22bytes+instead+of%22&patternType=regexp&sm=0&filters=%5B%5B%22type%22%2C%22Code%22%2C%22type%3Afile%22%5D%5D
Hints the ngx_conf_read_token function.
It tries to read a config file of size 796 but read only 126
Size is correct:
# ls -lah /etc/nginx/nginx.conf
-rw-r--r-- 1 root root 796 Mar 6 03:23 /etc/nginx/nginx.conf
Also pread man says (the same for read):
Note that it is not an error for a successful call to transfer
fewer bytes than requested (see read(2) and write(2)).
https://man7.org/linux/man-pages/man2/pread.2.html
It is really strange that ngx_read_file returned only 126 bytes. https://sourcegraph.com/github.com/nginx/nginx@ef96f5835468ff8d40df29b0ddbc04ec1e5e1582/-/blob/src/os/unix/ngx_files.c?L31
! Restarting Pod (recreating container) doesn't help. Issues persists.
Newer image however works well sha256:fdaba7709b3ad1bbf72f3adb96279160bbd636f357b1832cb5fb8d0e41280425
It is well known that read() may return less than requested. Why nginx tried to read config file in one read() and fails otherwise?
What might be the reason to return less than actual files size on EKS k8s node running as EC2 instance using AMI ami-0c2a649f9d523b271 ? (maybe other fresh kernels will reproduce this issue, older AMI works well)
strace:
futex(0xf7d7b550, FUTEX_WAKE_PRIVATE, 2147483647) = 0
openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY|O_LARGEFILE) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=11118, ...}) = 0
read(4, "...", 4096) = 4096
read(4, "...", 4096) = 4096
read(4, "...", 4096) = 2926
read(4, "", 4096) = 0
close(4) = 0
...
openat(AT_FDCWD, "/etc/nginx/nginx.conf", O_RDONLY|O_LARGEFILE) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=10, ...}) = 0
pread64(4, "\123J\123\123\123J\123\123\12b=X\12b=Xr an object signing certificate this would be used.\n# nsCertType = objsign\n\n# For normal client use this is ty", 10, 0) = 126
(replaced numbers with 123 because I do not understand what it is)
and we've got pread() returned only 126 bytes instead of 116.
The pread() has returned 126 bytes from previous fd 4?
Resembles open_log_file_cache except it fails on config file
http://nginx.org/en/docs/http/ngx_http_log_module.html#open_log_file_cache
it also might be due nginx image was for 32 bits and somehow was running on 64 bits OS