I'm new at Scylla DB and i was just creating Nodes using Commands given in documentation.
What happens is when i create Two nodes no matter what they're referred as, they happen to connect to Scylla DB server, but when it comes to third node i face this error.
For Example
After successfully creating Node_X by using
docker run --name Node_X -d scylladb/scylla:5.2.0 --overprovisioned 1 --smp 1
I checked Node Tool Status for Node_X using
docker exec -it Node_X nodetool status
and got
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.17.0.3 1.03 MB 256 ? 57765ee9-bb4b-4287-a7c1-b47878e22926 rack1
UN 172.17.0.2 256 KB 256 ? bce9e5b7-e5ab-47aa-88d3-3413fd6e4980 rack1
Same goes with Node_Y which is made using command docker run --name Node_Y -d scylladb/scylla:5.2.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)" --overprovisioned 1 --smp 1 which is running perfectly fine. Giving it's nodetool status successfully.
BUT when it comes to Node_Z (Third Node) it's giving error. For Creating i used this command
docker run --name Node_Z -d scylladb/scylla:5.2.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)" --overprovisioned 1 --smp 1
For Status Checking used this command
docker exec -it Node_Z nodetool status
And blessed with the error:
nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused (Connection refused)
See 'nodetool help' or 'nodetool help <command>'.
So as usual i docker logged Node_Z which is not connecting and found out
FATAL: Exception during startup, aborting: std::runtime_error (Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application)
Fair enough, i got to their documentation and found that we can increase value of aio-max-nr using echo "fs.aio-max-nr = 1048576" >> /etc/sysctl.conf but before that we need to locate this file, FROM HERE the real problem started I used this command to located file but blessed with another error
cat /proc/sys/fs/aio-max-nr
cat: /proc/sys/fs/aio-max-nr: No such file or directory
For save changing we use this and it's giving:
sysctl -p /etc/sysctl.conf
sysctl: illegal option -- p
usage: sysctl [-bdehiNnoqx] name[=value] ...
sysctl [-bdehNnoqx] -a
Used other techniques such as
sudo sysctl -w fs.aio-max-nr=1048576
sysctl: unknown oid 'fs.aio-max-nr'
Then another technique
sudo nano /etc/sysctl.conf
fs.aio-max-nr = 1048576
After that the used another technique:
docker exec -it Node_Z bash
Then after entering into bash, change aio-max-nr using:
sudo sysctl fs.aio-max-nr=1048576
sysctl: setting key "fs.aio-max-nr", ignoring: Read-only file system
it will change that number, you can check using:
cat /proc/sys/fs/aio-max-nr
but in my case value wasn't changed, it remains:
65536
then i found out that There's Read-Only restriction so i tried to change the File method to Read and Write using :
sudo mount -o remount,rw /
mount: /: permission denied.
This is as far as i got and don't know how to deal with it after spending days. Some says that MAC OS uses different values as kern.aiomax instead fs.aio-max-nr but kern.aiomax is not able to take value as large as fs.aio-max-nr=1048576 which is necessary to resolve this error.
HELP ME EXPERTS.
DEVICE:
MacBook Pro
Intel Chip
MAC OS VENTURA Version 13.6.4
Just downgrade ScyllaDb to 4.5.0.
instead of creating all nodes (in your case Node_X, Node_Y, Node_Z) with 5.2.0 use 4.5.0
It will run your three nodes but NOT MORE THEN THREE NODES OR MULTIPLE DATA Centers ON MAC OS.