I've recently encountered an issue with my applications hosted on Fly.io. Initially, all my applications were running smoothly in the Chennai (MAA) region. However, I received a notification from Fly.io stating that the MAA region would be deprecated and my applications would be transferred to the Mumbai (BOM) region.
On Monday, following the region transfer, my database applications were suspended unexpectedly. The logs indicate a system restart followed by a series of actions leading to the shutdown of various processes, including the database server, proxy, and supervisor processes (logs provided below for reference). Notably, timestamps in the logs are missing, displaying "Invalid date".
Invalid date
[5313503.257433] reboot: Restarting system
Invalid date
Umounting /dev/vdb from /data
Invalid date
Starting clean up.
Invalid date
keeper | Process exited 0
Invalid date
keeper | server stopped
Invalid date
keeper | done
Invalid date
keeper | waiting for server to shut down....2024-03-25 17:45:46.925 UTC [588] LOG: database system is shut down
Invalid date
keeper | 2024-03-25 17:45:46.916 UTC [590] LOG: shutting down
Invalid date
keeper | 2024-03-25 17:45:46.916 UTC [588] LOG: background worker "logical replication launcher" (PID 595) exited with exit code 1
Invalid date
keeper | 2024-03-25 17:45:46.913 UTC [588] LOG: aborting any active transactions
Invalid date
proxy | exit status 130
Invalid date
keeper | 2024-03-25 17:45:46.911 UTC [588] LOG: received fast shutdown request
Invalid date
proxy | [WARNING] 084/174546 (538) : All workers exited. Exiting... (130)
Invalid date
proxy | [ALERT] 084/174546 (538) : Current worker #1 (561) exited with code 130 (Interrupt)
Invalid date
proxy | [NOTICE] 084/174546 (538) : path to executable is /usr/sbin/haproxy
Invalid date
proxy | [NOTICE] 084/174546 (538) : haproxy version is 2.2.9-2+deb11u3
Invalid date
sentinel | Process exited 0
Invalid date
exporter | signal: interrupt
Invalid date
proxy | [WARNING] 084/174546 (538) : Exiting Master process...
Invalid date
proxy | Stopping interrupt...
Invalid date
sentinel | Stopping interrupt...
Invalid date
keeper | Stopping interrupt...
Invalid date
exporter | Stopping interrupt...
Invalid date
supervisor stopping
Invalid date
Got terminated, stopping
Invalid date
Sending signal SIGTERM to main child process w/ PID 521
Upon investigation, I noticed that while my volume appears to be intact in the MAA region, there are no machines listed for my application. This has left me unable to restart my applications and I am particularly concerned about potential data loss.
Has anyone faced a similar situation, or can anyone provide guidance on how to restart my applications in the BOM region without losing any data? Any help or pointers towards relevant documentation or steps to follow would be greatly appreciated.