We have upgraded our database server from MySQL 5.1 to MariaDB 5.5 (5.5.40-MariaDB-1~wheezy-log).
After this upgrade, some long running processes mysql connection si being dropped. Common scenario for those processes is:
- Connect to MySQL
- Run some queries
- Do some heavy lifting without connecting to MySQL for at least one minute
- Try to query against the original connection
- Exception giving a 2600 error - MySQL server has gone away
This does happen in PHP CLI scripts (php 5.3), but also in a Ruby application (Redmine 2.5.1). It was not happening with MySQL 5.1 and there were no changes on the applications side, so it should not be app-related.
The %timeout% variables in MariaDB are:
+----------------------------+----------+
| Variable_name | Value |
+----------------------------+----------+
| connect_timeout | 5 |
| deadlock_timeout_long | 50000000 |
| deadlock_timeout_short | 10000 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| slave_net_timeout | 3600 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
+----------------------------+----------+
We are not using thread pooling:
+---------------------------+---------------------------+
| Variable_name | Value |
+---------------------------+---------------------------+
| thread_cache_size | 128 |
| thread_concurrency | 10 |
| thread_handling | one-thread-per-connection |
| thread_pool_idle_timeout | 60 |
| thread_pool_max_threads | 500 |
| thread_pool_oversubscribe | 3 |
| thread_pool_size | 12 |
| thread_pool_stall_limit | 500 |
| thread_stack | 294912 |
+---------------------------+---------------------------+
When the thing happens, there is also an event logged in syslog, everytime looking the same:
Dec 16 13:00:14 DB01 mysqld: 141216 13:00:14 [Warning] Aborted connection 9202885 to db: 'some_db_name' user: 'user' host: 'app' (Unknown error)
Besides that, there are also weird root account disconnection messages:
Dec 16 13:05:02 DB01 mysqld: 141216 13:05:02 [Warning] Aborted connection 9225621 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:10:00 DB01 mysqld: 141216 13:10:00 [Warning] Aborted connection 9218291 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:10:12 DB01 mysqld: 141216 13:10:12 [Warning] Aborted connection 9232561 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:17:01 DB01 /USR/SBIN/CRON[41343]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 16 13:20:02 DB01 mysqld: 141216 13:20:02 [Warning] Aborted connection 9248777 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:20:02 DB01 mysqld: 141216 13:20:02 [Warning] Aborted connection 9248788 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:20:12 DB01 mysqld: 141216 13:20:12 [Warning] Aborted connection 9248798 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Out of those settings is there any, that should be changed to fix the weird server has gone away errors?
In the end we have found out that the DB is not the cause of exceptions dropping, as the drops are appearing also in other unrelated systems.