can't to start clickhouse service after restart

Question

I was working with ckickhouse and running query but my clickhouse hanged!. I restarted the server but the clickhouse service is not running and produces the following log.

2024.03.10 16:15:23.798502 [ 110215 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
2024.03.10 16:15:23.798552 [ 110215 ] {} <Fatal> BaseDaemon: (version 23.12.2.59 (official build), build id: 7F4C1A822F9C67A4D137A58F9A95BD4B0F1B6A8A, git hash: 17ab210e76170917215be8cc3ae07d465a341d42) (from thread 110196) Received signal
 6
2024.03.10 16:15:23.798584 [ 110215 ] {} <Fatal> BaseDaemon: Signal description: Aborted
2024.03.10 16:15:23.798591 [ 110215 ] {} <Fatal> BaseDaemon:
2024.03.10 16:15:23.798608 [ 110215 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007fa96aaa800b 0x00007fa96aa87859 0x0000000016eacf24 0x0000000016eca40d 0x000000000c9786e8 0x0000000012de0bb3 0x0000000012e083e7 0x0000000012eb2a92 0x00000000109
cab4d 0x00000000109dd644 0x00000000109e1117 0x00000000109d847e 0x00000000109ef28c 0x000000000c7c00d8 0x00007fa96ac5f609 0x00007fa96ab84353
2024.03.10 16:15:23.798623 [ 110215 ] {} <Fatal> BaseDaemon: ########################################
2024.03.10 16:15:23.798634 [ 110215 ] {} <Fatal> BaseDaemon: (version 23.12.2.59 (official build), build id: 7F4C1A822F9C67A4D137A58F9A95BD4B0F1B6A8A, git hash: 17ab210e76170917215be8cc3ae07d465a341d42) (from thread 110196) (no query) Rece
ived signal Aborted (6)
2024.03.10 16:15:23.798640 [ 110215 ] {} <Fatal> BaseDaemon:
2024.03.10 16:15:23.798646 [ 110215 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007fa96aaa800b 0x00007fa96aa87859 0x0000000016eacf24 0x0000000016eca40d 0x000000000c9786e8 0x0000000012de0bb3 0x0000000012e083e7 0x0000000012eb2a92 0x00000000109
cab4d 0x00000000109dd644 0x00000000109e1117 0x00000000109d847e 0x00000000109ef28c 0x000000000c7c00d8 0x00007fa96ac5f609 0x00007fa96ab84353
2024.03.10 16:15:23.798659 [ 110215 ] {} <Fatal> BaseDaemon: 2. ? @ 0x00007fa96aaa800b in ?
2024.03.10 16:15:23.798665 [ 110215 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00007fa96aa87859 in ?
2024.03.10 16:15:23.798703 [ 110215 ] {} <Fatal> BaseDaemon: 4. ? @ 0x0000000016eacf24 in /usr/bin/clickhouse
2024.03.10 16:15:23.798712 [ 110215 ] {} <Fatal> BaseDaemon: 5. ? @ 0x0000000016eca40d in /usr/bin/clickhouse
2024.03.10 16:15:23.798764 [ 110215 ] {} <Fatal> BaseDaemon: 6. DB::IAST::formatImpl(DB::IAST::FormatSettings const&, DB::IAST::FormatState&, DB::IAST::FormatStateStacked) const @ 0x000000000c9786e8 in /usr/bin/clickhouse
2024.03.10 16:15:23.798784 [ 110215 ] {} <Fatal> BaseDaemon: 7. DB::ASTCreateQuery::formatQueryImpl(DB::IAST::FormatSettings const&, DB::IAST::FormatState&, DB::IAST::FormatStateStacked) const @ 0x0000000012de0bb3 in /usr/bin/clickhouse
2024.03.10 16:15:23.798796 [ 110215 ] {} <Fatal> BaseDaemon: 8. DB::ASTQueryWithOutput::formatImpl(DB::IAST::FormatSettings const&, DB::IAST::FormatState&, DB::IAST::FormatStateStacked) const @ 0x0000000012e083e7 in /usr/bin/clickhouse
2024.03.10 16:15:23.798812 [ 110215 ] {} <Fatal> BaseDaemon: 9. DB::serializeAST(DB::IAST const&) @ 0x0000000012eb2a92 in /usr/bin/clickhouse
2024.03.10 16:15:23.798835 [ 110215 ] {} <Fatal> BaseDaemon: 10. DB::DDLTaskBase::formatRewrittenQuery(std::shared_ptr<DB::Context const>) @ 0x00000000109cab4d in /usr/bin/clickhouse
2024.03.10 16:15:23.798852 [ 110215 ] {} <Fatal> BaseDaemon: 11. DB::DDLWorker::initAndCheckTask(String const&, String&, std::shared_ptr<zkutil::ZooKeeper> const&) @ 0x00000000109dd644 in /usr/bin/clickhouse
2024.03.10 16:15:23.798867 [ 110215 ] {} <Fatal> BaseDaemon: 12. DB::DDLWorker::scheduleTasks(bool) @ 0x00000000109e1117 in /usr/bin/clickhouse
2024.03.10 16:15:23.798881 [ 110215 ] {} <Fatal> BaseDaemon: 13. DB::DDLWorker::runMainThread() @ 0x00000000109d847e in /usr/bin/clickhouse
2024.03.10 16:15:23.798907 [ 110215 ] {} <Fatal> BaseDaemon: 14. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorke
r::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000109ef28c in /usr/bin/clickhouse
2024.03.10 16:15:23.798933 [ 110215 ] {} <Fatal> BaseDaemon: 15. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleI
mpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c7c00d8 in /usr/bin/clickhouse
2024.03.10 16:15:23.798940 [ 110215 ] {} <Fatal> BaseDaemon: 16. ? @ 0x00007fa96ac5f609 in ?
2024.03.10 16:15:23.798951 [ 110215 ] {} <Fatal> BaseDaemon: 17. ? @ 0x00007fa96ab84353 in ?
2024.03.10 16:15:24.005274 [ 110215 ] {} <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: AC32B04C37311E5E413148969FBCE1EA)
2024.03.10 16:15:24.005582 [ 110215 ] {} <Fatal> BaseDaemon: Report this error to https://github.com/ClickHouse/ClickHouse/issues
2024.03.10 16:15:30.031478 [ 109467 ] {} <Fatal> Application: Child process was terminated by signal 6.

Answer 1

according to

DB::ASTQueryWithOutput::formatImpl
DB::DDLTaskBase::formatRewrittenQuery

Looks like you have Distributed DDL query (something with ON CLUSTER) which raise segmentation fault during execution

try to look to

SELECT * FROM system.distributed_ddl_queue FORMAT Vertical

or look to zookeeper (zkCli.sh) in path described in <distributed_ddl> section look details https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings#server-settings-distributed_ddl

something like

SELECT * FROM system.zookeeper WHERE path='/clickhouse/task_queue/ddl' FORMAT Vertical