Setup three Geth private network nodes(Clique) via docker container. They are deployed on three different VMs. Among 3 nodes, there is a miner node to mine the block, a RPC node to accept RPC request, and a boot peer node to connect both of them. This boot node connect to miner and RPC node peer by peer while RPC and miner node has no directly connection.
Issue found: Sometimes when there are transactions, they are stuck in pending state and never broadcast. This situation happens randomly, sometimes the transactions can be processed by miner but sometimes they just stuck in pending transaction forever, and restart the Geth node does not work at all.
There is no problem with the gas and the nonce. Also, if I connect my RPC node to miner node by admin.addPeer() manually, which make this 3-node private network fully-connected, then this situation totally disappear and everything become normal, which, the transactions can be processed by miner. The question seems to be the RPC node sometimes does not broadcast my transactions if it does not connect to a miner directly.
Another situation worth mentioning: the blocks can sync within the private network, get the same number when eth.blockNumber, but sometimes RPC node may sync slow, for example the miner has now mined 100 blocks, and the RPC may only sync to 80 blocks while eth.syncing returns false, although finally it will catch up with the block, it sometimes lag behind the miner.
System information
Geth version: Geth/v1.11.6
OS & Version: Ubuntu 22.04
Miner start commands:
--nodekey=miner.key --bootnodes="enode://${BOOTNODE_KEY}@${BOOTNODE_IP}:${BOOTNODE_PORT}" --syncmode=full --gcmode=archive --miner.etherbase=${ADDR} --mine --miner.threads=1 --networkid=${NETWORK_ID} --unlock=${ADDR} --password=password --allow-insecure-unlock --verbosity=5
RPC start commands:
--nodekey=rpc.key --bootnodes="enode://${BOOTNODE_KEY}@${BOOTNODE_IP}:${BOOTNODE_PORT}" --syncmode=full --gcmode=archive --networkid=${NETWORK_ID} --http --http.vhosts="" --http.addr="0.0.0.0" --http.port=8545 --http.corsdomain=""
Expected behaviour Transactions are successfully broadcasted and mined.
Actual behaviour
Transaction sometime not broadcast within the private network.
txpool of RPC
txpool.inspect { pending: { 0x55FAac070a63F809eDF68D72856Ea88c183171C8: { 0: "contract creation: 0 wei + 5171633 gas × 2000000000 wei", 1: "contract creation: 0 wei + 4398916 gas × 2000000000 wei", 2: "contract creation: 0 wei + 5131938 gas × 2000000000 wei" } }, queued: {} }
txpool of miner and boot node
txpool.inspect { pending: {}, queued: {} }
I have posted the same issue on Github issue: https://github.com/ethereum/go-ethereum/issues/28518 Also, I think my issue is very similar to https://github.com/ethereum/go-ethereum/issues/22308#issuecomment-781940794
Transactions are successfully broadcasted and mined.