IPFS not announcing CIDs on indirect peers and unable to get through cmd as CID is not propagated using DHT

Question

I have four nodes NodeB, NodeA, Node0, and Node3

NodeB is peer of NodeA,
NodeA is peer of NodeB and Node0
Node0 is a peer of NodeA and Node3
Node3 is a peer of Node3

When I upload file on NodeB it is accessible on NodeA but not on Node0 and Node3 as these nodes are not directly added as peer but have same swarm key. what might be the reason? I have configured all nodes as dhtserver and turned the dht server check to “active” but its not working.

When we get content from direct peer node, content is retrieved but for indirect peer the content is not accessible directly.

we have done the following

1. Updated config with Routing.Type: dhtserver and dhtclient
2. Updated config with DHT.Server: true
3. Updated config with Tested with AccelaratedDHTClient: true
4. manually announced the CID using ipfs dht provide <CID> then tried to fetch it using ipfs dht findprovs <CID> but no peer was returned.
5. Tried annoucing using ipfs dht provide -r <CID>
6. Tried with ipfs bitswap reprovide** (command did not work)

All the nodes are connected through public IP and accessible publicly and the swarm key is same.

[ NodeB <--> NodeA ] <==>[ Node0 <--> Node3 ]

The nodes are hosted on azure cloud services:

Version

ipfs version 0.25.0

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/ip4/__.__.100.178/tcp/4001/p2p/12D3KooJLq6rP69XfYWJkkK_____"
  ],
  "DHT": {
    "Server": true
  },
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "12D3KooWAAUpb6sUWUkdGngrMTb4QgjokRVxWghnG5FeG7QqJjMz"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null,
    "Type": "dhtserver"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {}
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

NodeB and NodeA are in virtual network 1. Node0 and Node3 are in the virual network 2.

NodeB:

ipfs stats dht

DHT wan (0 peers):
  Bucket  0 (0 peers) - refreshed never:
    Peer                                  last useful  last queried  Agent Version
Error: no LAN client found

ipfs swarm peers

/ip4/__.__.99.165/tcp/4001/p2p/12D3KooWSsdB2mJyQkq____

NodeA

ipfs stats dht

root@nodeA:/home/ssiadmin# ipfs stats dht
DHT wan (0 peers):
  Bucket  0 (0 peers) - refreshed never:
    Peer                                  last useful  last queried  Agent Version

Error: no LAN client found

ipfs swarm peers

root@nodeA:/home/ssiadmin# ipfs swarm peers
/ip4/__.__.99.199/tcp/4001/p2p/12D3KooWDpY9uHAEDxd____
root@nodeA:/home/ssiadmin#

Node0

ipfs stats dht

root@node0:/home/ssiadmin# ipfs stats dht
    DHT wan (0 peers):
      Bucket  0 (0 peers) - refreshed never:
        Peer                                  last useful  last queried  Agent Version

    Error: no LAN client found

ipfs swarm peers

root@node0:/home/ssiadmin# ipfs swarm peers
/ip4/__.__.98.222/tcp/4001/p2p/12D3KooWLPM8shRFvonSnn____
/ip4/__.__.53.99/tcp/4001/p2p/12D3KooWNy35zuZFaqu3QXa____
root@node0:/home/ssiadmin#

Node3

ipfs stats dht

root@node3:/home/ssiadmin# ipfs stats dht
DHT wan (0 peers):
  Bucket  0 (0 peers) - refreshed never:
    Peer                                  last useful  last queried  Agent Version

Error: no LAN client found

ipfs swarm peers

root@node3:/home/ssiadmin# ipfs swarm peers

/ip4/__.__.98.178/tcp/4001/p2p/12D3KooWETehntUtx6i63HBvrme____
root@node3:/home/ssiadmin#