I have got an issue that occurs when i deploy my project using docker swarm, the creation of a new object in world state database happens fine but while updating that object sometimes a mvcc error occurs. When i checked the logs it shows that the database does not exist, I feel the issue might be because of peer communication and batch timeout (this is my guess). I am using 1.4.6 version of fabric. The same scenario, i had tried to recreate while deploying using docker compose but it doesn't
Configtx.yaml
Organizations:
- &OrdererOrg
Name: OrdererMSP
ID: OrdererMSP
MSPDir: crypto-config/ordererOrganizations/xxxx.com/msp
Policies:
Readers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Writers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Admins:
Type: Signature
Rule: "OR('OrdererMSP.admin')"
- &Org1
Name: xxxxMSP
ID: xxxxMSP
MSPDir: crypto-config/peerOrganizations/xxxx.xxxx.com/msp
Policies:
Readers:
Type: Signature
Rule: "OR('xxxxMSP.admin', 'xxxxMSP.member')"
Writers:
Type: Signature
Rule: "OR('xxxxMSP.admin', 'xxxxMSP.member')"
Admins:
Type: Signature
Rule: "OR('xxxxMSP.admin')"
AnchorPeers:
- Host: peer0.xxxx.xxxx.com
Port: 7051
- &Org2
Name: xxxxMSP
ID: xxxxMSP
MSPDir: crypto-config/peerOrganizations/xxxx.xxxx.com/msp
Policies:
Readers:
Type: Signature
Rule: "OR('xxxxMSP.admin', 'xxxxMSP.member')"
Writers:
Type: Signature
Rule: "OR('xxxxMSP.admin', 'xxxxMSP.member')"
Admins:
Type: Signature
Rule: "OR('xxxxMSP.admin')"
AnchorPeers:
- Host: peer0.xxxx.xxxx.com
Port: 9051
Capabilities: Channel: &ChannelCapabilities V1_4_3: true V1_3: false V1_1: false Orderer: &OrdererCapabilities V1_4_2: true V1_1: false Application: &ApplicationCapabilities V1_4_2: true V1_3: false V1_2: false V1_1: false Application: &ApplicationDefaults Organizations: Policies: Readers: Type: ImplicitMeta Rule: "ANY Readers" Writers: Type: ImplicitMeta Rule: "ANY Writers" Admins: Type: ImplicitMeta Rule: "MAJORITY Admins" Capabilities: <<: *ApplicationCapabilities Orderer: &OrdererDefaults OrdererType: etcdraft Addresses: - orderer0.xxxx.com:7050 - orderer1.xxxx.com:7050 - orderer2.xxxx.com:7050 BatchTimeout: 2s BatchSize: MaxMessageCount: 10 AbsoluteMaxBytes: 99 MB PreferredMaxBytes: 512 KB EtcdRaft: Consenters: - Host: orderer0.xxxx.com Port: 7050 ClientTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer0.xxxx.com/tls/server.crt ServerTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer0.xxxx.com/tls/server.crt - Host: orderer1.xxxx.com Port: 7050 ClientTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer1.xxxx.com/tls/server.crt ServerTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer1.xxxx.com/tls/server.crt - Host: orderer2.xxxx.com Port: 7050 ClientTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer2.xxxx.com/tls/server.crt ServerTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer2.xxxx.com/tls/server.crt Organizations: Policies: Readers: Type: ImplicitMeta Rule: "ANY Readers" Writers: Type: ImplicitMeta Rule: "ANY Writers" Admins: Type: ImplicitMeta Rule: "MAJORITY Admins" BlockValidation: Type: ImplicitMeta Rule: "ANY Writers" Channel: &ChannelDefaults Policies: Readers: Type: ImplicitMeta Rule: "ANY Readers" Writers: Type: ImplicitMeta Rule: "ANY Writers" Admins: Type: ImplicitMeta Rule: "MAJORITY Admins" Capabilities: <<: *ChannelCapabilities Profiles: xxxxGenesis: <<: *ChannelDefaults Orderer: <<: *OrdererDefaults Organizations: - *OrdererOrg Capabilities: <<: *OrdererCapabilities Consortiums: SampleConsortium: Organizations: - *Org1 - *Org2 CV_Channel: Consortium: SampleConsortium <<: *ChannelDefaults Application: <<: *ApplicationDefaults Organizations: - *Org1 - *Org2 Capabilities: <<: *ApplicationCapabilities
Peer container logs
{"log":"\u001b[34m2020-09-15 08:03:37.496 UTC [couchdb] CreateDatabaseIfNotExist -\u003e INFO 02b\u001b[0m Created state database cvchannel_\n","stream":"stderr","time":"2020-09-15T08:03:37.497361704Z"} {"log":"\u001b[31m2020-09-15 08:03:37.674 UTC [couchdb] runWarmIndexAllIndexes -\u003e ERRO 02c\u001b[0m Error detected during WarmIndexAllIndexes(): error handling CouchDB request. Error:not_found, Status Code:404, Reason:Database does not exist.\n","stream":"stderr","time":"2020-09-15T08:03:37.674619911Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchInstance).handleRequest\n","stream":"stderr","time":"2020-09-15T08:03:37.674649981Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1847\n","stream":"stderr","time":"2020-09-15T08:03:37.674656059Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).handleRequest\n","stream":"stderr","time":"2020-09-15T08:03:37.674661185Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1650\n","stream":"stderr","time":"2020-09-15T08:03:37.674666013Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).ListIndex\n","stream":"stderr","time":"2020-09-15T08:03:37.674670852Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1151\n","stream":"stderr","time":"2020-09-15T08:03:37.674675596Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).WarmIndexAllIndexes\n","stream":"stderr","time":"2020-09-15T08:03:37.674680273Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1324\n","stream":"stderr","time":"2020-09-15T08:03:37.674684939Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).runWarmIndexAllIndexes\n","stream":"stderr","time":"2020-09-15T08:03:37.674689595Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1311\n","stream":"stderr","time":"2020-09-15T08:03:37.674708559Z"} {"log":"runtime.goexit\n","stream":"stderr","time":"2020-09-15T08:03:37.674713109Z"} {"log":"\u0009/opt/go/src/runtime/asm_amd64.s:1337\n","stream":"stderr","time":"2020-09-15T08:03:37.674717108Z"} {"log":"\u001b[34m2020-09-15 08:03:37.692 UTC [kvledger] CommitWithPvtData -\u003e INFO 02d\u001b[0m [cvchannel] Committed block [0] with 1 transaction(s) in 172ms (state_validation=1ms block_and_pvtdata_commit=64ms state_commit=94ms) commitHash=[]\n","stream":"stderr","time":"2020-09-15T08:03:37.693007188Z"} {"log":"\u001b[34m2020-09-15 08:03:37.695 UTC [ledgermgmt] CreateLedger -\u003e INFO 02e\u001b[0m Created ledger [cvchannel] with genesis block\n","stream":"stderr","time":"2020-09-15T08:03:37.695884567Z"} {"log":"\u001b[34m2020-09-15 08:03:37.712 UTC [gossip.gossip] JoinChan -\u003e INFO 02f\u001b[0m Joining gossip network of channel cvchannel with 2 organizations\n","stream":"stderr","time":"2020-09-15T08:03:37.712518745Z"} {"log":"\u001b[34m2020-09-15 08:03:37.712 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 030\u001b[0m No configured anchor peers of xxxxMSP for channel cvchannel to learn about\n","stream":"stderr","time":"2020-09-15T08:03:37.712607056Z"} {"log":"\u001b[34m2020-09-15 08:03:37.712 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 031\u001b[0m No configured anchor peers of xxxxMSP for channel cvchannel to learn about\n","stream":"stderr","time":"2020-09-15T08:03:37.712680067Z"} {"log":"\u001b[34m2020-09-15 08:03:37.740 UTC [gossip.state] NewGossipStateProvider -\u003e INFO 032\u001b[0m Updating metadata information for channel cvchannel, current ledger sequence is at = 0, next expected block is = 1\n","stream":"stderr","time":"2020-09-15T08:03:37.741210227Z"} {"log":"\u001b[34m2020-09-15 08:03:37.742 UTC [sccapi] deploySysCC -\u003e INFO 033\u001b[0m system chaincode lscc/cvchannel(github.com/hyperledger/fabric/core/scc/lscc) deployed\n","stream":"stderr","time":"2020-09-15T08:03:37.743086696Z"} {"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [cscc] Init -\u003e INFO 034\u001b[0m Init CSCC\n","stream":"stderr","time":"2020-09-15T08:03:37.74339807Z"} {"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [sccapi] deploySysCC -\u003e INFO 035\u001b[0m system chaincode cscc/cvchannel(github.com/hyperledger/fabric/core/scc/cscc) deployed\n","stream":"stderr","time":"2020-09-15T08:03:37.743424102Z"} {"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [qscc] Init -\u003e INFO 036\u001b[0m Init QSCC\n","stream":"stderr","time":"2020-09-15T08:03:37.743544721Z"} {"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [sccapi] deploySysCC -\u003e INFO 037\u001b[0m system chaincode qscc/cvchannel(github.com/hyperledger/fabric/core/scc/qscc) deployed\n","stream":"stderr","time":"2020-09-15T08:03:37.743558369Z"} {"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [sccapi] deploySysCC -\u003e INFO 038\u001b[0m system chaincode (+lifecycle,github.com/hyperledger/fabric/core/chaincode/lifecycle) disabled\n","stream":"stderr","time":"2020-09-15T08:03:37.743790508Z"} {"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [endorser] callChaincode -\u003e INFO 039\u001b[0m [][a3cc0bd2] Exit chaincode: name:"cscc" (431ms)\n","stream":"stderr","time":"2020-09-15T08:03:37.744038661Z"} {"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [comm.grpc.server] 1 -\u003e INFO 03a\u001b[0m unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=10.0.4.4:33316 grpc.code=OK grpc.call_duration=435.114623ms\n","stream":"stderr","time":"2020-09-15T08:03:37.744207412Z"} {"log":"\u001b[34m2020-09-15 08:03:43.750 UTC [gossip.election] beLeader -\u003e INFO 03b\u001b[0m 5fc3148715eaea54bba5cae87f301cbeca436e2d8842d7e5b9469c5b8398d314 : Becoming a leader\n","stream":"stderr","time":"2020-09-15T08:03:43.751403096Z"} {"log":"\u001b[34m2020-09-15 08:03:43.751 UTC [gossip.service] func1 -\u003e INFO 03c\u001b[0m Elected as a leader, starting delivery service for channel cvchannel\n","stream":"stderr","time":"2020-09-15T08:03:43.751480343Z"} {"log":"\u001b[34m2020-09-15 08:03:43.752 UTC [deliveryClient] StartDeliverForChannel -\u003e INFO 03d\u001b[0m This peer will retrieve blocks from ordering service and disseminate to other peers in the organization for channel cvchannel\n","stream":"stderr","time":"2020-09-15T08:03:43.752972274Z"} {"log":"\u001b[34m2020-09-15 08:03:43.765 UTC [deliveryClient] RequestBlocks -\u003e INFO 03e\u001b[0m Starting deliver with block [1] for channel cvchannel\n","stream":"stderr","time":"2020-09-15T08:03:43.765848602Z"} {"log":"\u001b[34m2020-09-15 08:03:43.774 UTC [gossip.privdata] StoreBlock -\u003e INFO 03f\u001b[0m [cvchannel] Received block [1] from buffer\n","stream":"stderr","time":"2020-09-15T08:03:43.774744553Z"} {"log":"\u001b[34m2020-09-15 08:03:43.782 UTC [gossip.gossip] JoinChan -\u003e INFO 040\u001b[0m Joining gossip network of channel cvchannel with 2 organizations\n","stream":"stderr","time":"2020-09-15T08:03:43.782642453Z"} {"log":"\u001b[34m2020-09-15 08:03:43.782 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 041\u001b[0m No configured anchor peers of xxxxMSP for channel cvchannel to learn about\n","stream":"stderr","time":"2020-09-15T08:03:43.782658448Z"} {"log":"\u001b[34m2020-09-15 08:03:43.782 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 042\u001b[0m Learning about the configured anchor peers of xxxxMSP for channel cvchannel : [{peer0.xxxx.xxxx.com 7051}]\n","stream":"stderr","time":"2020-09-15T08:03:43.782662723Z"} {"log":"\u001b[34m2020-09-15 08:03:43.782 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 043\u001b[0m Anchor peer with same endpoint, skipping connecting to myself\n","stream":"stderr","time":"2020-09-15T08:03:43.782666538Z"} {"log":"\u001b[34m2020-09-15 08:03:43.783 UTC [committer.txvalidator] Validate -\u003e INFO 044\u001b[0m [cvchannel] Validated block [1] in 9ms\n","stream":"stderr","time":"2020-09-15T08:03:43.784169783Z"} {"log":"\u001b[31m2020-09-15 08:03:43.845 UTC [couchdb] runWarmIndexAllIndexes -\u003e ERRO 045\u001b[0m Error detected during WarmIndexAllIndexes(): error handling CouchDB request. Error:not_found, Status Code:404, Reason:Database does not exist.\n","stream":"stderr","time":"2020-09-15T08:03:43.845975731Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchInstance).handleRequest\n","stream":"stderr","time":"2020-09-15T08:03:43.846007175Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1847\n","stream":"stderr","time":"2020-09-15T08:03:43.846012195Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).handleRequest\n","stream":"stderr","time":"2020-09-15T08:03:43.846016704Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1650\n","stream":"stderr","time":"2020-09-15T08:03:43.846020817Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).ListIndex\n","stream":"stderr","time":"2020-09-15T08:03:43.846024956Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1151\n","stream":"stderr","time":"2020-09-15T08:03:43.846028939Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).WarmIndexAllIndexes\n","stream":"stderr","time":"2020-09-15T08:03:43.846033069Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1324\n","stream":"stderr","time":"2020-09-15T08:03:43.846037135Z"} {"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).runWarmIndexAllIndexes\n","stream":"stderr","time":"2020-09-15T08:03:43.846041182Z"} {"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1311\n","stream":"stderr","time":"2020-09-15T08:03:43.846045266Z"} {"log":"runtime.goexit\n","stream":"stderr","time":"2020-09-15T08:03:43.846049541Z"} {"log":"\u0009/opt/go/src/runtime/asm_amd64.s:1337\n","stream":"stderr","time":"2020-09-15T08:03:43.846053454Z"} {"log":"\u001b[34m2020-09-15 08:03:43.865 UTC [kvledger] CommitWithPvtData -\u003e INFO 046\u001b[0m [cvchannel] Committed block [1] with 1 transaction(s) in 81ms (state_validation=0ms block_and_pvtdata_commit=29ms state_commit=39ms) commitHash=[47dc540c94ceb704a23875c11273e16bb0b8a87aed84de911f2133568115f254]\n","stream":"stderr","time":"2020-09-15T08:03:43.865566497Z"} {"log":"\u001b[34m2020-09-15 08:03:43.946 UTC [endorser] callChaincode -\u003e INFO 047\u001b[0m [][29451530] Entry chaincode: name:"lscc" \n","stream":"stderr","time":"2020-09-15T08:03:43.94617674Z"} {"log":"\u001b[34m2020-09-15 08:03:44.017 UTC [couchdb] CreateDatabaseIfNotExist -\u003e INFO 048\u001b[0m Created state database cvchannel_lscc\n","stream":"stderr","time":"2020-09-15T08:03:44.018004477Z"} {"log":"\u001b[34m2020-09-15 08:03:44.020 UTC [lscc] executeInstall -\u003e INFO 049\u001b[0m Installed Chaincode [certcc] Version [1.0] to peer\n","stream":"stderr","time":"2020-09-15T08:03:44.020291974Z"} {"log":"\u001b[34m2020-09-15 08:03:44.020 UTC [endorser] callChaincode -\u003e INFO 04a\u001b[0m [][29451530] Exit chaincode: name:"lscc" (74ms)\n","stream":"stderr","time":"2020-09-15T08:03:44.020511347Z"} {"log":"\u001b[34m2020-09-15 08:03:44.020 UTC [comm.grpc.server] 1 -\u003e INFO 04b\u001b[0m unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=10.0.4.4:33328 grpc.code=OK grpc.call_duration=75.164566ms\n","stream":"stderr","time":"2020-09-15T08:03:44.020765429Z"} {"log":"\u001b[34m2020-09-15 08:04:00.231 UTC [comm.grpc.server] 1 -\u003e INFO 04c\u001b[0m unary call completed grpc.service=gossip.Gossip grpc.method=Ping grpc.request_deadline=2020-09-15T08:04:02.231Z grpc.peer_address=10.0.4.4:57698 grpc.peer_subject="CN=peer1.xxxx.xxxx.com,OU=peer+OU=xxxx,O=Hyperledger,ST=North Carolina,C=US" grpc.code=OK grpc.call_duration=184.503µs\n","stream":"stderr","time":"2020-09-15T08:04:00.231875809Z"} {"log":"\u001b[34m2020-09-15 08:04:00.243 UTC [comm.grpc.server] 1 -\u003e INFO 04d\u001b[0m streaming call completed grpc.service=gossip.Gossip grpc.method=GossipStream grpc.request_deadline=2020-09-15T08:04:10.233Z grpc.peer_address=10.0.4.4:57698 grpc.peer_subject="CN=peer1.xxxx.xxxx.com,OU=peer+OU=xxxx,O=Hyperledger,ST=North Carolina,C=US" error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call_duration=9.822291ms\n","stream":"stderr","time":"2020-09-15T08:04:00.243456691Z"}