Why does pact broker cause a timeout error in the database?

118 Views Asked by At

Encountered an issue with our pact broker where the endpoint for /for-verification is returning a 500. Traced the error back to the broker with the following error trace. Any help appreciated.

PactBroker::Errors::ErrorLogger","message":"Error reference HRDcOsuEiZ","exception":{"name":"Sequel::DatabaseError","message":"PG::QueryCanceled: ERROR:  canceling statement due to statement timeout
","stack_trace":["/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:154:in `exec'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:154:in `block in execute_query'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:154:in `execute_query'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:142:in `block in execute'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:119:in `check_disconnect_errors'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:142:in `execute'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:498:in `_execute'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:316:in `block (2 levels) in execute'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:519:in `check_database_errors'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:316:in `block in execute'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/connection_pool/threaded.rb:88:in `hold'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/database/connecting.rb:269:in `synchronize'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:316:in `execute'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/dataset/actions.rb:1142:in `execute'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/adapters/postgres.rb:610:in `fetch_rows'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/dataset/actions.rb:150:in `each'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/dataset/actions.rb:51:in `block in all'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/dataset/actions.rb:1059:in `_all'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.58.0/lib/sequel/dataset/actions.rb:51:in `all'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.102.2/lib/pact_broker/pacts/pacts_for_verification_repository.rb:277:in `remove_non_wip_for_branch'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.102.2/lib/pact_broker/pacts/pacts_for_verification_repository.rb:240:in `find_wip_pact_versions_for_provider_by_provider_branch'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.102.2/lib/pact_broker/pacts/pacts_for_verification_repository.rb:56:in `find_wip'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.102.2/lib/pact_broker/pacts/repository.rb:161:in `find_wip_pact_versions_for_provider'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.102.2/lib/pact_broker/pacts/service.rb:130:in `find_for_verification'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.102.2/lib/pact_broker/api/resources/provider_pacts_for_verification.rb:42:in `pacts'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.102.2/lib/pact_broker/api/resources/provider_pacts_for_verification.rb:60:in `to_json'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.102.2/lib/pact_broker/api/resources/provider_pacts_for_verification.rb:28:in `process_post'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/webmachine-1.6.0/lib/webmachine/decision/flow.rb:428:in `n11'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/webmachine-1.6.0/lib/webmachine/decision/fsm.rb:31:in `block (2 levels) in run'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/webmachine-1.6.0/lib/webmachine/decision/fsm.rb:51:in `handle_exceptions'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/webmachine-1.6.0/lib/webmachine/decision/fsm.rb:31:in `block in run'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/webmachine-1.6.0/lib/webmachine/decision/fsm.rb:29:in `loop'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/webmachine-1.6.0/lib/webmachine/decision/fsm.rb:29:in `run'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/webmachine-1.6.0/lib/webmachine/dispatcher.rb:46:in `block in dispatch'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/as-notifications-1.0.2/lib/as/notifications.rb:161:in `instrument'","/pact_broker/vendor/bundle/ruby/2.7.0/gems/webmachine-1.6.0/lib/webmachine/events.

Tried running verification tests for the provider using the consumer-driven approach. This issue started happening suddenly and no pact tests were commited to the code for a long time.

1

There are 1 best solutions below

1
On

It needs maintenance, it might be the reason that you had too much data.

See how to clean up, you can setup a scheduler to do so.

I had this issue and this helped me.