Rails Delayed Job keeps retrying failed jobs beyond max_attempts

Question

I'm using DelayedJob as the backend for ActiveJob. I'm running into a strange phenomenon. I want the following job to retry max 3 times. In this example I'm trying to use ActiveJob's retry_on directive with attempts set to 3. I have also set Delayed Job's max_attempts to 1, as the retry_on documentation states the following:

If the exception keeps getting raised beyond the specified number of attempts, the exception is allowed to bubble up to the underlying queuing system, which may have its own retry mechanism or place it in a holding queue for inspection.

Monitoring the progress of the job with delayed_job_web, I can see that the three retries happen correctly. During those retries, the job status is pending. After that it's failed. Then comes the part that I don't understand: in the status failed, the job is scheduled to be retried again!

Here's the code:

class ExecuteWebhookRequestJob < ApplicationJob
  queue_as :default

  include Exceptions

  retry_on WebhookDeliveryFailed, wait: 10.seconds, attempts: 3

  def perform(request, created_callback=nil)
    if request.webhook.enabled?
      begin
        unless request.deliver(created_callback)
          raise WebhookDeliveryFailed.new(nil, request.action, request.request, request.id)
        end
      rescue Exception => e
        raise WebhookDeliveryFailed.new(e.message, request.action, request.request, request.id)
      ensure 
        request.update(attempts: executions)
      end
    end
  end

  def max_attempts
    1
  end
end

I've also tried the following variants:

1. remove the retry_on entirely and set the max_attempts to 3
2. leave the retry_on in, but set attempts to 0 and set the max_attempts to 3
3. set the max_attempts to 0

In all these scenarios the jobs keep getting retried after the three allowed retries.

What am I doing wrong and what is the correct way to approach this?