What is best practice to test error responses with Dredd?

Question

I want to describe my existing API with openapi 3 and prove my description with dredd. I know openapi 3 implementation is experimental, but I don't use any of the elements which are not supported yet.

This is part of my spec.yaml

paths:
  /login:
    post:
      summary: Login
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/login_request'
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/login_response'
        "401":
          description: Not authenticated
        "404":
          description: Other Error

...

  schemas:
    login_request:
      type: object
      properties:
        n:
          type: string
          description: Username
          example: super
        p:
          type: string
          description: Password
          example: user123

So I can test the case of correct entered credentials (assumed the test user is there), because they are given as example in the spec. But How about the error cases? Of course I could skip them in the hooks.js file:

var hooks = require('hooks');

hooks.before('/login > Login > 401', function (transaction, done) {
    transaction.skip = true;
    done();
});

hooks.before('/login > Login > 404', function (transaction, done) {
    transaction.skip = true;
    done();
});

Note: In my case, maybe due to the use of openapi 3, Dredd behaves contradictory to what is said here: Error responses are not automatically skipped.

But is that desirable? I would rather prefer to test correct error reactions as well. But how? I doubt it's supported right now, but at least in theory I could insert multiple examples to the spec and thereby include examples leading to errors as well. But on the other hand, arbitrary false examples would not be anything to be contained in a documentation, wouldn't it?

What is best practice here?

Answer 1

The area you are asking about is documented in the Dredd docs, but only for the API Blueprint and OpenAPI 2 formats:

The OpenAPI 2 format allows to specify multiple responses for a single operation. By default Dredd tests only responses with 2xx status codes. Responses with other codes are marked as skipped and can be activated in hooks - see the Multiple Requests and Responses how-to guide.

However, for OpenAPI 3, this is undefined, untested and undocumented. There is a GitHub issue about the problem and until it's closed, I believe your question cannot be answered.

The issue about future of Dredd has "make OpenAPI 3 a first-class citizen in Dredd" declared as one of the priorities, so this is just a matter of time.