Swagger response body missing for failure responses when the Success response Media Type is image/png

Question

I use Swagger OAS3 and trying to display an elaborated error message when the endpoint which is supposed to return an image fails. When I set the MIME type of the success response to image/png, the *Response body doesn't appear if an error is thrown. If I change it to application/json, it appears but the actual success response we expect is an image, not a json. So I am not sure whether it's the correct approach!

Response body doesn't appear with the error

If I add both MIME types on the success response, and select json before executing, it appears again. But it never respond to the MIME types I add for the failure responses.

                    final ObjectNode responses = httpMethod.putObject("responses");
                    final ObjectNode success = responses.putObject("200");
                    success.put(DESCRIPTION, "Success");
                    final ObjectNode content = success.putObject("content");

                    final ObjectNode mime = content.putObject(IMAGE_PNG);
                    final ObjectNode schema = mime.putObject(SCHEMA);
                    schema.put("type", "string");
                    schema.put("format", "binary");

                    final ObjectNode mimeTextFail = content.putObject(APPLICATION_JSON);
                    final ObjectNode schemaTextfail = mimeTextFail.putObject(SCHEMA);
                    schemaTextfail.put("type", OBJECT);


                    final ObjectNode fail = responses.putObject("401");
                    fail.put(DESCRIPTION, "Error: Unauthorized");
                    final ObjectNode contentfail = fail.putObject("content");
                    final ObjectNode mimefail = contentfail.putObject(APPLICATION_JSON);
                    final ObjectNode schemafail = mimefail.putObject(SCHEMA);
                    schemafail.put("type", OBJECT);

Is it not possible or am I doing it incorrectly? Any help is really appreciated.

(If I remove all MIME types, the response body appears but the format is html by default and I cannot change it to JSON, other than by specifying the MIME type of successful response to json as stated above.)