Own hosted openrouteservice instance not working

Question

I have my own hosted instance of openrouteservice using their Docker image (https://github.com/GIScience/openrouteservice). I downloaded the .pbf file of the Netherlands from https://download.geofabrik.de/europe/netherlands.html, and I set to use this file in the docker-compose file.

When I run a simple query on this instance, like:

url:
http://my-domain/ors/v2/directions/driving-car/json?
Headers:
{
  "headers": {
    "User-Agent": "ORSClientPython.v2.2.3",
    "Content-type": "application/json",
    "Authorization": null
  },
  "timeout": 60,
  "json": {
    "coordinates": [
      [
        4.451162,
        51.926484
      ],
      [
        4.301373,
        52.049112
      ],
      [
        4.528832,
        52.06009
      ],
      [
        4.322566,
        52.057557
      ],
      [
        4.256713,
        52.071065
      ],
      [
        4.223623,
        52.024614
      ],
      [
        4.451162,
        51.926484
      ]
    ]
  }
}

I get the response Could not find point 0: 4.4511620 51.9264840 within a radius of 350.0 meters. While, when I run this exact same query, I do get a succesful response.

Does anyone know what is going wrong here?