Blocking Tyk Gateway middleware causing requests to queue up

Question

I'm running Tyk v3.0.1 on a AWS t3a.small node with 2 vCPUs in a Docker container. We have custom logic implemented via custom_middleware_bundle (Python).

I was seeing intermittent errors which I was able boil down to what looks like a "single process / single thread" kind of problem. Basically when I do anything within the Python middleware that is blocking, a long running HTTP request or for demonstration purposes a time.sleep, the entire this and all other APIs using that middleware become unresponsive, with requests to the Tyk Gateway being queued until the blocking operation finishes. Requests to other APIs that don't use a middleware are being served successfully. I know that Python is single threaded. However, according to the documentation, "Tyk will automatically spread itself across all cores to handle traffic" which made me assume that it at least runs middleware in multiple processes being able to offer some kind of concurrency.

I'm running Tyk via /opt/tyk-gateway/tyk --conf=/opt/tyk-gateway/tyk.conf. This is my tyk.conf

{
  "listen_port": 8080,
  "secret": "XXXXX",
  "template_path": "/opt/tyk-gateway/templates",
  "tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
  "middleware_path": "/opt/tyk-gateway/middleware",
  "use_db_app_configs": false,
  "app_path": "/opt/tyk-gateway/apps/",
  "storage": {
    "type": "redis",
    "host": "127.0.0.1",
    "port": 6379,
    "username": "",
    "password": "",
    "database": 0,
    "optimisation_max_idle": 2000,
    "optimisation_max_active": 4000
  },
  "enable_analytics": false,
  "analytics_config": {
    "type": "csv",
    "csv_dir": "/tmp",
    "mongo_url": "",
    "mongo_db_name": "",
    "mongo_collection": "",
    "purge_delay": -1,
    "ignored_ips": []
  },
  "health_check": {
    "enable_health_checks": true,
    "health_check_value_timeouts": 60
  },
  "optimisations_use_async_session_write": true,
  "enable_non_transactional_rate_limiter": true,
  "enable_sentinel_rate_limiter": false,
  "enable_redis_rolling_limiter": false,
  "allow_master_keys": false,
  "policies": {
    "policy_source": "file",
    "policy_record_name": "/opt/tyk-gateway/policies/policies.json"
  },
  "hash_keys": true,
  "close_connections": false,
  "http_server_options": {
    "enable_websockets": true,
    "read_timeout": 300,
    "write_timeout": 300
  },
  "allow_insecure_configs": true,
  "coprocess_options": {
    "enable_coprocess": true,
    "coprocess_grpc_server": "",
    "python_path_prefix": "/opt/tyk-gateway"
  },
  "enable_bundle_downloader": true,
  "bundle_base_url": "http://127.0.0.1:8888",
  "global_session_lifetime": 100,
  "force_global_session_lifetime": false,
  "max_idle_connections_per_host": 500,
  "log_level": "info",
  "proxy_default_timeout": 300
}

Our AWS t3a.small node has 2 vCPUs. Here are the CPU specific Docker settings for that container:

            "CpuShares": 0,
            "NanoCpus": 0,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "CpuCount": 0,
            "CpuPercent": 0,

What can I do to make sure Tyk Gateway can handle multiple parallel requests while still using the Python middleware?

Answer 1

I think the single threaded process is a plugin caveat / gotcha.