Why Apache Traffic Server does not cache https content even though SSL termination has been setup?

Question

I'm using ATS as forward proxy. The http content cache without any problem, but the https content just don't work. Here are my configures.

# records.config
CONFIG proxy.config.http.server_ports STRING 8125 8443:ssl
CONFIG proxy.config.http.cache.required_headers INT 0
CONFIG proxy.config.url_remap.remap_required INT 0
CONFIG proxy.config.reverse_proxy.enabled INT 0
CONFIG proxy.config.ssl.server.cert.path STRING /usr/local/etc/ssl/
CONFIG proxy.config.ssl.server.private_key.path STRING /usr/local/etc/ssl/
CONFIG proxy.config.ssl.client.certification_level INT 0

# ssl_multicert.config
dest_ip=*      ssl_cert_name=fullchain-from-letsencrypt.pem ssl_key_name=key-from-letsencrypt.pem

In which fullchain-from-letsencrypt.pem and key-from-letsencrypt.pem was generated with certbot. This should make SSL termination works, but it's not.

export http_proxy=http://ats.domain:8125
export https_proxy=https://ats.domain:8443;

# http works fine.
wget https://releases.ubuntu.com/jammy/ubuntu-22.04.2-live-server-amd64.iso.zsync

# https will not cache, and got error messages.
wget https://releases.ubuntu.com/jammy/ubuntu-22.04.2-live-server-amd64.iso.zsync

And I got this error message.

--2023-08-04 10:38:30--  https://releases.ubuntu.com/jammy/ubuntu-22.04.2-live-server-amd64.iso.zsync
Resolving ats.domain (ats.domain)... 10.147.19.193
Connecting to ats.domain (ats.domain)|10.147.19.193|:8443... connected.
Failed reading proxy response: Success
Retrying.

curl fetch the file properly both http and https.

# http works fine
curl http://releases.ubuntu.com/jammy/ubuntu-22.04.2-live-server-amd64.iso.zsync > /dev/null

# https not cache without error
curl https://releases.ubuntu.com/jammy/ubuntu-22.04.2-live-server-amd64.iso.zsync > /dev/null

But https does not cache according to the log.

traffic_logcat -f /usr/local/var/log/trafficserver/squid.blog

1691116320.693 6 TCP_HIT/200 3859845 GET http://releases.ubuntu.com/jammy/ubuntu-22.04.2-live-server-amd64.iso.zsync - NONE/- application/x-iso9660-image
1691116855.993 7849 TCP_MISS/200 105 CONNECT releases.ubuntu.com:443/ - DIRECT/releases.ubuntu.com -

And this is the curl log

* Uses proxy env variable https_proxy == 'https://ats.domain:8443'
* Connected to ats.domain (10.147.19.193) port 8443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [25 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3971 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Proxy certificate:
*  subject: CN=ats.domain
*  start date: Aug  2 02:37:07 2023 GMT
*  expire date: Oct 31 02:37:06 2023 GMT
*  subjectAltName: host "ats.domain" matched cert's "ats.domain"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* allocate connect buffer!
* Establish HTTP proxy tunnel to releases.ubuntu.com:443
} [5 bytes data]
> CONNECT releases.ubuntu.com:443 HTTP/1.1
> Host: releases.ubuntu.com:443
> User-Agent: curl/7.74.0
> Proxy-Connection: Keep-Alive
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
< HTTP/1.1 200 OK
< Date: Fri, 04 Aug 2023 03:39:37 GMT
< Proxy-Connection: keep-alive
< Server: ATS/9.2.1
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CONNECT phase completed!
* CONNECT phase completed!
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [25 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2622 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=releases.ubuntu.com
*  start date: Jul 13 21:45:45 2023 GMT
*  expire date: Oct 11 21:45:44 2023 GMT
*  subjectAltName: host "releases.ubuntu.com" matched cert's "releases.ubuntu.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
> GET /jammy/ubuntu-22.04.2-live-server-amd64.iso.zsync HTTP/1.1
> Host: releases.ubuntu.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [297 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 04 Aug 2023 03:39:37 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Last-Modified: Thu, 23 Feb 2023 17:47:38 GMT
< ETag: "3ae46d-5f5619bd762ba"
< Accept-Ranges: bytes
< Content-Length: 3859565
< Content-Type: application/x-iso9660-image
< 
{ [5 bytes data]
* Connection #0 to host ats.domain ⇦ intact

Are there anything that I missed in the configuration for SSL termination?

Answer 1

In you http case, curl used the GET method (1691116320.693 6 TCP_HIT/200 3859845 GET), but, in the https, it used CONNECT (squid.log + curl log -- the 'Uses proxy env variable https_proxy...' probably does that?). In the latter case, your ATS will not see the object nor headers -- its more like a tunnel between curl and the end destination. If you follow the curl log, your ATS gets the request lines starting with ">" from CONNECT releases.ubuntu.com:443 HTTP/1.1 to the > Proxy-....

Can you try: curl "https://releases.ubuntu.com/jammy/ubuntu-22.04.2-live-server-amd64.iso.zsyn" --connect-to ::$ATS_IP -D- -o/dev/null ... thinking this would force it to be a GET, and perhaps allow further troubleshooting.