Chef apt_repository started failing with SSL verification errors

Question

From last 2 days we started seeing chef execution failures are happening related to apt_repository resource. Seen similar failures with chef-client execution too

Chef-solo version: 12.19.36 OS: Ubuntu v18 & Ubuntuv14

See below error from chef-solo execution.

==> core: [2021-10-04T14:36:46+00:00] ERROR: SSL Validation failure connecting to host: www.postgresql.org - SSL_connect returned=1 errno=0 state=error: certificate verify failed
==> core:
==> core:
==> core: ================================================================================
==> core: Error executing action create on resource 'remote_file[/var/chef/cache/https___www_postgresql_org_media_keys_ACCC4CF8_asc]'
==> core: ================================================================================
==> core:
==> core: OpenSSL::SSL::SSLError
==> core: ----------------------
==> core: SSL Error connecting to https://www.postgresql.org/media/keys/ACCC4CF8.asc - SSL_connect returned=1 errno=0 state=error: certificate verify failed
==> core:
==> core:
==> core: Resource Declaration:
==> core: ---------------------
==> core: # In /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/provider/apt_repository.rb
==> core:
==> core: 166: declare_resource(type, cached_keyfile) do
==> core: 167: source new_resource.key
==> core: 168: mode "0644"
==> core: 169: sensitive new_resource.sensitive
==> core: 170: action :create
==> core: 171: end
==> core: 172:
==> core:
==> core: Compiled Resource:
==> core:
==> core: ------------------
==> core: # Declared in /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/provider/apt_repository.rb:166:in `install_key_from_uri'
==> core:

Answer 1

This is related to last Thursday's Root CA Certificate Expiration for Let's Encrypt certificates.

For OpenSSL, you need to upgrade to at least 1.1 I believe due to a bug with how 1.0.1 handles the certificate chain. If that's not possible you will need to remove the DST Root CA X3 certificate from the OS.

Chef also provides it's own root certificates it uses instead of the OS. I had to remove the above certificate from /opt/chef/embedded/ssl/certs/cacert.pem and also add ISRG Root X1 to it.

Answer 2

You're on a pretty old version of Chef there. Note that chef has since released new versions that remove the expired certs as well as upgrading OpenSSL. The latest Chef 16 & 17 versions should address this problem.

Answer 3

The simpler fix, which also keeps your Chef client certificates in sync with your system, is to allow Chef client to use your system certificates - assuming you are not getting these errors outside of the Chef client runs.

To do this, remove the existing symbolic link, then re-create it pointing to your system certificates (Ubuntu 18.04 in this example):

$ ls -l /opt/chefdk/embedded/ssl/cert.pem
lrwxrwxrwx 1 root root 16 Jun  2  2020 /opt/chefdk/embedded/ssl/cert.pem -> certs/cacert.pem

# sudo rm /opt/chefdk/embedded/ssl/cert.pem
# sudo ln -s /etc/ssl/certs/ca-certificates.crt /opt/chefdk/embedded/ssl/cert.pem

$ ls -l /opt/chefdk/embedded/ssl/cert.pem
lrwxrwxrwx 1 root root 34 Mar 10 09:16 /opt/chefdk/embedded/ssl/cert.pem -> /etc/ssl/certs/ca-certificates.crt

To check that all is OK:

$ /opt/chefdk/embedded/bin/openssl s_client -connect some.domain.com:443 -showcerts
CONNECTED(00000003)                                             
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1         
verify return:1                                                 
depth=1 C = US, O = Let's Encrypt, CN = R3                      
verify return:1                                                                 
depth=0 CN = some.domain.com                                    
verify return:1
---                                                             
Certificate chain 

<snip>

    Start Time: 1646864178                                      
    Timeout   : 300 (sec)                                       
    Verify return code: 0 (ok)

<snip>