Openssl s_server DTLSv1.2 is not working with certificate RSA-PSS

Question

I have some issue using DTLS with RSA-PSS certificate.

My end entity certificate with rsa-pss key and signed with rsa-pss signature algorithm: My end entity is enrolled under my root certificate CA_ROOT_PKCS1_5.crt Certificate extract: .... Subject Public Key Info: Public Key Algorithm: rsassaPss RSA-PSS Public-Key: (2048 bit) ... Signature Algorithm: rsassaPss Hash Algorithm: sha512 Mask Algorithm: mgf1 with sha512 Salt Length: 0x01BE Trailer Field: 0xBC (default) ... enter image description here

For the test purpose, i use the same Key pair at both client and server side.

I do TLSv1.2 connection (tls mutual authentication) between a client and a server:

openssl s_server -cert ee_underRoot_pss_sign_pss.crt -key ee_pss_RSA.key -CAfile CA_ROOT_PKCS1_5.crt -tls1_2 -accept 127.0.0.1:30000 -security_debug_verbose

openssl s_client -showcerts -tls1_2 -cert ee_underRoot_pss_sign_pss.crt -key ee_pss_RSA.key -CAfile CA_ROOT_PKCS1_5.crt -connect 127.0.0.1:30000 -security_debug_verbose

Everything is working well !

I try now to do the same thing but in DTLSv1.2:

openssl s_server -cert ee_underRoot_pss_sign_pss.crt -key ee_pss_RSA.key -CAfile CA_ROOT_PKCS1_5.crt -dtls1_2 -accept 127.0.0.1:30000 -security_debug_verbose

openssl s_client -showcerts -dtls1_2 -cert ee_underRoot_pss_sign_pss.crt -key ee_pss_RSA.key -CAfile CA_ROOT_PKCS1_5.crt -connect 127.0.0.1:30000 -security_debug_verbose

but the handshake failed with the following error: "no shared cipher"

enter image description here

When i look at the ciphersuites supported by the client, there is the same list that when i did with TLS enter image description here

Here the trace at server side:

verify depth is 10, must return a certificate
Using default temp DH parameters
Security callback: Certificate chain EE key=RSA-PSS, bits=2048, security bits=112: yes
Security callback: Certificate chain CA digest=RSASSA-PSS, security bits=256: yes
ACCEPT
Security callback: Version=???: yes
Security callback: Version=???: yes
Security callback: Version=???: yes
Security callback: : yes
Security callback: Shared Signature Algorithm digest=SHA256, algorithm=ECDSA, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algorithm=ECDSA, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algorithm=ECDSA, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA256, algid=9, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algid=10, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algid=11, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA256, algid=4, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algid=5, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algid=6, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA256, algorithm=RSA, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algorithm=RSA, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algorithm=RSA, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA224, algorithm=ECDSA, security bits=112: yes
Security callback: Shared Signature Algorithm digest=SHA1, algorithm=ECDSA, security bits=80: yes
Security callback: Shared Signature Algorithm digest=SHA224, algorithm=RSA, security bits=112: yes
Security callback: Shared Signature Algorithm digest=SHA1, algorithm=RSA, security bits=80: yes
Security callback: Shared Signature Algorithm digest=SHA224, algorithm=DSA, security bits=112: yes
Security callback: Shared Signature Algorithm digest=SHA1, algorithm=DSA, security bits=80: yes
Security callback: Shared Signature Algorithm digest=SHA256, algorithm=DSA, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algorithm=DSA, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algorithm=DSA, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA256, algorithm=ECDSA, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algorithm=ECDSA, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algorithm=ECDSA, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA256, algid=9, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algid=10, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algid=11, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA256, algid=4, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algid=5, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algid=6, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA256, algorithm=RSA, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algorithm=RSA, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algorithm=RSA, security bits=256: yes
Security callback: Shared Signature Algorithm digest=SHA224, algorithm=ECDSA, security bits=112: yes
Security callback: Shared Signature Algorithm digest=SHA1, algorithm=ECDSA, security bits=80: yes
Security callback: Shared Signature Algorithm digest=SHA224, algorithm=RSA, security bits=112: yes
Security callback: Shared Signature Algorithm digest=SHA1, algorithm=RSA, security bits=80: yes
Security callback: Shared Signature Algorithm digest=SHA224, algorithm=DSA, security bits=112: yes
Security callback: Shared Signature Algorithm digest=SHA1, algorithm=DSA, security bits=80: yes
Security callback: Shared Signature Algorithm digest=SHA256, algorithm=DSA, security bits=128: yes
Security callback: Shared Signature Algorithm digest=SHA384, algorithm=DSA, security bits=192: yes
Security callback: Shared Signature Algorithm digest=SHA512, algorithm=DSA, security bits=256: yes
ERROR
540409864:error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:ssl/statem/statem_srvr.c:2263:
shutting down SSL
CONNECTION CLOSED

(if instead, i use certificate containing a "public key rsaencryption" instead of "public key rsapss", DTLS works)

I do not understand why it is working with TLS1.2 but not with DTLS1.2. Could you help me ?

(I have the issue with both openssl1.1.1 and openssl3.0)'

Answer 1

I can't prove a negative -- and although this is about code I'm not certain it's really ontopic -- but I suspect it's because nothing says it should work.

Neither 5246 nor 6347 said anyting about PSS (although 4055 back in 2005 implemented it, and also OAEP, in PKIX certs, which are linked to TLS and DTLS to some extent). 8446 requires protocol signatures use PSS (if RSA) and prefers certificates to do so (sigals and sigalgs-cert) in 4.2.3 and states that

   -  Implementations that advertise support for RSASSA-PSS (which is
      mandatory in TLS 1.3) MUST be prepared to accept a signature using
      that scheme even when TLS 1.2 is negotiated.  In TLS 1.2,
      RSASSA-PSS is used with RSA cipher suites.

also reaffirmed in 1.3. No similar statement is made about using PSS in DTLS 1.2, and the only reference to 6347 is about using cookie to offload HRR state.

At any rate, that's what the code implements -- a PSS cert-and-key enables selection of a 1.2 suite with aRSA only in TLS version 1.2 not (any) DTLS.

Note DTLS 1.3 implementation (which must use PSS, and might provide it for 1.2 as a 'bonus') only recently got started and as it will presumably require API changes and 3.2 is already in beta it will probably have to wait for 3.3 at least.