I have multiple devices each exposing an HTTPS server with a X509 certificate signed by my own CA. The clients call these servers by IP in the local network, but the IP is not known in advance, so the certs do not include the IP, but they have a custom (non-resolvable) DNS name.
On the client side (right now a mobile app), I am using dart HttpClient with a SecureContext to set my CA's certificate as Trusted and call the server.
However the cert verification and connection fail because, although the cert is signed by my CA, the cert name is not equal to the local IP address used to call the server.
This use case is supported by Android/Java that allows a HostnameVerifier. And it also seems that is possible to do it in iOS with a SecPolicyCreateSSL.
So I could write a Flutter plugin in each platform, but it looks to me that the right approach would be to support this directly in Dart, which uses BoringSSL to build a SecureSocket.
I am trying to write a Dart lib that allows a hostname verifier, and I think I have to somehow override the SecureSocket that the HttpClient uses, but I am not sure if I have to make calls directly to the BoringSSL library included with Dart (how do I do that?) or hopefully just do it purely in Dart.
Looks like this validation can be done with openssl. And handshake_client.cc in BoringSSL seems to have most of what is needed.
Can anyone give me some ideas / tips on how to accomplish this in Dart?
BTW, I've tried the badCertificateCallback, but it does not return the server certificate consistently, in Android it returns the leaf/server certificate, in iOS it returns the issuer CA certificate. Either way, it does seem to include enough data to validate that the server cert is actually correctly signed by the CA.