So I am trying right now to introduce a service mesh to an already existing distributed system. A core software component uses a proprietary TCP protocol to communicate with another component.
(arrives TCP: 7010 and 7011)
Service A ===> Service B
With Linkerd2 there is Protocol Detection:
While protocol detection is designed to allow Linkerd “just work”, there are some cases where it cannot: the infamous server-speaks-first protocols. These protocols, which include things like MySQL and SMTP, work by having the client establish a connection and then waiting for the server to respond. This is a perfectly legal behavior from the TCP perspective, but means that Linkerd cannot detect the protocol as the relevant information comes from the server, not the client.
(Why not simply use the server’s bytes to detect the protocol? Because at the time of protocol detection Linkerd hasn’t even established a connection to the server. Choosing which server to speak to is a function of the load balancer, and which load balancer to use is a function of the protocol. It’s a delicious, TCP-flavored chicken-and-egg problem.)
So I have tried annotating both service A and B with config.linkerd.io/opaque-ports: 7010,7011. This has not worked at all, the connection can still not be established when service A calls service B.
The only way I can have the connection working is when I add config.linkerd.io/skip-outbound-ports: 7010,7011 which circumvents Linkerd's sidecar proxies and established the connection outside of Linkerd's realm of control.
This means that mTLS encryption is also not enforced for all the traffic going through both services through the on port 7010 and 7011 established connections.
How can this be and how can this even make sense? I do not understand it as all the documentation states that using opaque-ports works arround this and that skip-outbound-ports should only be used in debug settings as opaque-ports should be sufficient.
In my case however I can only establish traffic flow by circumventing linkerd. Does anyone have any ideas what is happening here? Thanks in advance!