New ask Hacker News story: Ask HN: How important is it for telemetry apps to have something listening?

Ask HN: How important is it for telemetry apps to have something listening?
2 by m3047 | 2 comments on Hacker News.
Company A offers a network software product which is ubiquitous on the internet (seriously, it has lost some market share over the years depending on how you measure it so maybe "pervasive" or "endemic" would be better characterizations). Company B was founded by Company A's chief architect and tech leadership to do things with telemetry related to Company A's product and eventually prevailed on Company A to implement a direct telemetry tap in the product, which I'm perfectly happy about in principle. For whatever reasons company B chose to implement unix domain sockets as the only streaming protocol / mechanism for the telemetry tap. The protocol uses a handshake initiated by the client at setup, the software is the "client" and whatever receives the telemetry is the "server". The tap is logging, not auditing: if messages can't be received then they are dropped, instead of the application pausing processing. You might suspect that backpressure might make the application pause, but that is incorrect because the protocol client drops messages with no in-band indicator if backpressure occurs. The C and Go language implementations of the protocol were both written by Company B; the only other implementation was written by me (pure Python). Both the chief architect and myself have been writing networking and telemetry software for over 30 years. I'd really like the software application to send its telemetry directly to the network. I recently asked them: why not direct-to-network, and why not UDP (with a header containing a packet counter and a start/continuation indicator)? Their answer was "too easy to send the stream to somebody who's not listening". I am puzzled by that answer. Granted, there's no positive connection indicator with UDP as opposed to TCP (where ACKs still have to be sent back to the client) but the answer as given still needs unpacking, because what kind of indicator exists in the unix socket when at the application layer the protocol client drops messages with no fanfare? Other than the initial handshake and the observable that the connection is blocked but still intact I can't see any. Why not use UDP and let the network drop packets for free? How is that worse than network drops a TCP packet and retries, so eventually backpressure develops and the stream is stalled by the client? UDP is going to get an ICMP response to the client/sender that the port, host or network is unreachable. I don't think it's so bad that servers can come and go and telemetry still occurs oblivious, maybe that's all there is to it (plus there's the option for multicast this way which kind of makes the point). In terms of security "rings of hell" you can restrict the traffic to localhost, a VPC, a LAN, a subnet (and you'd want to with this data). If you don't want to offload that responsibility, you could do it from the client/sender by limiting the number of hops the packet can make (IP datagram TTL). Is there really any important difference TCP vs UDP other than those outlined above? Is the difference in operating dynamics between TCP and unix domain sockets sufficient that equivalency should even be entertained?