UPDATE
It turned out that the problem is unrelated to NAT and UDP hole punching. So I have created a new Q.
I've implemented UDP hole punching using the following approach:
A known server (no NAT), at a known port, is handling incoming UDP.
A client behind NAT sends a datagram to the server, after which the server responds by sending the address of a peer to this client.
The client now starts communicating with the peer using UDP over the same socket (but to different address of course.)
If after a while, the client stops communication with peer, and sends another datagram to the server again, this datagram does not arrive (unless I keep channels open by sending keepalive datagrams from server to client.)
My question: How come it is OK to switch from datagrams between server<->client to datagrams between peer<->client, but as soon as I switch back and the client sends to the server again, they don't arrive?
I suspect this behavior is specific to your NAT. It sounds like the port-mapping logic has issues. You could likely put in a different NAT box on your network and see different behavior.
My advice would be to always use a different local port and socket for each peer session. So let's say you were using port 2000 in the first session to talk to the server and peer, then the next session should use a completely different local port. Then whatever issues the NAT is having with the previous port won't impact this new socket.
In general, if you want to keep a UDP channel open between a pair of hosts, you should have at least one packet flowing every 45 seconds.