You can adjust the memory overcommit policy system-wide via sysctl calls, but is it possible to adjust this policy on a per-process basis from within the process itself?
I want to make one particular real-time process never use overcommit, though the rest of the system can overcommit.
No, it's not possible.
Somewhat related, for a real-time process, you probably want to use the mlock or mlockall functions to pin your address space to RAM.