.NET (Core) on Unix: In-process modification of environment variables via P/Invoke system calls is unexpectedly not seen by .NET

Question

A notable limitation of the .NET environment-variable APIs as of .NET 8 is the inability to define environment variables without a value or, to put it differently, with a value that is the empty string.

Both Windows and Unix-like platforms do support this natively, however, via system calls, which is why calling the latter directly is of interest.

• Bringing this ability to .NET itself in the future is the subject of GitHub issue #50554.

While using P/Invoke declarations to access the relevant system calls works as expected on Windows, it does not on Unix-like platforms, as of .NET 8:

• .NET does not see the P/Invoke-mediated environment modifications and continues to report the old environment, both via Environment and when launching child processes.

Is this a bug, or is the code below missing something?

---

Sample C# console-application code, tested with version 8.0.101 of the .NET SDK:

• Run on a Unix-like platform (Linux or MacOS), as the content of the Program.cs file in a project created with dotnet new console

• Add <AllowUnsafeBlocks>true</AllowUnsafeBlocks> inside the <PropertyGroup> element of the *.csproj file in order for the project to compile.

using System.Diagnostics;
using System.Runtime.InteropServices;

partial class Program
{
  [LibraryImport("libc", StringMarshalling = StringMarshalling.Utf8)]
  private static partial int setenv(string name, string value, int overwrite);

  [LibraryImport("libc", StringMarshalling = StringMarshalling.Utf8)]
  private static partial IntPtr getenv(string name); // Convert to string with Marshal.PtrToStringUTF8()

  [LibraryImport("libc", StringMarshalling = StringMarshalling.Utf8)]
  private static partial int system(string command);

  static void ThrowSysCallError() => throw new System.ComponentModel.Win32Exception(Marshal.GetLastSystemError());

  static void Main(string[] args)
  {
    // Define / set env. var. 'FOO' with / to value 'new' 
    if (-1 == setenv("FOO", "new", 1)) ThrowSysCallError();

    Console.WriteLine(
      $"In-process value of FOO after setting it to 'new', via syscall: [{Marshal.PtrToStringUTF8(getenv("FOO"))}]"
    );

    Console.Write(
      "Value of FOO in a syscall-launched child process: "
    );
    system("echo [$FOO]");

    Console.WriteLine(
      $"In-process value of FOO per Environment.GetEnvironmentVariable(\"FOO\"): [{Environment.GetEnvironmentVariable("FOO")}]"
    );

    Console.Write(
      "Value of FOO in a .NET-launched child process: "
    );
    Process.Start(
      new ProcessStartInfo()
      {
        FileName = "sh",
        ArgumentList = { "-c", @"echo [$FOO]" },
      }
    )?.WaitForExit();

  }

}

The expected output would be:

In-process value of FOO after setting it to 'new', via syscall: [new]
Value of FOO in a syscall-launched child process: [new]
In-process value of FOO per Environment.GetEnvironmentVariable("FOO"): [new]
Value of FOO in a .NET-launched child process: [new]

The actual output is - note how .NET apparently doesn't see the FOO variable that the setenv() syscall set:

In-process value of FOO after setting it to 'new', via syscall: [new]
Value of FOO in a syscall-launched child process: [new]
In-process value of FOO per Environment.GetEnvironmentVariable("FOO"): []
Value of FOO in a .NET-launched child process: []

Answer 1

This was an interesting rabbit hole, thank you.

If we look at the runtime source, GetEnvironmentVariable gets delegated to the platform-specific GetEnvironmentVariableCore.

On Windows, this results in the appropriate interop call to Kernel32 (also when Set-ing). Nothing to see there.

---

On Unix, things look a little more interesting. It contains a static dictionary cache s_environment. If we're very lucky and it hasn't been populated yet, we'll get our desired interop call to getenv.

Unfortunately, any call to GetEnvironmentVariables (the plural version) and SetEnvironmentVariable will populate the cache - courtesy of EnsureEnvironmentCached.

Once that happens, all operations will happen on the cache. So if anything in the runtime has called either of them (it does. in so many places), we're out of luck.

This explains our first missing value. Calling GetEnvironmentVariable will likely end up with us being served a cached value, skipping the getenv call.

---

But what about the child process? Shouldn't that inherit the process's environment - which we *did* set via setenv?

If only we were so lucky. Like before, Process.Start() gets delegated to the platform-specific StartCore. Here it is in all its glory. Now notice the string[] envp = CreateEnvp(startInfo); line. If we keep digging, this envp keeps getting passed deeper, eventually ending up in the execve syscall, after forking.

This is a problem because CreateEnvp gets its values from ProcessStartInfo.Environment, which gets its values from... our good friend Environment.GetEnvironmentVariables() which we know from the last episode will serve us cached values.

---

The take home message from all this is that, in Unix/Linux, we should be safe if we stick to Environment.[Set,Get]EnvironmentVariable calls, as we'll be confined to operating with cached values.

But as you've established in your question, we're out of luck if we want to use empty variables.

Answer 2

NPras' excellent sleuthing has unearthed the de facto behavior of .NET; let me add some background information:

On Unix-like platforms - unlike on Windows - .NET by design maintains a private copy of the process' original environment-variable block that only managed code can see and modify,^[1] which has the following ramifications with respect to (in-process) calls to native libraries:

• Native libraries (called via P/Invoke) do not see changes to the managed process' environment variables that are later performed in-process by managed callers with System.Environment.SetEnvironmentVariable.

  • That is, any code in native libraries that queries the process environment (of necessity via system calls such as getenv) sees the original process environment, not the modified private copy maintained by .NET.

  • This is (tersely) documented, namely in System.Environment.SetEnvironmentVariable:

    • On non-Windows systems, calls to the SetEnvironmentVariable(String, String) method have no effect on any native libraries that are, or will be, loaded.

• Native libraries cannot modify the environment of a managed caller.

  • That is, any code in native libraries that modifies the process environment (of necessity via system calls such as setenv) has no effect on a manage caller.

  • As of this writing, this is not documented, neither in the documentation for the System.Environment.GetEnvironmentVariable method, nor in the System.Environment.GetEnvironmentVariable methods fundamentals article.

    • GitHub docs issue #39448 aims to change that.

---

The design rationale is purely one of technical necessity:

• On Unix-like platforms, the environment variable-related system calls are NOT guaranteed to be thread-safe.

• While explicit synchronization of access to environment variables is possible, its use cannot be guaranteed: native libraries could perform unsynchronized access.

• .NET therefore decided to create its own, managed environment "world", where thread safety can be guaranteed (because only managed callers can access it).

• For background information, see:

  • The setenv fiasco, for the Unix perspective.

  • GitHub issue #9529, for the .NET perspective.

---

^{[1] However, child processes created via System.Diagnostics.Process do inherit this private copy.}