Why CookieContainer does not replace cookies with the same name?

Question

Problem

I'm trying to scrap data from one famous website and found a question doing this.

This site uses cookies for authentication, and it sets the cookie, then replaces it during the authentication flow.

My problem is that CookieContainer does not replace cookies with the same name, same domain, but in the second case domain starts with dot.

But any browser or Postman does this.

So result of this is double-sent cookie, and the site fails authentication flow.

Sample

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using NUnit.Framework;

...

[Test]
public void Cookies()
{
    CookieContainer container = new CookieContainer();
    Uri uri = new Uri("https://example.com/item");
    string cookie1 = "myCookie=value1; Domain=example.com;  Expires=Sat, 11-Feb-2090 02:41:39 GMT; Path=/; HttpOnly";
    string cookie2 = "myCookie=value2; Domain=.example.com; Expires=Sat, 11-Feb-2090 02:41:39 GMT; Path=/; HttpOnly";

    container.SetCookies(uri, cookie1);
    container.SetCookies(uri, cookie2);

    List<Cookie> cookies = container.GetCookies(uri).ToList();
    foreach (Cookie cookie in cookies)
    {
        Console.WriteLine(cookie);
    }

    Assert.AreEqual(1, cookies.Count);
}

The output is:

myCookie=value1
myCookie=value2

But expected output is:

myCookie=value2

Why SetCookies ?

The SetCookies method is in the default implementation of CookieHelper which is used inside of Http2Stream of System.Net.Http.

It is used when the UseCookies property of HttpClientHandler is set to true:

HttpClientHandler handler = new HttpClientHandler()
{
    CookieContainer = new CookieContainer(),
    UseCookies = true,
};
var client = new HttpClient(handler);

//Make requests

My runtime version is NET 5, so HttpClientHandler hides SocketsHttpHandler implementation if it is necessary.

My questions:

• Why this could happen? I have not been able to replicate this behavior in any browser available to me
• Is it possible to fix this behavior somehow except the manually handling of cookies inside of Set-Cookie headers?

Thanks in advance!

---

UPD 1: Sample with mockserver

Step 1: set up example.com to localhost in hosts file.

127.0.0.1 example.com

Step 2: create mockserver

initializerJson.json

[
  {
    "httpRequest": {
      "method": "GET",
      "path": "/1",
      "secure": true
    },
    "httpResponse": {
      "statusCode": 200,
      "headers": {
        "Set-Cookie": [
          "myCookie=value1; Domain=example.com;  Expires=Sat, 11-Feb-2090 02:41:39 GMT; Path=/; HttpOnly"
        ]
      },
      "body": "1"
    }
  },
  {
    "httpRequest": {
      "method": "GET",
      "path": "/2",
      "secure": true
    },
    "httpResponse": {
      "statusCode": 200,
      "headers": {
        "Set-Cookie": [
          "myCookie=value2; Domain=.example.com; Expires=Sat, 11-Feb-2090 02:41:39 GMT; Path=/; HttpOnly"
        ]
      },
      "body": "2"
    }
  },
  {
    "httpRequest": {
      "method": "GET",
      "path": "/3",
      "secure": true
    },
    "httpResponseTemplate": {
      "template": "cookie = request.headers.Cookie; return { statusCode: 200, body: cookie }; ",
      "templateType": "JAVASCRIPT"
    }
  }
]

docker-compose.yml

version: "2.4"
services:
  mockServer:
      image: mockserver/mockserver:latest
      ports:
        - 5000:1080
      environment:
        MOCKSERVER_WATCH_INITIALIZATION_JSON: "true"
        MOCKSERVER_PROPERTY_FILE: /config/mockserver.properties
        MOCKSERVER_INITIALIZATION_JSON_PATH: /config/initializerJson.json
      volumes:
        - type: bind
          source: .
          target: /config

Run mockserver in terminal:

docker-compose up

After that you'll have a http server on your port 5000 with 3 endpoints:

• /1 - Sets cookie myCookie=value1; Domain=example.com; Expires=Sat, 11-Feb-2090 02:41:39 GMT; Path=/; HttpOnly
• /2 - Sets cookie myCookie=value2; Domain=.example.com; Expires=Sat, 11-Feb-2090 02:41:39 GMT; Path=/; HttpOnly
• /3 - Returns cookies sent to server

Step 3: test it

The testing flow is:

1. Request https://example.com:5000/1 endpoint
2. Request https://example.com:5000/2 endpoint
3. Request https://example.com:5000/3 endpoint and check response

Sample C# code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using NUnit.Framework;

...

[Test]
public async Task CheckCookies()
{
    CookieContainer container = new CookieContainer();
    HttpClientHandler handler = new HttpClientHandler()
    {
        CookieContainer = container,
        UseCookies = true,

        //Bypass self-signed https cert
        ServerCertificateCustomValidationCallback = (message, certificate2, arg3, arg4) => true,
    };
    HttpClient client = new HttpClient(handler);
    
    await client.GetAsync("https://example.com:5000/1");
    await client.GetAsync("https://example.com:5000/2");
    
    var response = await client.GetAsync("https://example.com:5000/3");
    string responseText = await response.Content.ReadAsStringAsync();
    Console.WriteLine(responseText);
    
    Assert.AreEqual("[ \"myCookie=value2\" ]", responseText);
}

Results

After testing that I got the following results:

• Chrome 97: [ "myCookie=value2" ]
• Firefox 97.0b6: [ "myCookie=value2" ]
• Postman 8.11.1 [ "myCookie=value2" ]
• Code above [ "myCookie=value1; myCookie=value2" ]

Any suggestions about that?

Answer 1

It's a bug in dotnet runtime: https://github.com/dotnet/runtime/issues/60628

Pending PR: https://github.com/dotnet/runtime/pull/64038