WeakReference returns wrong object

Question

I've noticed a strange behavior in one of our applications recently.

Exception=System.InvalidCastException: Unable to cast object of type 'System.Data.SqlClient.SqlTransaction' to type 'System.Byte[]'.
   at ServiceStack.Text.Pools.BufferPool.GetCachedBuffer(Int32 minSize) in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\Pools\BufferPool.cs:line 55
   at ServiceStack.Redis.RedisNativeClient..ctor(RedisEndpoint config) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 447
   at ServiceStack.Redis.RedisClient..ctor(RedisEndpoint config) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisClient.cs:line 66
   at ServiceStack.Redis.RedisConfig.<>c.<.cctor>b__35_0(RedisEndpoint c) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisConfig.cs:line 22
   at ServiceStack.Redis.RedisResolver.CreateRedisClient(RedisEndpoint config, Boolean master) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisResolver.cs:line 76
   at ServiceStack.Redis.RedisManagerPool.GetClient() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisManagerPool.cs:line 214

...

Or

Exception=System.InvalidCastException: Unable to cast object of type 'System.Byte[]' to type 'System.Transactions.SafeIUnknown'.
   at System.Transactions.Transaction.JitSafeGetContextTransaction(ContextData contextData)
   at System.Transactions.Transaction.FastGetTransaction(TransactionScope currentScope, ContextData contextData, Transaction& contextTransaction)
   at System.Transactions.Transaction.get_Current()
   at System.Data.ProviderBase.DbConnectionPool.GetFromTransactedPool(Transaction& transaction)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at ServiceStack.OrmLite.OrmLiteConnection.Open() in C:\BuildAgent\work\27e4cc16641be8c0\src\ServiceStack.OrmLite\OrmLiteConnection.cs:line 86
   ...

Exceptions similar to these two, are being thrown from various parts of our application. The only thing in common to all of those exceptions is the WeakReference object, but I can see no obvious cause for this exception. For example, the ServiceStack.Text code that throws this exception is:

    private class CachedBuffer
    {
        private readonly WeakReference _reference;

        public int Size { get; }

        public bool IsAlive => _reference.IsAlive;
        public byte[] Buffer => (byte[])_reference.Target;

        public CachedBuffer(byte[] buffer)
        {
            Size = buffer.Length;
            _reference = new WeakReference(buffer);
        }
    }

Exception is thrown from Buffer property getter, apparently _reference.Target points to SqlTransaction object and not byte[], but _reference is only initialized once in the constructor and can't be changed afterwards, so how could this exception be thrown?

Additionally none of this code has changed recently, so it makes no sense, that it would suddenly start throwing errors. I also can't see any way we could have caused this bug by some change in our code, or am I wrong about that?

Could this be a bug in .net clr and if so, how could I diagnose it? Our application uses .NET 4.8 framework and we've been seeing those bugs on Windows server 2012 and Windows server 2019 machines within our testing environments, but I have not been able to reproduce them locally on my development machine.

Answer 1

I've managed to locate the problem, it was indeed our own code that caused this error and it was related to reflection. Long story short, one of our developers introduced a code, that invoked deep clone on ExpandoObject.Keys property. This property is not a simple collection of strings, but also contains a reference to entire ExpandoObject and deep inside ExpandoObject there are also some WeakReference fields. I still don't understand exactly what happens, but I guess that cloning those WeakReferences inside ExpandoObject somehow caused the bug we experienced.

Thanks for your help. I inspected that code several times, but completely missed the deep clone invocation, because that extension method had a very generic name.