Can I use SafeHandle to manage GCHandle?

Question

SafeHandle gives a very strong guarantee that the resource it manages will eventually be released, even if an asynchronous exception occurs. Even using and try finally cannot guarantee that.

I am thinking to use this guarantee to free a pinned GCHandle, however, there seems to be no ready to use implementation of the GCHandle managed by SafeHandle in C#.

So I created my own implementation that looks like this:

public class ArraySafeHandle : SafeHandleZeroOrMinusOneIsInvalid
{
    private GCHandle _handle;

    public ArraySafeHandle(byte[] array) : base(true)
    {
        _handle = GCHandle.Alloc(array, GCHandleType.Pinned);
        SetHandle(_handle.AddrOfPinnedObject());
    }

    protected override bool ReleaseHandle()
    {
        _handle.Free();
        return true;
    }
}

This works, however, I am concerned that nobody is using GCHandle in this way. Also, setting the IntPtr in a SafeHandle manually does not seem to be a common way to create the SafeHandle (see, for example, this remark in the official docs).

What is the correct implementation? Is it even a working solution to wrap GCHandle in SafeHandle or it can fail at any time?