How do I convert a refresh token to an access token using the LiveConnect API (C#)

Question

I'm trying to create a LiveConnectClient with only a refresh token that was provided to me via asp.net identity (using OWIN) and the ProviderKey. It looks like the only way to do this without needing HttpContextBase is via InitializeSessionAsync.

When I try and create the client I'm getting:

Microsoft.Live.LiveAuthException: The user ID from the given RefreshTokenInfo instance does not match the refresh token.

Not really sure what user ID it is expecting as I'm giving it the provider key that was passed via ASP.NET Identity (17 chars in my case). Below is my code.

public class Class1
{   
    protected async Task<LiveConnectClient> GetLiveConnectClient()
    {
        var authClient = new LiveAuthClient(_clientId, _clientSecret, null, new RefreshTokenHandler(_refreshToken, _providerKey));
        var session = await authClient.InitializeSessionAsync("http://.../signin-microsoft");
        return new LiveConnectClient(session.Session);
    }
}

public class RefreshTokenHandler : IRefreshTokenHandler
{
    private readonly string _refreshToken;
    private readonly string _userId;
    public RefreshTokenHandler(string refreshToken, string userId)
    {
        _refreshToken = refreshToken;
        _userId = userId;
    }

    public Task<RefreshTokenInfo> RetrieveRefreshTokenAsync()
    {
        return Task.FromResult(new RefreshTokenInfo(_refreshToken, _userId));
    }

    public async Task SaveRefreshTokenAsync(RefreshTokenInfo tokenInfo)
    {
        await Task.Delay(0);
    }
}