C# method CheckSignature returns false for valid XAdES signed document

Question

I try to implement XAdES signature verification in C#. It works fine for sample documents like this. However on actual document like this method CheckSignature returns false, even though it is a valid document (additionally verified here).

Here's my code

public bool Verify(Stream xmlFileStream)
{
    var xmlDoc = new XmlDocument();
    xmlDoc.Load(xmlFileStream);

    var signatureNodes = xmlDoc.GetElementsByTagName("ds:Signature");
    if (signatureNodes == null || signatureNodes.Count != 1)
    {
        throw new Exception("Wrong number of signature nodes in document");
    }

    var signedXml = new SignedXml(xmlDoc);
    signedXml.LoadXml((XmlElement)signatureNodes[0]);            
    var signatureValid = signedXml.CheckSignature();
    return signatureValid;
}

I've tried loading xml with and without preserving whitespaces, even tried removing namespaces from the document, nothing is working.

EDIT
I've turned on diagnostic logging and it says

System.Security.Cryptography.Xml.SignedXml Information: 12: [SignedXml # 0060fb3c, VerificationFailure] Verification failed while checking the Reference.

Unfortunately it doesn't say much to me, I still have no idea why the reference verification fail here while it seems ok for online verification service.