Problems with Let semantics in Linq-to-Objects and Linq-to-XML

Question

Please consider the following sample, consisting of a definition of a nested XElement and a pair of Linq expressions. The first expression, which works as expected, iteratively fetches the first and last XElements at the bottom level by selecting over a tmp made by getting the bots (for bottoms), stored in new instances of an anonymous type for reuse of the name "bots." The second expression attempts to do the same thing, just using a "Let", but it does not work at all. First, the compiler complains about type inference not working, and then, when I put in explicit types, it goes off into IObservable and gets even more lost. I expected this to be utterly straightforward and was quite surprised and flummoxed at the failure. I'd be grateful if anyone has a moment to look and advise. You can paste the following into LinqPad, add a reference to System.Interactive, and see the failed compilation.

var root = new XElement("root",
    new XElement("sub",
        new XElement("bot", new XAttribute("foo", 1)),
        new XElement("bot", new XAttribute("foo", 2))),
    new XElement("sub",
        new XElement("bot", new XAttribute("foo", 3)),
        new XElement("bot", new XAttribute("foo", 4))));
root.Dump("root");

root.Descendants("sub")
    .Select(sub => new {bots = sub.Descendants("bot")})
    .Select(tmp => new{fst = tmp.bots.First(), snd = tmp.bots.Last()})
    .Dump("bottoms")
    ;

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => new{fst = bots.First(), snd = bots.Last()})
    .Dump("bottoms2")
    ;

Answer 1

let is just a keyword that simplifies transforms like tmp => new{fst = tmp.bots.First(), snd = tmp.bots.Last()}. Rather than using a Let extension method, just use the Select method:

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Select(bots => new{fst = bots.First(), snd = bots.Last()})
    .Dump("bottoms2");

Answer 2

ok, found it, though I don't completely understand the answer. Here are two expressions that produce the desired results, one using "Let" and the other using "Select:"

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => bots.Select(bot => new{fst = bot.First(), snd = bot.Last()}))
    .Dump("bottoms2")
    ;

root.Descendants("sub")
    .Select(sub => new {bots = sub.Descendants("bot")})
    .Select(tmp => new{fst = tmp.bots.First(), snd = tmp.bots.Last()})
    .Dump("bottoms")
    ;

The first "Select," .Select(sub => sub.Descendants("bot")), in the first of the two expressions, the "Let" form, produces an enumerable of enumerables of XElements, or, more precisely,

System.Linq.Enumerable+WhereSelectEnumerableIterator`2[System.Xml.Linq.XElement,System.Collections.Generic.IEnumerable`1[System.Xml.Linq.XElement]]

The first "Select," .Select(sub => new {bots = sub.Descendants("bot")), in the second of the two expressions, the "Select" form, produces an enumerable of anonymous types, each of which contains an enumerable, named "bots" of XElements:

System.Linq.Enumerable+WhereSelectEnumerableIterator`2[System.Xml.Linq.XElement,<>f__AnonymousType0`1[System.Collections.Generic.IEnumerable`1[System....

We want to transform each of the inner enumerables into a {fst, snd} pair. Begin by noting that the following two expressions produce the same results, but are not semantically identical, as we show below. The only difference between these two expressions is that the first one has "Let" on line 3, and the second one has "Select" on line 3. They're just like the "answer" expressions except they don't have the inner transformations.

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => bots.Select(bot => bot))
    .Dump("bottoms3")
    ;

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Select(bots => bots.Select(bot => bot))
    .Dump("bottoms4")
    ;

The type of "bots" in the outer "Let" in the first expression differs from the type of "bots" in the outer "Select" in the second expression. In the "Let," the type of "bots" is (roughly) IEnumerable<IEnumerable<XElement>> (its full name is

System.Linq.Enumerable+WhereSelectEnumerableIterator`2[System.Xml.Linq.XElement,System.Collections.Generic.IEnumerable`1[System.Xml.Linq.XElement]]

We can see in more detail by Selecting over the insides that each "bot" in "bots" is an IEnumerable<XElement>:

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => 
    {
        bots.GetType().Dump("bots in Let"); 
        return bots.Select(bot => bot.GetType());
    })
    .Dump("Types of bots inside the LET")
    ;

Types of bots inside the LET

IEnumerable<Type> (2 items)

typeof (IEnumerable<XElement>)

typeof (IEnumerable<XElement>)

In the outer "Select," the type of "bots" is

System.Xml.Linq.XContainer+<GetDescendants>d__a

By a parallel analysis to the above, we see that each "bot" in "bots" is an IEnumerable of something, and that something is an XElement.

    root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => 
    {
        bots.GetType().Dump("bots in Let"); 
        return bots.Select(bot => bot.GetType());
    })
    .Dump("Types of bots inside the LET")
    ;

Types of bots inside the SELECT

IEnumerable<IEnumerable<Type>> (2 items)

IEnumerable<Type> (2 items)

typeof (XElement)

typeof (XElement)

IEnumerable<Type> (2 items)

typeof (XElement)

typeof (XElement)

It's tempting to think of these as semantically the same, but they're not. There is one level more of implicit packaging at the type level in the "Select" form than there is in the "Let" form, or vice versa depending on your point of view.

Also, obviously, the "Let" "runs" once over the result of .Select(sub => sub.Descendants("bot")), whereas the "Select" runs multiple times, once over each result The following is wrong because it ignores that "level of packaging."

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => new{fst = bots.First(), snd = bots.Last()})
    .Dump("bottoms2")
    ;

As I said, I don't completely understand every detail of this phenomenon, yet. Perhaps with a few more examples and another night's lost sleep over it, I'll begin to develop a more refined intuition about it. Here is my full LinqPad script in case you're so motivated to play with this subtlety:

void Main()
{
Console.WriteLine ("Here is a sample data set, as XML:");
var root = new XElement("root",
new XElement("sub",
    new XElement("bot", new XAttribute("foo", 1)),
    new XElement("bot", new XAttribute("foo", 2))),
new XElement("sub",
    new XElement("bot", new XAttribute("foo", 3)),
    new XElement("bot", new XAttribute("foo", 4))));
root.Dump("root");

Console.WriteLine ("The following two expressions produce the same results:");

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => bots.Select(bot => new{fst = bot.First(), snd = bot.Last()}))
    .Dump("LET form: bottoms1")
    ;

root.Descendants("sub")
    .Select(sub => new {bots = sub.Descendants("bot")})
    .Select(tmp => new{fst = tmp.bots.First(), snd = tmp.bots.Last()})
    .Dump("SELECT form: bottoms2")
    ;

Console.WriteLine ("Analysis of LET form");

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Dump("Top-Level Select in the \"Let\" form:")
    ;

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .GetType()
    .Dump("Type of the top-Level Select in the \"Let\" form:")
    ;

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => bots.Select(bot => bot))
    .Dump("Let(bots => bots.Select(bot => bot))")
    ;

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Let(bots => 
    {
        bots.GetType().Dump("bots in Let"); 
        return bots.Select(bot => bot.GetType());
    })
    .Dump("Types of bots inside the LET")
    ;

Console.WriteLine ("Analysis of SELECT form");

root.Descendants("sub")
    .Select(sub => new {bots = sub.Descendants("bot")})
    .Dump("Top-level Select in the \"Select\" form:")
    ;

root.Descendants("sub")
    .Select(sub => new {bots = sub.Descendants("bot")})
    .GetType()
    .Dump("Type of the top-level Select in the \"Select\" form:")
    ;

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Select(bots => bots.Select(bot => bot))
    .Dump("bots => bots.Select(bot => bot)")
    ;

root.Descendants("sub")
    .Select(sub => sub.Descendants("bot"))
    .Select(bots =>         
    {
        bots.GetType().Dump("bots in Select"); 
        return bots.Select(bot => bot.GetType());
    })
    .Dump("Types of bots inside the SELECT")
    ;
}