Azure Synapse .NET C# Sparkpool: Fail to start interpreter

Question

When I am working on a .NET Spark (C#) Notebook in Azure Synapse I always get the following error message: Fail to start interpreter. detail: org.apache.spark.api.dotnet.DotnetBackend. When changing the language from .NET Spark (C#) to Python or Scala the notebook is working as expected.

What is causing this error?

I have tried:

• Working in a completely new resource group
• Working with a larger Spark Node
• Running the notebook Hitchhiker's Guide to .NET for Apache Spark available via the Gallery
• Search online for a solution

Answer 1

Short answer: Use spark 3.2 runtime.

I asked the same question on the microsoft Q&A where I got a working solution. https://learn.microsoft.com/en-us/answers/questions/1179148/azure-synapse-net-c-sparkpool-fail-to-start-interp