Tendermint, GRPC and C# - Stream terminated by RST_STREAM with error code: PROTOCOL_ERROR

Question

I need Tendermint in one of my projects but have never used it before so I am trying to implement a very simple example from here first: https://docs.tendermint.com/master/tutorials/java.html but in C# (.NET 5.0).

(Download: Minimal Example)

I have created a simple GRPC Service trying to follow the guide as closely as possible:

Startup.cs:

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddGrpc();
    }
    
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env, IConfiguration conf)
    {
        if (env.IsDevelopment())
            app.UseDeveloperExceptionPage();

        app.UseRouting();
        app.UseEndpoints(endpoints =>
        {
            endpoints.MapGrpcService<KVStoreService>();
            endpoints.MapGet("/", async context =>
            {
                await context.Response.WriteAsync("Communication with gRPC endpoints must be made through a gRPC client. To learn how to create a client, visit: https://go.microsoft.com/fwlink/?linkid=2086909");
            });
        });

        app.UseGlobalHostEnvironemnt(env); // These are for tests and are irrelevant for the problem
        app.UseGlobalConfiguration(conf);
        app.UseGlobalLogger();
        app.UseTendermint(); // This starts tendermint process
    }
}

Starting tendermint uses basic Process.Start calls. I left the .toml config file with defaults and yes there is a typo in documentation, --proxy_app flag should be typed with underscore (errors were verbose about that):

public static void ConfigureTendermint()
{
    var tendermint = Process.Start(new ProcessStartInfo
    {
        FileName = @"Tendermint\tendermint.exe",
        Arguments = "init validator --home=Tendermint"
    });
    tendermint?.WaitForExit();

    Process.Start(new ProcessStartInfo
    {
        FileName = @"Tendermint\tendermint.exe",
        Arguments = @"node --abci grpc --proxy_app tcp://127.0.0.1:5020 --home=Tendermint --log_level debug"
    });
}

This is the project file where .proto files are being processed, they are all generated successfully and work:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Grpc" Version="2.39.1" />
    <PackageReference Include="Grpc.AspNetCore" Version="2.39.0" />
    <PackageReference Include="Grpc.Tools" Version="2.39.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="LiteDB" Version="5.0.11" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\CommonLib.AspNet\CommonLib.AspNet\CommonLib.AspNet.csproj" />
    <ProjectReference Include="..\..\CommonLib\CommonLib\CommonLib.csproj" />
  </ItemGroup>

  <ItemGroup>
    <Protobuf Include="Source\Protos\gogoproto\gogo.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\crypto\keys.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\crypto\proof.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\types\block.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\types\canonical.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\types\events.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\types\evidence.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\types\params.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\types\types.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\types\validator.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\version\types.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
    <Protobuf Include="Source\Protos\tendermint\abci\types.proto" GrpcServices="Server" ProtoRoot="Source/Protos" />
  </ItemGroup>

  <ItemGroup>
    <Folder Include="Source\Database\" />
  </ItemGroup>

</Project>

Here is the service itself, it contains the example (linked above) translated to C#:

public class KVStoreService : ABCIApplication.ABCIApplicationBase
{
    private static LiteDatabase _env;
    private static ILiteCollection<KV> _store;

    public LiteDatabase Env => _env ??= new LiteDatabase(WebUtils.Configuration?.GetConnectionString("LiteDb"));
    public ILiteCollection<KV> Store => _store ??= Env.GetCollection<KV>("kvs");

    public override Task<ResponseEcho> Echo(RequestEcho request, ServerCallContext context)
    {
        return Task.FromResult(new ResponseEcho { Message = $"Validator is Running: {DateTime.Now:dd-MM-yyyy HH:mm}" });
    }

    public override Task<ResponseCheckTx> CheckTx(RequestCheckTx request, ServerCallContext context)
    {
        var (code, _) = Validate(request.Tx);
        return Task.FromResult(new ResponseCheckTx { Code = code, GasWanted = 1 });
    }
    
    public override Task<ResponseBeginBlock> BeginBlock(RequestBeginBlock request, ServerCallContext context)
    {
        Env.BeginTrans();
        Store.EnsureIndex(x => x.Key, true);
        return Task.FromResult(new ResponseBeginBlock());
    }
    
    public override Task<ResponseDeliverTx> DeliverTx(RequestDeliverTx request, ServerCallContext context)
    {
        var (code, kv) = Validate(request.Tx);
        if (code == 0)
            Store.Insert(kv);
        return Task.FromResult(new ResponseDeliverTx { Code = code });
    }

    public override Task<ResponseCommit> Commit(RequestCommit request, ServerCallContext context)
    {
        Env.Commit();
        return Task.FromResult(new ResponseCommit { Data = ByteString.CopyFrom(new byte[8]) });
    }

    public override Task<ResponseQuery> Query(RequestQuery request, ServerCallContext context)
    {
        var k = request.Data.ToBase64();
        var v = Store.FindOne(x => x.Key == k)?.Value;
        var resp = new ResponseQuery();
        if (v == null)
            resp.Log = $"There is no value for \"{k}\" key";
        else 
        {
            resp.Log = "KVP:";
            resp.Key = ByteString.FromBase64(k);
            resp.Value = ByteString.FromBase64(v);
        }

        return Task.FromResult(resp);
    }

    private (uint, KV) Validate(ByteString tx) 
    {
        var kv = tx.ToStringUtf8().Split('=').Select(kv => kv.UTF8ToBase64()).ToKV();
        if (kv.Key.IsNullOrWhiteSpace() || kv.Value.IsNullOrWhiteSpace())
            return (1, kv);
        
        var stored = Store.FindOne(x => x.Key == kv.Key)?.Value;
        if (stored != null && stored == kv.Value)
            return (2, kv);

        return (0, kv);
    }
}

Now if I create a simple client for my service and start both projects at the same time they will both work flawlessly:

public class Program
{
    public static async Task Main()
    {
        LoggerUtils.Logger.Log(LogLevel.Info, $@"Log Path: {LoggerUtils.LogPath}");
        using var channel = GrpcChannel.ForAddress("http://localhost:5020");
        var client = new ABCIApplication.ABCIApplicationClient(channel);

        var echo = string.Empty;
        while (echo.IsNullOrWhiteSpace())
        {
            try
            {
                echo = client.Echo(new RequestEcho()).Message;
            }
            catch (Exception ex) when (ex is HttpRequestException or RpcException)
            {
                await Task.Delay(1000);
                LoggerUtils.Logger.Log(LogLevel.Info, "Server not Ready, retrying...");
            }
        }
        
        var beginBlock = await client.BeginBlockAsync(new RequestBeginBlock());
        var deliver = await client.DeliverTxAsync(new RequestDeliverTx { Tx = ByteString.CopyFromUtf8("tendermint=rocks") });
        var commit = await client.CommitAsync(new RequestCommit());
        var checkTx = await client.CheckTxAsync(new RequestCheckTx { Tx = ByteString.CopyFromUtf8("tendermint=rocks") });
        var query = await client.QueryAsync(new RequestQuery { Data = ByteString.CopyFromUtf8("tendermint") });

        System.Console.WriteLine($"Echo Status: {echo}");
        System.Console.WriteLine($"Begin Block Status: {(beginBlock != null ? "Success" : "Failure")}");
        System.Console.WriteLine($"Delivery Status: {deliver.Code switch { 0 => "Success", 1 => "Invalid Data", 2 => "Already Exists", _ => "Failure" }}");
        System.Console.WriteLine($"Commit Status: {(commit.Data.ToByteArray().SequenceEqual(new byte[8]) ? "Success" : "Failure")}");
        System.Console.WriteLine($"CheckTx Status: {checkTx.Code switch { 0 => "Success", 1 => "Invalid Data", 2 => "Already Exists", _ => "Failure" }}");
        System.Console.WriteLine($"Query Status: {query.Log} {query.Key.ToStringUtf8()}{(query.Key == ByteString.Empty || query.Value == ByteString.Empty ? "" : "=")}{query.Value.ToStringUtf8()}");

        System.Console.ReadKey();
    }
}

Server working:

Client working:

So far so good, however as soon as I plug Tendermint into the pipeline (so clients can call my app through it), for some reason, I am getting this:

Echo failed - module=abci-client connection=query err="rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR"`

It happens regardless how I start the process and the error repeats. Obviously, since Tendermint fails to connect to the proxy app, it is not callable either:

The errors posted above are all I get with logs set to debug (including GRPC logs).

"Logging": {
  "LogLevel": {
    "Default": "Debug",
    "Microsoft": "Debug",
    "Microsoft.Hosting.Lifetime": "Debug",
    "Grpc": "Debug"
  }

Similar threads point out to the problem with metadata (like \n in it) but I don't know how that would be relevant to my particular problem, you can go here for reference.

I am pretty sure it is a case of simple misconfiguration somewhere that prevents Tendermint from talking to the actual app, but I can't figure it out. Any help would be appreciated.

Answer 1

[@artur's] comment got me thinking and I have finally figured it out. Actually, even before I posted this question, my first thought was that this should indeed be http, despite the documentation saying otherwise, but no, http://127.0.0.1:5020 wouldn't work. So I tried to put it in .toml file instead, I have even tried with https, but also without luck. Trying with http didn't throw any errors, unlike in the case when address was preceeded with tcp, it was just hanging on Waiting for Echo message (similarly to when pointing to the wrong address, which was weird). I've been always, eventually reverting to the tcp version. The solution was simple, remove protocol altogether...

The documentation doesn't give any clues, so for completion, at least when working with C# (.NET 5), there are 3 things that you HAVE TO DO to make it work, all of them are trivial but you have to figure them out by yourself first:

1. Remove protocol from your configuration when pointing to the proxy app: tcp://127.0.0.1:<port> should be 127.0.0.1:<port> and YES, it will throw regardless if you have protocol specified in the .toml file or as a flag in the console.
2. The flag is --proxy_app NOT --proxy-app.
3. Additionally to following the tutorial, you also have to EXPLICITLY override and implement Info(), Echo() and InitChain(), otherwise it will throw an Unimplemented Exception.

Since my understanding of Tendermint is still scarce, the initial approach had some design issues. Here is the code for anybody facing similar problems:

https://github.com/rvnlord/TendermintAbciGrpcCSharp