C# Incremental Generator - How I can read additional files? AdditionalTextsProvider not working as expected

Question

I am trying to get some values from the appsettings.json. But whatever I try with the AdditionalTextsProvider doesn't work. Here is my code

IncrementalValuesProvider<AdditionalText> textFiles = context.AdditionalTextsProvider.Where(static file => file.Path.Contains("appsettings.json")); // tried many things here, like EndsWith(".json") etc..
IncrementalValuesProvider<(string name, string content)> namesAndContents = textFiles.Select((text, cancellationToken) => (name: Path.GetFileNameWithoutExtension(text.Path), content: text.GetText(cancellationToken)!.ToString()));

context.RegisterSourceOutput(namesAndContents, (spc, nameAndContent) =>
    {
         nameAndContent.content; //always empty 
         nameAndContent.name; //always empty 
    });

From the other hand, when I implement the ISourceGenerator (same solution, same projects) this line of code just works!

var file = context.AdditionalFiles.FirstOrDefault(x => x.Path.Contains("appsettings.json"));

The project which is referencing the code generator:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Dapper" Version="2.0.123" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="6.0.4" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="6.0.3" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="6.0.3" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="6.0.3">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Scrutor" Version="4.1.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.3.0" />
  </ItemGroup>
  <ItemGroup>
        <ProjectReference Include="..\myproject\myproject.csproj" />
    <ProjectReference Include="..\myproject.EFCore\myproject.EFCore.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
  </ItemGroup>

  <ItemGroup>
    <AdditionalFiles Include="appsettings.json" />
  </ItemGroup>

</Project>

Code generator project :

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <LangVersion>latest</LangVersion>    
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild> <!-- Generates a package at build -->
    <IncludeBuildOutput>false</IncludeBuildOutput> <!-- Do not include the generator as a lib dependency -->
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.1.0" PrivateAssets="all" />
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.3" PrivateAssets="all" />
  </ItemGroup>
  <ItemGroup>
    <!-- Generator dependencies -->
    <PackageReference Include="Newtonsoft.Json" Version="13.0.1" GeneratePathProperty="true" PrivateAssets="all" />
  </ItemGroup>
 <PropertyGroup>   <GetTargetPathDependsOn>$(GetTargetPathDependsOn);GetDependencyTargetPaths</GetTargetPathDependsOn>
  </PropertyGroup>
  <Target Name="GetDependencyTargetPaths">
    <ItemGroup>
      <TargetPathWithTargetPlatformMoniker Include="$(PKGNewtonsoft_Json)\lib\netstandard2.0\Newtonsoft.Json.dll" IncludeRuntimeDependency="false" />
    </ItemGroup>
  </Target>
    <ItemGroup>
    <!-- Package the generator in the analyzer directory of the nuget package -->
    <None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />
    <!-- Package the props file -->
  </ItemGroup>
</Project>

Answer 1

If you combine the additional text provider with the compilation it works for me (inspired by this great post):

public void Initialize(IncrementalGeneratorInitializationContext context)
{
    var files = context.AdditionalTextsProvider
        .Where(a => a.Path.EndsWith("appsettings.json"))
        .Select((a, c) => (Path.GetFileNameWithoutExtension(a.Path), a.GetText(c)!.ToString()));

    var compilationAndFiles = context.CompilationProvider.Combine(files.Collect());
            
    context.RegisterSourceOutput(compilationAndFiles, (productionContext, sourceContext) => Generate(productionContext, sourceContext));
}

void Generate(SourceProductionContext context, (Compilation compilation, ImmutableArray<(string, string)> files) compilationAndFiles)
{
    //emit generated files...
}

Update:

Tested again with Visual Studio 17.4. and combination with compilation seems not to be necessary anymore. Generator is called as expected when appsettings.json changes with only AdditionalTextsProvider registered.

Answer 2

"AdditionalFiles" in your consuming project file is the key to ensuring these files are captured by your source/incremental generator.

Found out that using a "<AdditionalFiles Include='...' />" is not necessary in some projects such as ASPNET Core projects for Source/IncrementalGenerators. The .cshtml files are included as additional files by default.

Also, file globbing patterns work with the "Include" parameter.

Answer 3

I have tried replicate but everything works for me.

I define my source generator as this

namespace ConsoleAppSourceGenerator
{
    [Generator]
    public class TxtGenerator : IIncrementalGenerator
    {
        private static int counter;

        public void Initialize(IncrementalGeneratorInitializationContext initContext)
        {
            IncrementalValuesProvider<AdditionalText> textFiles = initContext.AdditionalTextsProvider
                .Where(static file => file.Path.EndsWith(".txt"));

            IncrementalValuesProvider<(string name, string content)> namesAndContents = textFiles
                .Select((text, cancellationToken) => (name: Path.GetFileNameWithoutExtension(text.Path), content: text.GetText(cancellationToken)!.ToString()));

            initContext.RegisterSourceOutput(namesAndContents, (spc, nameAndContent) =>
            {
                spc.AddSource($"TxtFile.{nameAndContent.name}.g.cs", $@"
// Counter {Interlocked.Increment(ref counter)}
public static partial class TxtFile
{{
    public const string {nameAndContent.name} = ""{nameAndContent.content}"";
}}");
            });
        }
    }
}

and source generator project as this

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
      <Nullable>enable</Nullable>
      <LangVersion>Latest</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.3">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.1.0" />
  </ItemGroup>
</Project>

Then in another project, where I have made a dummy foo.txt file I have

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
      <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
      <CompilerGeneratedFilesOutputPath>Generated</CompilerGeneratedFilesOutputPath>
  </PropertyGroup>

    <ItemGroup>
        <Compile Remove="$(CompilerGeneratedFilesOutputPath)/**/*.cs" />
    </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\ConsoleAppSourceGenerator\ConsoleAppSourceGenerator.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
  </ItemGroup>
    
    <ItemGroup>
        <AdditionalFiles Include="foo.txt"/>
    </ItemGroup>

</Project>

If I through Visual Studio look at Dependencies->Analyzers->ConsoleAppSourceGenerator>TxtFile.foo.g.cs then it correctly increase the counter on every change I to in the file (see step 9).

I store generated files in path ConsoleApp/Generated/ConsoleAppSourceGenerator/ConsoleAppSourceGenerator.TxtGenerator and whenever I rebuild the project it correctly updates the files.

Have you remembered to add the [Generator] attribute to the IIncrementalGenerator?

Could you test this out and validates this doesn't either work for you?