Branded string as key for dictionary

Question

Similarly to question TS branded string as key of the object I would like to use branded strings as keys for my dictionary but then take it one step further. In my use case the branded type of the key is related to the object saved under that key.

type HandlerA = { type: "A" };
type HandlerB = { type: "B" };

type HandlerVariant = HandlerA | HandlerB

type Id<T extends HandlerVariant> = string & { __type: T["type"] };

type HandlerDictionary = {
    [id: string]: HandlerVariant;
}

By using a Map type I could enforce the usage of my branded Id type for key of the dictionary like below

type HandlerMap<T extends HandlerVariant> = Map<Id<T>, T>;

This unfortunately means that this whole map would have to be of one of the handler types or the union HandlerVariant type which has almost same effect as just using the dictionary from the first example.

What I would like to be able to do is depending on the brand, deduce correct type of the handler under it like this

const dict = {} as HandlerDictionary;
const key = "" as Id<HandlerA>;
const handler = dict[key];

Since key is branded with HandlerA type, therefore under that key should be an object of type HandlerA but typescript sadly deduces handler to be of type HandlerVariant.

Is there a way to type this dictionary so index access is aware of the key brand/value relationship? I know there are ways to do this by implementing a getter method but in my case that would mean a substencial refactoring efforts.

// Invalid TS code example of the relationship
type BrandedHandlerDictionary = {
    <T extends HandlerVariant>[id: Id<T>]: T;
}