Event Store - Filter out any deleted streams from projections

Question

I have deleted some streams but I'm still getting those streams when my projection starts.

Is there a way to filter out any deleted streams from projections?

I know that the streams have been deleted because a StreamDeletedException exception is being raised when I use code like this:

var events = _eventStoreConnection.ReadStreamAsync(
                    Direction.Forwards,
                    e.Event.EventStreamId,
                    StreamPosition.Start);
  await foreach (var @event in events)
  {
  }

Answer 1

Your question needs a lot more context than what you provided.

A bit of clarification. Projections don't store streams. Projections don't store events either. Projections are computation that happens to emit events to stream(s).

It seems you are relying on $streams or something similar that linked events from streams you deleted. The distinction is important because what is stored in streams like $streams are actually links that point to events belonging to different streams. As such, when streams got deleted, the links that point to their events are not deleted.

In the case of $streams, what is stored is links to each stream first event. Links are events with a payload following this pattern {version_number}@{stream_name} and with the $> event type.

In EventStoreDB you can't delete events as you would delete a row in an SQL database, but you can delete streams (which you can see as tables if it helps)

Answer 2

The reason why you receive events from deleted streams in your subscription is because, as Yo wrote, events aren't deleted as such when you delete a stream.

If you subscribe to a regular stream, or a projection stream (like category), you should not get events from deleted streams. When subscribing to streams, which contain links, like category streams, you need to set resolveLinkTos to true.

If you, however, subscribe to $all, you will get events from deleted streams, if you start from the beginning. It's because when reading from $all we read directly from the event log, where events are still present. The issue is known, and we plan to solve it, but for now it's how it is. It's not causing a lot of issues because subscriptions to $all usually run in real-time mode and deletions aren't relevant in this case. However, when reading from the start, you indeed hit this issue.

When you run scavenging, events for deleted streams, truncated streams, also events that expired according to stream TTL and max count, will be deleted. There's an issue there too since EventStoreDB is unable to scavenge the currently active (last) chunk. It's because events are being written to this chunk, so it cannot be closed and recreated (that's how scavenging works).

The scavenging issue occurs when you have small volumes (dev, testing) and the active chunk remains the only chunk, so it never gets scavenged.