We appear to have encountered an issue within a Service Fabric cluster where by the state of an Actor service has grown to the point where the Temporary Storage (D:) drive had filled up. As I understand, Actor state and reliable collection state is persisted to to disk on this drive. For one service we had amassed 190 GB of space taken up by the ActorStateStore file.
We were working on the assumption that space was taken up by a lot of stale actors that we no longer needed in our system, so we added a call to the service to purge the unwanted actors using the mechanism detailed here (https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-actors-lifecycle).
ActorId actorToDelete = new ActorId(id);
IActorService myActorServiceProxy = ActorServiceProxy.Create(
new Uri("fabric:/MyApp/MyService"), actorToDelete);
await myActorServiceProxy.DeleteActorAsync(actorToDelete, cancellationToken)
We cycled through the full list off actors in state and called the delete on anything we didn't want to keep, which would have been the vast majority of them, expecting this to reduce our space consumption on the temporary storage drive. However, the space did not seem to free up. Does anyone know what the lifecycle of this process is? Are there lead times before we should expect the drive space to appear free?
Is there a mechanism that can be used to free this drive space up or what is the best way to perform housekeeping to remove old actors that we would no longer be interested in?