There are many questions on checking finding a GUID in a list etc. But I could not find any for just determining if a message was seen before or not.
I have an API which receives requests with a message-id GUID field in them. I want to drop any message-id that has been received before. The simplest option would be to keep a hashset of the last e.g. 1000 message IDs and check if new ID is found. However, this creates a trade-off between the size of the list and the additional added delay.
I was wondering if there is an algorithm out there that behaves like a branch predictor where for most messages it can confidently say a GUID of specific length has not been encountered, or with high probability that it was encountered before (so that we can limit the lookup to only messages that hit this high probability). Does such a thing exist? Is it theoretically possible?