UnauthorizedAccessException while trying to rename file

Question

I'm trying to create a program that generates random file prefixes and renames all files in a directory. It runs perfectly fine in most cases, but after multiple successful tests, I sometimes encountered a strange UnauthorizedAccessException. For example, the exception is thrown every time the directory is located on the desktop. Weirdly, there are also cases in which the renaming process is successful, but after trying it again in the same directory (after restoring the old file names; so like a fresh start), the exception is thrown. This is how I try to rename the files:

    var files = Directory.GetFiles(dir);      
    int max = files.Length, success = 0;

    foreach (var filePath in files) {
        var file = Path.GetFileNameWithoutExtension(filePath);
        var prefix = GenerateRandomPrefix(_runtimeSettings.Pattern, _runtimeSettings.CharList, _runtimeSettings.ReplacementLength);
        var renamedFile = filePath.Replace(file, string.Concat(prefix, file));

        try {
            File.Move(filePath, renamedFile);
            success++;
        } catch (Exception ex) {
            return;
        }
    }

I've already checked multiple things, like if the directory or any file is read-only. I've also tried running in administrator mode, but to my surprise, not even that fixes the occurrence of the UnauthorizedAccessException. According to the docs of File.Move(), it explicitly says:

UnauthorizedAccessException: The caller does not have the required permission.

So at least running in administrator should fix the error, but why does it persist?