Can't access contents of a Security-Scoped directory URL

Question

I have the following code to enumerate through a directory. The URL to the directory is passed in as a parameter to the function.

The startAccessingSecurityScopedResource returns true at the start of the function, however when I start to enumerate it returns false for all the following files and subdirectories inside the original folder selected by the user.

This is the code for the function.

func readDirectory(file: URL){
    
    let localFileManager = FileManager()
    let resourceKeys = Set<URLResourceKey>([.nameKey, .isDirectoryKey])
    let directoryEnumerator = localFileManager.enumerator(
        at: file,
        includingPropertiesForKeys: Array(resourceKeys),
        options: .skipsHiddenFiles
    )!

    
    for case let fileURL as URL in directoryEnumerator {
        guard let resourceValues = try? fileURL.resourceValues(forKeys: resourceKeys),
            let isDirectory = resourceValues.isDirectory,
            let name = resourceValues.name else {
            continue
        } 
    }
}

From the above code, the file parameter passed does return true for access. However, in the directoryEnumerator loop, the fileURL doesn't have access.

Is there a reason that I am losing access to the subdirectories and files inside the original directory that it says I do have access to?