file transfer bewteen two windows machine,but the files will be deleted immediately

Question

Two Windows A and B There is a directory D on top of A, which can be accessed by B. There are always 25M files generated under directory D, which will be deleted in about 100-200ms. How can all the files generated under the D directory of machine A be fully transferred to machine B?

The current method of scanning directory D in a loop on B will always be lost because transferring files also takes a certain amount of time, and the interval between scanning directories also takes time.i tried FileSystemWatcher with C#, but some files also will be lost. And I don't know if the newly created file in directory D is complete

Here's my code. I tried files cache('C:\test3') for directory D('C:\test2') on machine A

using System;
using System.Collections.Concurrent;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace DirListener
{
    internal class Program
    {


        static ConcurrentQueue<string> _queue = new ConcurrentQueue<string>();
        public static void Main(string[] args) {

            Task.Run(() => TransferFile());

            string directoryPath = @"C:\test2";
            FileSystemWatcher watcher = new FileSystemWatcher(directoryPath, "*.*")
            {
                NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.LastAccess | NotifyFilters.Attributes,
                IncludeSubdirectories = false
            };
            watcher.Created += Watcher_Created;
            watcher.Changed += Watcher_Changed;
            watcher.EnableRaisingEvents = true;
            Console.ReadLine();
        }

        public static void TransferFile()
        {
            while (true)
            {
                try
                {
                    if (_queue.TryDequeue(out string path))
                    {
                        File.Copy(path, $"C:\\test3\\{Path.GetFileName(path)}", true);
                    }
                    else 
                    { 
                        Thread.Sleep(10);
                    }
                } catch (Exception ex)
                { 
                }
            }
        }

        private static void Watcher_Changed(object sender, FileSystemEventArgs e)
        {
            _queue.Enqueue(e.FullPath);
        }

        private static void Watcher_Created(object sender, FileSystemEventArgs e)
        {
            _queue.Enqueue(e.FullPath);
        }
    }
}

Answer 1

I would MOVE the folder to a temp folder, somewhere on the same physical drive (so there isn't any copying that takes time), process the files, and then delete the temp folder.

Answer 2

Can you share your example ? I have a C# script to manage and check if files are ready, and if not, loop again. You could compare to your version and adjust to your demand.

using System; using System.IO; using System.Threading; using System.Threading.Tasks;

class FileManager
{
    static void Main()
    {
        string directoryPath = @"C:\Path\To\DirectoryD";
        FileSystemWatcher watcher = new FileSystemWatcher();
        watcher.Path = directoryPath;
        watcher.Filter = "*.*";
        watcher.Created += OnFileCreated;
        watcher.EnableRaisingEvents = true;

        Console.WriteLine($"Monitoring directory: {directoryPath}");
        Console.WriteLine("Press Enter to exit.");
        Console.ReadLine();

        watcher.Dispose();
    }

    private static async void OnFileCreated(object sender, FileSystemEventArgs e)
    {
        try
        {
            await WaitForFileReady(e.FullPath);
            TransferFile(e.FullPath);
            Console.WriteLine($"File transferred: {e.Name}");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error transferring file: {ex.Message}");
        }
    }

    private static async Task WaitForFileReady(string filePath)
    {
        const int maxAttempts = 10;
        const int delayBetweenAttempts = 100;

        for (int i = 0; i < maxAttempts; i++)
        {
            try
            {
                using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.ReadWrite, FileShare.None))
                {
                    return;
                }
            }
            catch (IOException)
            {
                await Task.Delay(delayBetweenAttempts);
            }
        }

        throw new InvalidOperationException($"Timeout waiting for file to be ready: {filePath}");
    }

    private static void TransferFile(string filePath)
    {
        string destinationPathOnMachineB = @"\\MachineB\DestinationPath\" + Path.GetFileName(filePath);
        File.Copy(filePath, destinationPathOnMachineB, true);
        File.Delete(filePath);
    }
}