Why are transfer times for several small files slower than one big file?

Kushagra June 25, 2011
Pinterest Stumbleupon Whatsapp

Why is transferring many small files (e.g. wallpapers or photos) slower than the transfer of a big file of the same size.

  1. Doug Mielke
    November 29, 2011 at 11:10 pm

    Is it possible that when you say "transfer", you mean over a network between different locations?  

    If so, it's due to the transfer management overhead that is devoted to every individual file as it's transferred compared to the same amount of overhead to deal with a single file (say zipped) of the same size.
    We make software called AIRShip for sending data securely, reliably and fast. 
    Very recently we added new code that deals with this type of issue without the need to "pack" a folder or directory.

    The result is AIRShip now moves an "open" folder of individual files faster than the same folder packed. 

    In fact, our benchmarking says: 100% more efficient than the same folder zipped and about 400% better than the same folder with our old code (that was pretty fast as it was!)

    If you are interested, you can contact me.

    Doug Mielke
    818 321 3275
    doug.mielke@useairware.com

  2. Wayne
    September 13, 2011 at 10:28 am

    In short, due to indexing/memory. 
    Simple explanation:
    While transferring, the computer has to keep an index of all the files it is copying. Each one would have a separate index or so. This creates fragmentation and such.

    Real-life scenario:
    Imagine you're moving houses. Wouldn't it be easier to just have one big box with all the things in it rather than 5 separate boxes? Having separate boxes means you would have to keep track of all those little items in it and things can go wrong easily.

    Anyway, that's just my quick insight. Might be worth reading other people's comments if you're more interested

  3. Anonymous
    June 25, 2011 at 8:29 pm

    if one or more files in those small files are corrupted, locked by windows then this can explian why the rate is slower than compared to the single larger files.

    Also if those small files are located in different directories rather than in one directory then this can explain why the rate will be slower.

    transfert files out in the game cpu and ram usage.Win 7 applies high amounts of RAM to obtain these accelerated file transfer rates

    if you have some problems then try TeraCopy or extreme copy and see hw will go in both cases.

  4. Mike
    June 25, 2011 at 6:32 pm

    There are various reasons...

    Assuming your Hard Drive is rather well defragmented copying a single large file will be a continuous read with very little movement for the heads while various small files are more likely to be spread all over it causing the head to do a lot of jumping for every file.

    Also copying files is more than reading it's data at one place and write it to another. There are metadata (security options, file attributes, ...) which needs to be read from the source, written to the target and centralized data like the Master File Table needs to be updated for every single file.

    If you copy files to another File System there are also other operations necessary for example FAT32 doesn't support all attributes and metadata NTFS and has a different cluster size. These are things the system has to take care of too.

    If you copy files to another system or volume over the network additional things come into play like validation, etc.

    This is why a common backup strategy is to compress all data locally [into a single large compressed or image file] and then transfer this one big file over the network to the storage.