So, generally, uploading a single large zip file is faster than uploading 1,000 individual files of the same total size (10 GB in this case). Here’s why:
- Fewer Connection Overheads: Every time you upload a file, there is an overhead associated with establishing and closing a connection to the server. With 1,000 individual files, this overhead happens 1,000 times. With a single zip file, it only occurs once, which reduces the time spent on these connection setups and closures.
- Network Efficiency: Uploading a single file allows for a continuous data transfer, often at a more consistent speed. Multiple files may lead to network interruptions and fluctuations, as each file can experience delays when starting and ending.
- Reduced Metadata Overhead: For each file uploaded, metadata (like file permissions and timestamps) must also be transferred and processed. A single zip file only has one set of metadata, while 1,000 files have 1,000 sets of metadata, adding extra data that has to be processed during the upload.
- Parallelization Limitations: Many systems limit the number of files that can be uploaded in parallel. Uploading thousands of files individually could strain the system, especially with network latency and processing limitations, whereas a single file avoids this issue.
- Compression Benefits: Zipping can compress the data, which might reduce the file size slightly, depending on the data type. This reduction, even if minor, can still contribute to a faster upload time.
It would be great to be able to upload a zip of all my data instead of thousands of individual images, especially over slow internet (staying at a hotel).
