Large file uploads: second uploads, intermittent uploads, slice uploads

File uploading is an old topic. For relatively small files, you can just convert the file to a stream of bytes and upload it to the server, but for larger files, uploading them in the normal way is not a good idea.convert scanned pdf to word online free large files After all, few people will put up with the unpleasant experience of uploading a file halfway through and then starting the upload all over again. Is there a better upload experience? The answer is that there are several ways to upload

1. Second Upload

Generally, if you upload something you want to upload, the server will do MD5 verification first. If the same thing is on the server, it will give you a new address directly. In fact, you are downloading the same file on the server that you want to transfer in seconds. Actually, as soon as you change the MD5, you have to modify the file itself (the name cannot be changed). For example, if you add a few more words in a text file, the MD5 will change, and will not be transmitted in a few seconds.

This article implements the second transfer core logic

Use redis's set method to store the file upload status, where key is the md5 of the file upload, and value is the flag bit of whether the upload is complete.

When the flag bit is true, the upload can be completed, if there is the same data file uploaded, it will enter the second transmission logic. If the flag bit is false, it means we haven't finished uploading, at this time we need to call the set method to save the path of the block number file information record, where the key is the md5 of the uploaded file plus a fixed prefix, and the value is the development path of the block number file management record.

2. Segmented upload

Segmented upload is to divide the whole file into several data blocks (we call it Part) according to a certain size and upload them separately, after the upload is completed, the server will aggregate all the uploaded files into the original file.

Scenarios for segmented upload: large file upload, bad network environment, risk of retransmission.

3. Breakpoint Relay

Breakpoint relay is when download or upload, download or upload task (file or zip) artificially divided into several parts, each part has a thread for uploading or downloading, in the case of network failure, you can continue to upload or download the unfinished part from the part that has been uploaded or downloaded, without the need to start from the beginning of the upload or download. In this paper, breakpoint deferred uploading is mainly for breakpoint uploading scenarios.

Application scenarios and breakpoint extension can be seen as derivatives of segmented upload, so segmented upload scenarios and breakpoint extension can be used.

To realize the core business logic of breakpoint delayed uploading, in the process of segmented uploading, if the uploading is interrupted due to the impact of abnormally important factors such as information system crashes or network interruptions, the client needs to record the progress of uploading at this time. When the society supports uploading again afterward, it can continue to upload from the place where the last upload was interrupted.

In order to avoid the problem of the client's progress data being deleted after uploading, the server can also provide corresponding interfaces for the client to query the uploaded data, so that the client knows that the data has been uploaded to the fragment, and then continues to upload from the next fragment data.