Can I retrieve Custom Metadata from an incomplete multipart upload?


If yes, how?

Context: I am implementing multipart upload with resuming ability. After an upload interruption, the user re-selects the same file from his drive (or maybe from another machine). I need to match that file against one of the existing incomplete UploadIds, if there is one for that file. Goal is to re-use that UploadId to resume the upload. How can I do this if the end-client has no idea of the UploadId?

I cannot fully rely on file names. It is possible that the user has changed the file name after the interruption, or that he has another file with the same file name (collision).

ListMultipartUploads gives me all the UploadId and the Key of incomplete uploads. But not any custom metadata that I have defined when using CreateMultipartUpload. I tried to to a headObject request to the Key of the upload to get the custom metadata, but I get a 404 (understandably).

Ideally I would do some sort of HeadObject request against the UploadId in order to get the custom metadata, in which I would have stored information that helps me identify the file (say a file checksum, or file checksum + user hash).

I don't think I can use ETags, since S3 calculates ETags as a checksum of the parts checksums, so S3 has no idea of the final ETag until the end, and cannot give it back to me when enquiering about the incomplete UploadId.

So the only option I can see is:

  • to use the Key as a match. I would need to name the file as a predictible checksum of the whole file itself, and name it this way. It feels wrong. Custom metadata feel more like the place for this.
  • maintain somewhere a key-value records matching my checksum and UploadId. Make sure to delete them when UploadId are aborted or completed... more complicated, also when using aborting rules with the bucket lifecycle

Am I doing this wrong, any ideas? Any way to access custom metadata form incomplete multiparts uploads? The data is somewhere, since the custom metadata is applied to the final object after the final CompleteMultipartUpload request is processed.

Thank you!

asked 4 months ago250 views
1 Answer

This is definitely an interesting question that I haven't seen yet before. I don't personally know a great answer, but did a quick search and it appears that boto3 may have the ability to tag the metadata in parts. I also came across an SDK doc for customizing multi-parts (PHP).

I hope this steers you towards the right direction and good luck!

P.S. Don't forget to setup a Lifecycle rule to delete off those incomplete multi-part uploads.

profile pictureAWS
answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions