Child Item

parallel upload failing - "lookup_failed/incorrect_offset"

we need to upload large files (several GiB) to dropbox using the API. i've written a script that uses cURL to do this in parallel. i upload the first segment of 150M in order to obtain the session ID, then upload a number of additional segments in parallel, then close the session once the parallel threads have all finished. this doesn't work. i get the error "lookup_failed/incorrect_offset" followed by what appears to be an arbitrary number of dots (usually one to three). i'm puzzled by this because i've verified the offsets and they are correct.

here is some sample debug output from an attempt to upload a 1Gb file using several parallel streams concurrently, with sensitive information redacted :

opening :
curl -x <redacted> -X POST https://content.dropboxapi.com/2/files/upload_session/start -H Content-Type:application/octet-stream -H Authorization:Bearer <redacted> -H Dropbox-API-Arg: {"close":false} --data-binary @-

parallel thread 1 :
curl -x <redacted> -X POST https://content.dropboxapi.com/2/files/upload_session/append_v2 -H Content-Type:application/octet-stream -H Authorization:Bearer <redacted> -H Dropbox-API-Arg: {"cursor":{"session_id":"<redacted>","offset":157286400},"close":false} --data-binary @-

parallel thread 2 :
...Dropbox-API-Arg: '{"cursor":{"session_id":"<redacted>","offset":314572800},"close":false}'

parallel thread 3 :
...Dropbox-API-Arg: '{"cursor":{"session_id":"<redacted>","offset":471859200},"close":false}'

parallel thread 4 :
...Dropbox-API-Arg: '{"cursor":{"session_id":"<redacted>","offset":629145600},"close":false}'

parallel thread 5 :
...Dropbox-API-Arg: '{"cursor":{"session_id":"<redacted>","offset":786432000},"close":false}'

closing :
curl -x <redacted> -X POST https://content.dropboxapi.com/2/files/upload_session/finish -H Content-Type:application/octet-stream -H Authorization:Bearer <redacted> -H Dropbox-API-Arg: {"cursor":{"session_id":"<redacted>","offset":943718400},"commit":{"path":"/par1.mp2","mode":"overwrite","autorename":false,"mute":false}} --data-binary @-

so it seems the offsets are, indeed, correct. i've also verified that the session id returned after the session is opened (redacted in the dump above) remains unchanged throughout the session.

is a parallel upload simply not possible with the dropbox API ?

Find more posts tagged with

API

Abuse

Comments

Greg-DB

By default, upload sessions only accept the data sequentially, meaning the app needs to only upload the pieces of the file for an upload session one by one in order.

You can upload in parallel if you want, but you need to set session_type:concurrent and follow a few other constraints. Please refer to the /2/files/upload_session/start documentation for information on using that.

xtolchinskii

thanks @Greg-DB, i missed that in the docs. this looks promising, i'll give it a try.

one question : since /upload_session/finish in a concurrent upload won't accept file data along with it, what should the offset be in the cursor data for this case ? the total number of bytes already sent with previous calls (i.e. the file size) ? 0 ? ...or maybe it's ignored ?

Greg-DB

Yes, for the offset on the /finish call for a concurrent upload session you should supply the total number of bytes successfully sent for the session previously (i.e., the total length of the file).