Uploading large files using chunks

newdbuser

Hello: I am using this Python code to upload a large zip file (2-4 GB). I am trying to upload the file in chunks, however I am getting a "dropbox.exceptions.ApiError('some_random_numbers', UploadSessionLookupError(u'incorrect_offset', UploadSessionOffsetError(correct_offset=540)))

 

Here is my code that is uploading the zip file using chunks (from StackOverflow)

 

 

f = open(outputName)
file_size = os.path.getsize(outputName)

CHUNK_SIZE = 10000

if file_size <= CHUNK_SIZE:

	print dbx.files_upload(f.read(), backupPath)

else:

	upload_session_start_result = dbx.files_upload_session_start(f.read(CHUNK_SIZE))
	cursor = dropbox.files.UploadSessionCursor(session_id=upload_session_start_result.session_id,
												   offset=f.tell())
	commit = dropbox.files.CommitInfo(path=backupPath)

	while f.tell() < file_size:
		if ((file_size - f.tell()) <= CHUNK_SIZE):
			print dbx.files_upload_session_finish(f.read(CHUNK_SIZE),
									cursor,
									commit)
		else:
			dbx.files_upload_session_append(f.read(CHUNK_SIZE),cursor.session_id,cursor.offset)
			#dbx.files_upload_session_append_v2(f.read(CHUNK_SIZE), cursor)
			cursor.offset = f.tell()

 

 I tried v2 for files_upload_session_append as well. Both doesn't work. Also tried different CHUNK_SIZEs, but doesn't work either. Any suggestions?

 

Greg-DB

An incorrect_offset error indicates that you're trying to upload to an offset that doesn't match what the server last saw for this upload session. For instance, this can happen if previous request was received and processed successfully but the client did not receive the response, e.g. due to a network error. The correct_offset value tells you where you should resume the upload from.

In this case, you're using chunk size of 10000 bytes, so it seems you're not actually uploading an entire chunk, since correct_offset is 540.

To help debug this, I recommend printing out the length of data you're getting out of f.read each time to check that you're uploading as much as you expect to.