Child Item

Dropbox API: Connection broken: IncompleteRead

Hi everyone,

I am using the API to zip and download folders each with 1000 .jpegs in them (approx 4GB - 5GB). Historically this has been no issue. In the last week however, I have started receiving connection timeouts. The error reads: "Connection broken: IncompleteRead", followed by a file specific: "(2013265920 bytes read, 2153556605 more expected)', IncompleteRead(2013265920 bytes read, 2153556605 more expected))".

I have tried downloading on multiple devices, using different Dropbox accounts and in various locations with different IP addresses and internet connections. The error is occuring around 60% of the time. This means that the downlaod process is taking dramatically longer than it should. As I mentioned, this was not an issue until recently.

Any help greatly appreciated.
M

Find more posts tagged with

API

Abuse

Comments

Greg-DB

Thanks for the report. Can you let me know:

Are you referring to using the /2/files/download_zip endpoint?
Does it always fail after 2013265920 bytes, or does that number vary?
Does it always fail after a specific amount of time? If so, how long is that?

denseishin

I'm facing a similar problem, but in my case I'm merely downloading a very large file (~125 GB). It usually crashes between 58000000000 - 49000000000 bytes read.

DB-Des

Hi @denseishin

I'll be happy to look further into this issue for you, please reply with:

the name and version number of the platform and SDK/library you are using, if any
the steps to reproduce the issue, including relevant code snippet(s), but don't
include any access or refresh token(s)
the full text of any error or unexpected output

Could you also confirm if requests are failing after a specific amount of time? And if so, how long is that?

denseishin

Python 3.8.10 on Ubuntu 20.04 with the Python dropbox package version 11.36.2

code snippet:

   def __init__():
       self.dbx = dropbox.Dropbox(API keys etc. here)
   def download(self, info: dict):
        dbx_path = info["path"]
        self.logger.log(logging.INFO,f"Downloading {dbx_path}")
        noroot = dbx_path.lstrip('/')
        dl_path = self.dl_path / pathlib.Path(noroot)
        dl_path.parent.mkdir(exist_ok=True, parents=True)
        metadata = self.dbx.files_download_to_file(str(dl_path), dbx_path)
        return metadata

INFO:dbx_downloader:Downloading (file name redacted)
INFO:dropbox:Request to files/download
ERROR:modulenameredacted:('Connection broken: IncompleteRead(54876571369 bytes read, 78618595296 more expected)', IncompleteRead(54876571369 bytes read, 78618595296 more expected))

I couldn't test for the amount of time yet

denseishin

Here's a traceback.

INFO:redacted_module:2025-02-18 16:53:13.153338+00:00: Downloading REDACTED
INFO:dropbox:Request to files/download
ERROR:redacted_module:('Connection broken: IncompleteRead(55015292051 bytes read, 78479874614 more expected)', IncompleteRead(55015292051 bytes read, 78479874614 more expected))
Traceback: Traceback (most recent call last):
  File "/path/to/.venv/lib/python3.8/site-packages/urllib3/response.py", line 748, in _error_catcher
    yield
  File "/path/to/.venv/lib/python3.8/site-packages/urllib3/response.py", line 894, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(55015292051 bytes read, 78479874614 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/.venv/lib/python3.8/site-packages/requests/models.py", line 820, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/path/to/.venv/lib/python3.8/site-packages/urllib3/response.py", line 1060, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/path/to/.venv/lib/python3.8/site-packages/urllib3/response.py", line 977, in read
    data = self._raw_read(amt)
  File "/path/to/.venv/lib/python3.8/site-packages/urllib3/response.py", line 902, in _raw_read
    self._fp.close()
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/path/to/.venv/lib/python3.8/site-packages/urllib3/response.py", line 772, in _error_catcher
    raise ProtocolError(arg, e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(55015292051 bytes read, 78479874614 more expected)', IncompleteRead(55015292051 bytes read, 78479874614 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/customscript.py", line 104, in _process_msg
    self._work_callback(body)
  File "/path/to/anothercustomscript.py", line 59, in download
    metadata = self.dbx.files_download_to_file(str(dl_path), dbx_path)
  File "/path/to/.venv/lib/python3.8/site-packages/dropbox/base.py", line 1459, in files_download_to_file
    self._save_body_to_file(download_path, r[1])
  File "/path/to/.venv/lib/python3.8/site-packages/dropbox/dropbox_client.py", line 687, in _save_body_to_file
    for c in http_resp.iter_content(chunksize):
  File "/path/to/.venv/lib/python3.8/site-packages/requests/models.py", line 822, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(55015292051 bytes read, 78479874614 more expected)', IncompleteRead(55015292051 bytes read, 78479874614 more expected))

It seems to repeat after almost exactly one hour

DB-Des

@denseishin,

Thank you for that additional information. If the downloads seem to be failing at about an hour it is most likely due to the server timeout. There isn't an option to delay or prevent that.

The /2/files/download endpoint itself is a Content-Download style endpoint and does support Range requests. Using range requests in this case would be a solution we would recommend to handle/resume incomplete downloads. Keep in mind that even requests with ranges set are subject to any server timeouts, so you will want to continue monitoring any failures and use multiple range requests as needed.

denseishin

Please keep in mind that I'm using the official Python SDK for Dropbox. Does that mean that it is not suitable for large files?

DB-Des

Hi @denseishin,

Our official Python SDK and download endpoint are suitable for handling large files. However, depending on factors such as network conditions and server response times, long-running downloads may sometimes encounter timeouts.

The option to use range requests was suggested to help mitigate this—range requests allow downloading files in smaller parts rather than in a single request. This can help reduce the likelihood of timeouts while still enabling successful file retrieval. That said, you should still be able to download large files without issue under normal conditions.