Child Item

The underlying connection was closed error.

Hi,

I'm using DropBox API v2 to get folders and files from user account. For that I'm using "files/list_folder", "files/list_folder/continue" endpoints and also I'm using recursive: true for root folder and cursor: cursorvalue while iterate. The problem here is the user account contains almost 400k+ files and while downloading them I'm getting "The underlying connection was closed: A connection that was expected to be kept alive was closed by the server." error after some time. Is this error related to any API limitations? Is there any chance to get this error when multiple colletions running at same time?

Thanks,

OTKY

Find more posts tagged with

Developers

Comments

Greg-DB

I'll be happy to help with any issues you're having with the Dropbox API, but I'll need some more information. Please reply with:

the name and version number of the SDK/library you are using, if any
the steps to reproduce the issue, including relevant code snippet(s), but don't include any access or refresh token(s)
the full text of any error or unexpected output

otky

Thanks for the quick response . I'm not using any sdk just I'm using c# HttpWebRequest with API endpoints.

Find the below endpoints and breif description of the code .

endpoint : https://api.dropbox.com/2/files/list_folder

and adding {"path":"","recursive":true} to request stream

headers: Dropbox-API-Select-User:{{userId}}

if response contains "has_more" then using the below endpoint

endpoint : https://api.dropbox.com/2/files/list_folder/continue
{cursor:cursorid} to request stream
headers: Dropbox-API-Select-User:{{userId}}

to download files I'm using
endpoint: https://api-content.dropbox.com/2/files/download
{"path":"rev:{revisionId}"}

Regarding the error messages after running for sometime(more than 1 day) I got "Unable to connect to the remote server , A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond" sometimes and "The underlying connection was closed: A connection that was expected to be kept alive was closed by the server." sometimes.

Thanks,

OTKY

Greg-DB

Thanks for following up. It's not clear from this why this would be happening, but a few possibilities/other questions:

Is there anything on your network connection, such as a firewall, proxy, VPN, anti-virus, or other security software that may be interfering with your connection to api-content.dropbox.com?
When you say it's "running for sometime(more than 1 day)", do you mean the overall process, or an individual request? How long is an individual /2/files/download request taking before failing?
About what percent of your /2/files/download requests are failing?
When you say you have "when multiple colletions running at same time", do you mean you have more than one request to /2/files/download running simultaneously? If so, how many?

otky

Hi,

Please find my answers in Italic.

Is there anything on your network connection, such as a firewall, proxy, VPN, anti-virus, or other security software that may be interfering with your connection to api-content.dropbox.com?

I'm not sure about these. But do these things cause the issue for accounts which have larger number of files?

When you say it's "running for sometime(more than 1 day)", do you mean the overall process, or an individual request? How long is an individual /2/files/download request taking before failing?

Yes the overall process for downloading the files one by one from DropBox account.

About what percent of your /2/files/download requests are failing?

Sometimes its failing at "/2/files/download" and sometimes its failing at "files/list_folder/continue" . But in both the cases its not failing immediately.

When you say you have "when multiple colletions running at same time", do you mean you have more than one request to /2/files/download running simultaneously? If so, how many?

Yes , 2-3 processes running simultaneously .Is there any chances of throttling here ? If there is a throttling issue , will all the processes running at that time will throw errors and stop execution from then?

Thanks,

OTKY.

Greg-DB

Things potentially interfering with your network connection wouldn't depend on how many files are in the connected account. I recommend checking for any such potential issues regardless of the file count.

For reference, the Dropbox API does have a rate limiting system, but if it rejects any particular call, it would respond with an explicit error response, with a 429 status code. It would not just close the connection as you're seeing here, so that does not seem to be the issue. In any case, 2-3 simultaneous requests is not excessive and so we would not expect such activity to be rate limited.

Also, since this is actually happening on both /2/files/download and /2/files/list_folder/continue (which use different hosts), it sounds like there's likely client-side issue here. We can't offer help with HttpWebRequest itself though, as that's not made by Dropbox.