Child Item

GET request to /files/download fails with "The network connection was lost"

Hi,

Our app uses Dropbox's HTTP API to upload and download files. Since a few days ago, some of our app's requests are failing with "The network connection was lost". This issue was first reported on May 2nd. Our app uses React Native.

The failing request downloads files from Dropbox using a GET request to https://content.dropboxapi.com/2/files/download (code). A simplified version of the request looks like this:

fetch("https://content.dropboxapi.com/2/files/download", {"headers":{"Dropbox-API-Arg":"{\"path\":\"/info.json\"}","Authorization":"Bearer TOKEN_HERE","Content-Type":"application/octet-stream"},"method":"GET"})

If I use POST instead of GET, the request consistently returns the content of the file. However, as commented here, using a POST request with an empty body seems to fail in React Native/iOS. Dropbox doesn't seem to allow a non-empty body for this request.

Sometimes, the above fetch does work. However, I can reproduce the failure consistently by sending the above request twice, but only in some environments:
• Running code from the Safari development tools, inspecting a WebView in our application.
• Running code from the React Native Hermes development tools.

I can't reproduce this when using the same code in a NodeJS REPL or the Firefox JavaScript console.

Downstream issue: https://github.com/laurent22/joplin/issues/10396

Find more posts tagged with

API

Abuse

Comments

personalizedrefrigerator

Hi,

The Joplin note taking app uses Dropbox's HTTP API to upload and download files. Since a few days ago, some of our its requests are failing with "The network connection was lost".

The failing request downloads files from Dropbox using a GET request to content.dropboxapi.com/2/files/download:

fetch(
    "https://content.dropboxapi.com/2/files/download",
    {
        "headers": {
            "Dropbox-API-Arg": "{\"path\":\"/info.json\"}",
            "Authorization": "Bearer " + API_TOKEN,
            "Content-Type": "application/octet-stream"
        },
        "method":"GET"
    }
)

Sometimes, the above fetch does work. However, I can reproduce the failure consistently by sending the above request twice, but only in some environments:
- Running code from the Safari development tools, inspecting a WebView embedded in Joplin.
- Running code from the Chrome development tools, connected to React Native Hermes.

I can't reproduce this when using the same code in a NodeJS REPL or the Firefox JavaScript console.

The examples in the Dropbox documentation send POST rather than GET requests to files/download. Switching to a POST request does seem to fix the issue on Android. However, it still fails on iOS.

Another post on this forum suggests that fetch on iOS doesn't support sending POST requests with an empty body. Dropbox doesn't seem to allow a non-empty body for this request.

Is this forum the correct place to be asking for help with this issue? Is there a way to continue using a GET request to /files/download to download files?

Notes:
- The original version of this post was marked as spam, perhaps due to links to Joplin's source code and issue tracker. These links have been removed and can be provided on request.

- Joplin uses React Native 0.71. The issue still happens after upgrading to React Native 0.73.8.

kzencorp

I have the opposite issue!
I also have a ReactNative app, and In the last few days (since April 30), I have seen this error occur only for my iOS users and only on the https://content.dropboxapi.com/2/files/download… endpoint.

Error Domain=NSURLErrorDomain Code=-1005 "The network connection was lost."

But I use POST for it. Once I change it to GET it seems to fix the issue on iOS.

WDYT?

Greg-DB

While we can't offer insight on the client platforms/libraries in particular, this is the right place for help with the Dropbox API itself.

For reference, the Dropbox does use POST for all API calls by default, but "content-download" style endpoints, such as /2/files/download, do additionally support GET. And it's correct that the endpoint does not support POST requests with a non-empty body.

I just tried it out, and confirmed that the API is still allowing both GET and POST for this endpoint in my testing:

curl -v -X POST https://content.dropboxapi.com/2/files/download \
    --header "Authorization: Bearer <ACCESS_TOKEN>" \
    --header "Dropbox-API-Arg: {\"path\":\"/test.txt\"}" \
    --header "Content-Type: application/octet-stream"

> POST /2/files/download HTTP/2
> Host: content.dropboxapi.com
> User-Agent: curl/8.4.0
> Accept: */*
> Authorization: Bearer <ACCESS_TOKEN>
> Dropbox-API-Arg: {"path":"/test.txt"}
> Content-Type: application/octet-stream
>
< HTTP/2 200
< content-type: application/octet-stream
< accept-ranges: bytes
< cache-control: no-cache
< dropbox-api-result: {"name": "test.txt", "path_lower": "/test.txt", "path_display": "/test.txt", "id": "id:25N5ksooX-sAAAAAAAQYrQ", "client_modified": "2024-03-11T15:24:40Z", "server_modified": "2024-03-11T15:24:40Z", "rev": "6136423edf446021eccc7", "size": 27, "is_downloadable": true, "content_hash": "cceac37da5288a6b6716737310c14305c9b16a6e0ff0abce4a306c3de3a59f75"}
< etag: W/"6136423edf446021eccc7"
< original-content-length: 27
< x-server-response-time: 375
< date: Mon, 06 May 2024 15:09:05 GMT
< server: envoy
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-robots-tag: noindex, nofollow, noimageindex
< content-length: 27
< x-dropbox-response-origin: far_remote
< x-dropbox-request-id: d0eb597d801d4203819d86c9b65b42a1
<
* Connection #0 to host content.dropboxapi.com left intact
This is the test file data.

curl -v -X GET https://content.dropboxapi.com/2/files/download \
    --header "Authorization: Bearer <ACCESS_TOKEN>" \
    --header "Dropbox-API-Arg: {\"path\":\"/test.txt\"}" \
    --header "Content-Type: application/octet-stream"

> GET /2/files/download HTTP/2
> Host: content.dropboxapi.com
> User-Agent: curl/8.4.0
> Accept: */*
> Authorization: Bearer <ACCESS_TOKEN>
> Dropbox-API-Arg: {"path":"/test.txt"}
> Content-Type: application/octet-stream
>
< HTTP/2 200
< content-type: application/octet-stream
< accept-ranges: bytes
< cache-control: no-cache
< content-disposition: attachment; filename=unspecified
< content-security-policy: sandbox
< dropbox-api-result: {"name": "test.txt", "path_lower": "/test.txt", "path_display": "/test.txt", "id": "id:25N5ksooX-sAAAAAAAQYrQ", "client_modified": "2024-03-11T15:24:40Z", "server_modified": "2024-03-11T15:24:40Z", "rev": "6136423edf446021eccc7", "size": 27, "is_downloadable": true, "content_hash": "cceac37da5288a6b6716737310c14305c9b16a6e0ff0abce4a306c3de3a59f75"}
< etag: W/"6136423edf446021eccc7"
< original-content-length: 27
< vary: Dropbox-API-Arg, Authorization
< x-content-security-policy: sandbox
< x-content-type-options: nosniff
< x-server-response-time: 340
< x-webkit-csp: sandbox
< date: Mon, 06 May 2024 15:09:29 GMT
< server: envoy
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-robots-tag: noindex, nofollow, noimageindex
< content-length: 27
< x-dropbox-response-origin: far_remote
< x-dropbox-request-id: 1eaaff94c0f84f2f91cf96059913ee87
<
* Connection #0 to host content.dropboxapi.com left intact
This is the test file data.

I included the "Content-Type" request header to match your example, but it's not actually required for this. Also, I tried the GET request several times in a row without issue.

So, it does sound like this issue is environment-specific. We'd be happy to look into it further on our side to see if there's something we can do about it from the Dropbox server side, but it's unclear what the issue is exactly so far. Is there a way you could enable some more verbose output (similar to -v for curl above) to gather more information about how exactly the request is failing in the affected environments?

personalizedrefrigerator

Thank you for checking!

> I just tried it out, and confirmed that the API is still allowing both GET and POST for this endpoint in my testing

Locally, I only experience the issue when sending a GET request for the same file two or more times. The first request is successful, but subsequent requests fail, until another file is requested.

> We'd be happy to look into it further on our side to see if there's something we can do about it from the Dropbox server side, but it's unclear what the issue is exactly so far. Is there a way you could enable some more verbose output (similar to -v for curl above) to gather more information about how exactly the request is failing in the affected environments?

I'm not sure how to do this for React Native. As commented above, I was able to reproduce part of the issue in Safari/iOS. I'm attaching screenshots of the "network" tab for a failing and passing GET request in this environment.

Edit: It may be necessary to click on the image above to view it with greater resolution.
Note that I can't reproduce the POST issue in Safari.

Greg-DB

Thanks! It looks like there's no response received. In cases like that, we'd need to refer to the network client error to see why that failed (e.g., DNS issue, timeout, etc.). It doesn't look like there's much there though, unfortunately. I do see the "TypeError: Load failed". Is there any additional information there if you expand those two dropdown arrows next to that? And is there anything in the "Preview" tab?

personalizedrefrigerator

The "Preview" tab shows an error message:

The dropdown by "TypeError: Load failed" similarly doesn't show additional information.

The right-click menu for the failed request has options that might be helpful:

"Copy HTTP request":

GET /2/files/download
Authorization: Bearer TOKEN_HERE
Content-Type: application/octet-stream
Origin: https://js-draw.web.app
Referer: https://js-draw.web.app/
Accept: */*
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15
Dropbox-API-Arg: {"path":"/info.json"}

"Copy HTTP response":

NaN null

"Copy as cURL":

curl 'https://content.dropboxapi.com/2/files/download' \
-X 'GET' \
-H 'Authorization: Bearer [[token]]' \
-H 'Content-Type: application/octet-stream' \
-H 'Origin: https://js-draw.web.app' \
-H 'Referer: https://js-draw.web.app/' \
-H 'Accept: */*' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Site: cross-site' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15' \
-H 'Dropbox-API-Arg: {"path":"/info.json"}'

Interestingly, the CURL request is successful when run multiple times in the MacOS terminal app.

I'm now trying to see if I can log network requests made by the iOS simulator (relevant Apple documentation).

Greg-DB

Thanks for sharing that. Yes, unfortunately it doesn't look like there's anything else useful there.

Just to check, I also tried the curl command a few times but that also worked fine for me.

Logging the failing network requests themselves sounds like a good next step, so let us know if you're able to get anything from that.

personalizedrefrigerator

> Logging the failing network requests themselves sounds like a good next step, so let us know if you're able to get anything from that.

Setting CFNETWORK_DIAGNOSTICS didn't give as much information as I had hoped. This is an excerpt relevant to the failed request:

default	15:15:31.519737-0700	Joplin	CFNetwork Diagnostics [3:272077] 15:15:31.519 {
HTTPCookieStorage::copyCookiesForURL: <CFHTTPCookieStorage 0xredacted [0xredacted]>
                         Request URL: https://content.dropboxapi.com/<redacted>
                    MainDocument URL: null
} [3:272077]
default	15:15:31.519624-0700	Joplin	CFNetwork Diagnostics [3:272076] 15:15:31.519 {
AddCookies Continue: request GET https://content.dropboxapi.com/<redacted>
       HTTPProtocol: Task: 0xredacted
} [3:272076]
default	15:15:31.519507-0700	Joplin	CFNetwork Diagnostics [3:272075] 15:15:31.519 {
Vary Validation: (null)
     Evaluation: PASS
        Request: https://content.dropboxapi.com/<redacted>
     VaryHeader: Dropbox-API-Arg, Authorization
} [3:272075]
default	15:15:31.519387-0700	Joplin	CFNetwork Diagnostics [3:272074] 15:15:31.519 {
LoaderWhatToDo: (null)
       Request: https://content.dropboxapi.com/<redacted>
   CachePolicy: 0
      WhatToDo: originload
   CreateToNow: 0.00093s
} [3:272074]
default	15:15:31.519254-0700	Joplin	CFNetwork Diagnostics [3:272073] 15:15:31.519 {
Vary Validation: (null)
     Evaluation: PASS
        Request: https://content.dropboxapi.com/<redacted>
     VaryHeader: Dropbox-API-Arg, Authorization
} [3:272073]
default	15:15:14.050982-0700	Joplin	Task <redacted-task-id>.<10> resuming, timeouts(0.0, 604800.0) QOS(0x19) Voucher (null)
default	15:15:14.050704-0700	Joplin	CFNetwork Diagnostics [3:271985] 15:15:14.050 {
HTTPCookieStorage::copyCookiesForURL: <CFHTTPCookieStorage 0xredacted [0xredacted]>
                         Request URL: https://content.dropboxapi.com/<redacted>
                    MainDocument URL: null
} [3:271985]
error	15:15:10.042740-0700	Joplin	Task <C49036A5-617D-4742-B3E3-58641BD95148>.<9> finished with error [-1005] Error Domain=NSURLErrorDomain Code=-1005 "The network connection was lost." UserInfo={_kCFStreamErrorCodeKey=57, NSUnderlyingError=0x600000c78060 {Error Domain=kCFErrorDomainCFNetwork Code=-1005 "(null)" UserInfo={NSErrorPeerAddressKey=<CFData 0xredacted [0xredacted]>{length = 16, capacity = 16, bytes = 0xredacted}, _kCFStreamErrorCodeKey=57, _kCFStreamErrorDomainKey=1}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <redacted-task-id>.<9>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <redacted-task-id>.<9>"
), NSLocalizedDescription=The network connection was lost., NSErrorFailingURLStringKey=https://content.dropboxapi.com/2/files/download, NSErrorFailingURLKey=https://content.dropboxapi.com/2/files/download, _kCFStreamErrorDomainKey=1}
default	15:15:10.042500-0700	Joplin	Task <redacted-task-id>.<9> summary for task failure {transaction_duration_ms=442, response_status=200, connection=8, reused=1, reused_after_ms=9620, request_start_ms=1, request_duration_ms=0, response_start_ms=440, response_duration_ms=0, request_bytes=179, response_bytes=96, cache_hit=true}
default	15:15:10.042383-0700	Joplin	CFNetwork Diagnostics [3:271970] 15:15:10.042 {
           Did Fail: (null)
             Loader: request GET https://content.dropboxapi.com/<redacted>
              Error: Error Domain=kCFErrorDomainCFNetwork Code=-1005 UserInfo={NSErrorPeerAddressKey=<private>, _kCFStreamErrorCodeKey=57, _kCFStreamErrorDomainKey=1}
init to origin load: 0.000844955s
         total time: 0.442753s
        total bytes: 0
} [3:271970]

Many of the `<redacted>` sections above were removed by XCode or iOS.

Additionally, I notice that the above log includes "cache_hit=true".

Greg-DB

Yes, unfortunately it's still not very clear with this output.

I do see that it has "response_status=200", which doesn't seem to match the other output we've seen (where there doesn't seem to be a response), but that may be because as you noted it says "cache_hit=true", so perhaps it's actually referring to a cached response from a previous call. That may imply an issue with how the platform's caching works. I'm not aware of a mistake with how the Dropbox API servers currently handle any caching headers, but if there seems to be an issue with that please let me know what exactly seems to cause an issue for your platform and I can raise that with the team.

Also, I notice there are some other error codes "_kCFStreamErrorCodeKey=57, _kCFStreamErrorDomainKey=1". I'm not familiar with those, and we can't provide support for the platform/network client itself as it's not made by Dropbox, but do you know what those particular codes may indicate?

personalizedrefrigerator

> Also, I notice there are some other error codes "_kCFStreamErrorCodeKey=57, _kCFStreamErrorDomainKey=1". I'm not familiar with those, and we can't provide support for the platform/network client itself as it's not made by Dropbox, but do you know what those particular codes may indicate?

I don't know what the `_kCF...` error codes mean and didn't find references to these particular error codes with a web search or in the Apple documentation.

> I do see that it has "response_status=200", which doesn't seem to match the other output we've seen (where there doesn't seem to be a response), but that may be because as you noted it says "cache_hit=true", so perhaps it's actually referring to a cached response from a previous call.

It's strange that this is a cache hit — the original response contains `Cache-Control: no-cache`. It would be interesting to compare the current headers with those from a few weeks ago to see if this, or other cache-related headers changed.

We merged a workaround that, on request failure, sends a request for a different file before retrying the original request. Even though failure doesn't occur for most `files/download` GET requests, the extra API requests required by this workaround aren't ideal.

Forcing iOS to avoid the cache may be a better solution. As such, I've also tried:
• Passing `Cache-Control: no-cache` as an additional request header. (Doesn't work -- same "network request failed" error).
• Adding a new header and adding/referencing it in the `Vary` header (see this StackOverflow discussion). (Doesn't work -- same "network request failed" error).

• Retrying with `?` appended to the Dropbox `files/download` URL on failure. (Doesn't work -- same "network request failed" error).

• Retrying with `?retry` appended to the Dropbox `files/download` URL on failure. This results in `Error (400): Error in URL parameters: Bad query field: 'retry'`.

Greg-DB

I found an old sample of a /2/files/download call from November in my records. The request didn't have the same browser/CORS headers, so it's not an exact comparison, but for reference the response did have "Cache-Control: no-cache".

personalizedrefrigerator

Thank you for checking!

The 200 OK response attached in a previous message has "Vary: Dropbox-API-Arg, Authorization". Based on MDN, the Vary header is related to caching and content-negotiation. Does the November response have a similar Vary header?

---
Edit:
The XCode network logs contain:

Vary Validation: (null)
     Evaluation: PASS
        Request: https://content.dropboxapi.com/<redacted>
     VaryHeader: Dropbox-API-Arg, Authorization
} [3:272075]

According to this article, the Vary header is often used to validate a cache record (to determine whether that record can be used or if the request should be treated as a cache miss).

Based on this, one possibility is that the original `Cache-Control: no-cache` causes the response to not be cached (or incorrectly cached). The next request checks the last request's "Dropbox-API-Arg" and "Authorization" because of the "Vary" header. In this case, they're the same, so the system treats this as a cache hit. However, the previous response isn't correctly cached, causing an error. Note that this is just a guess. It should be possible to test this with a local webserver, however.

Edit 2: "no-cache" means "revalidate before using the cache" while "no-store" means "don't store responses in the cache". Based on this, revalidating is expected.

Based on this, I've tested with setting the fetch `cache` to `no-store` to disable caching,

await fetch(
  "https://content.dropboxapi.com/2/files/download",
  {
    "headers": {
      "Dropbox-API-Arg": "{\"path\":\"/info.json\"}",
      "Authorization": "Bearer TOKEN_HERE",
      "Content-Type": "application/octet-stream"
    },
    "method": "GET",
    "cache": "no-store"
  }
)

I'm currently in the process of testing this.

personalizedrefrigerator

Unfortunately, "cache": "no-store" doesn't seem to help.

Based on a suggestion received elsewhere, I've also tried adding an If-None-Match set to a random value:

await fetch(
  "https://content.dropboxapi.com/2/files/download",
  {
    "headers": {
      "Dropbox-API-Arg": "{\"path\":\"/info.json\"}",
      "Authorization": "Bearer TOKEN_HERE",
      "Content-Type": "application/octet-stream",
      "If-None-Match": "JoplinIgnore-" + (Math.floor(Math.random() * 100000)),
    },
    "method": "GET"
  }
)

Our source code already had a similar workaround for a different sync provider with the comment,

> On iOS, the network lib appends a If-None-Match header to PROPFIND calls, which is kind of correct because the call is idempotent and thus could be cached. According to RFC-7232 though only GET and HEAD should have this header for caching purposes. It makes no mention of PROPFIND.
> [... additional commenting ...]

This is a GET request and not a PROPFIND request. However, I suspect that passing a random `If-None-Match` overrides the iOS network library's default and forces Dropbox to return a 200 OK response (rather than a 304 Not Modified, which I think may have been Dropbox's response when the issue occurred).

I'm in the process of testing this. So far, however, it seems to work. This is the pull request with the relevant change.

Thank you for helping me debug this!

Greg-DB

Thanks for following up and sharing this! I'm glad to hear you were able to work that out.

anmipo

I just faced the same error in a native iOS app, can confirm this is cache-related.

The app uses Apple's own URLRequest, no wrappers. When sending POST requests for the same file to "2/files/download" endpoint:

First request works fine and returns the file
The second request depends on urlRequest.cachePolicy:
- Fails with -1005 if cache policy is not set or allows caching
- Works out fine if set to .reloadIgnoringLocalCacheData

@personalizedrefrigerator , thank you for the hint in the right direction!