400 status code for a regular GET request

Hi @Здравко, @Greg-DB and sorry for the late response. Today I finally managed to understand the root cause of the issue and found a workaround to it. It is not related to the extra new line I thought earlier, but it is something with the redirect URL received after the first GET request. I was a bit confused by the automatic redirects everywhere so the actual error was tied to the second request. Below are my test scenario, issue description and a workaround.

Test scenario:

1. Send a GET request to this URL: https://www.dropbox.com/scl/fi/fc5rwkff82yvy66ktagv7/test.txt?rlkey=wqzs2qbbfazbgj3ttcaik9pq2&dl=1

2. Receive a 302 Found response with this header: Location: https://ucf3d30d8449d3b4b71630f2179a.dl.dropboxusercontent.com/cd/0/get/CUfGJ4UYzNP3f_SeeyAWDmkzqSHoJk98QNNmD7g4NUoVFnAS6PpVPrCmKLfGBS9_fNsxCkcNG3dT40ygtSc8ceWPxLQYy-NiCrRMDxJ-5u_ZftDOnRmXk-ta_m1czPFq5KRSEvknl3F7pFjk01onJrqL/file?dl=1#

3. Follow the redirect URL and get a 400 Bad Request response. Note: the URL above expires quite early, it'll return the 410 response now, but it is expected and not the actual issue.

Issue and workaround:

The redirect URL contains the # character at the end which causes the 400 response. It is interesting that both curl and Postman remove this character so that is why it is not reproduced there. Removing this character manually before following the redirect fixes the issue and I can get a valid response in my environment as well.

Not sure why this character is a part of the URL and if it is an HTTP rule, Dropbox bug or bug of the underlaying library I'm using, but at least now I have a clean understanding of the issue and a workaround for it. I want to know your thoughts about it and then if it is a library bug, I'll redirect it to its developer 🙂

All comments

Hi @MikkBenelis,

To be sure what's going on you need to dump your request and see what's there. Otherwise it's just a speculation.

@MikkBenelis wrote:
Seems like I'm facing the same or really similar issue. Just wanted to try Dropbox for my pet-project read-only data storage and found that it always responds with a 400 status code for a regular GET request.
...

If you just get 400 status, your case is definitely not the same! In the current thread discussed issue is additional symbols added at the request end that result in second response - erroneous of course. The original request is executed correctly though. That's why the response is 200, not 400. If you don't know what's in your cases start investigate from zero. Just supposing is not best guide. 😉 In this thread the issue is Texas Instruments library bug that messes the request. If you don't use the same library, then your issue, despite looking similar, cannot be the same.

Good luck.

PS: Keep in mind that Dropbox strictly follows HTTP standard. So if in your code is a small "deviation", in spite may tolerated on some places, will be an issue on Dropbox.

@MikkBenelis As Здравко said, it doesn't sound like this is exactly the same as the original issue in the thread you posted in, so I've moved your message to its own thread.

If you still need help with this, please share a sample of the full request and response, including both headers and bodies for both, so we can take a look. Please redact any secret values though.

URL is: https://www.dropbox.com/scl/fi/fc5rwkff82yvy66ktagv7/test.txt?rlkey=wqzs2qbbfazbgj3ttcaik9pq2&dl=1

First of all, there is a chance that this problem is relevant to software I'm using, but when I tried Google Drive and a couple of other direct links for the same task it worked properly. It would be nice if you can help me capture the request details because I don't have low-level access to this library and since it is a TLS request, I can't capture it normally with Wireshark. I can send my minimal executable if it is acceptable, but I think it is better to point me to the way of TLS traffic capturing and decrypting.

I'm sending a regular GET request with the manually added "Host" header with the value of "www.dropbox.com" just in case if it is not calculated automatically (I found that it is required in Postman). Below is the actual URL and all of the response details that I have.

Response code: 400

Headers:
- Content-Length: 1005
- Content-Type: text/html
- Vary: Accept-Encoding
- X-Dropbox-Response-Origin: local
- Date: Mon, 03 Jun 2024 16:39:53 GMT
- Server: envoy
- Connection: close

Body:

<!DOCTYPE html>
<html>
<head><meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Dropbox - 400</title>
<link href="https://cfl.dropboxstatic.com/static/metaserver/static/css/error.css" rel="stylesheet" type="text/css"/>
<link rel="shortcut icon" href="https://cfl.dropboxstatic.com/static/images/favicon.ico"/>
</head>
<body>
<div class="figure">
<img src="https://assets.dropbox.com/www/en-us/illustrations/spot/target-miss.svg" alt="Error: 400"/>
</div>
<div id="errorbox">
<h1>Error (400)</h1>Something went wrong. Don't worry, your files are still safe and the Dropbox team has been notified. Check out our <a href="https://status.dropbox.com">Status Page</a> to see if there is a known incident, our <a href="https://www.dropbox.com/help">Help Center</a> and <a href="https://forums.dropbox.com">forums</a> for help, or head back to <a href="https://www.dropbox.com/home">home</a>.
</div>
</body>
</html>

A response like that should indicate that the request was malformed, so we'd want to see the raw HTTP request you're sending. Unfortunately we can't provide guidance on how to get that kind of logging from your client though, as the client itself isn't made by Dropbox. You may need to refer to the client's documentation or support resources.

I'm trying to dump the raw HTTP request, but not sure that it'll be possible with the high-level API that I have... Is it possible to track the request on your side? The error message says "the Dropbox team has been notified" so I assume it is logged somewhere and maybe I can provide some sort of request ID or something.

Unfortunately I can't retrieve the original raw HTTP request itself, even with the request ID.

Hi @MikkBenelis,

I hope you know what netcat is and how it's used. If you transform https://www.dropbox.com/scl/fi/fc5rwkff82yvy66ktagv7/test.txt?rlkey=wqzs2qbbfazbgj3ttcaik9pq2&dl=1 to something like http://localhost:8080/scl/fi/fc5rwkff82yvy66ktagv7/test.txt?rlkey=wqzs2qbbfazbgj3ttcaik9pq2&dl=1 or similar, you'll be able see exactly what's in your HTTP request. 😉 Do left everything else the same.

Hope this gives direction.

Hey, you are a genius, simple and clean suggestion 🙂

As a backend developer, I don't know why I did not thought about it...

So the failing library produced this request:

GET /scl/fi/fc5rwkff82yvy66ktagv7/test.txt?rlkey=wqzs2qbbfazbgj3ttcaik9pq2&dl=1 HTTP/1.1
Host: www.dropbox.com
Accept-Encoding: gzip, deflate
User-Agent: GodotEngine/4.2.2.stable.official (Windows)
Accept: */*

And this what was produced by Postman (added headers to match previous request):

GET /scl/fi/fc5rwkff82yvy66ktagv7/test.txt?rlkey=wqzs2qbbfazbgj3ttcaik9pq2&dl=1 HTTP/1.1
Host: www.dropbox.com
Accept-Encoding: gzip, deflate
User-Agent: GodotEngine/4.2.2.stable.official (Windows)
Accept: */*

The only difference is this extra new line at the end. I can't control it since it is an external library. And it means that the issue is exactly the same in original thread where I posted my message. Potentially, we are using same libraries.

Thanks for the sample! It appears to manifest slightly differently since you're unable to get a success response from a single request, whereas in the original issue an extra new line caused an issue only on the second request on the same connection, not the first request. Regardless, it does sound like the issue is that the client is sending the unexpected new line, resulting in a malformed request. We're looking into it and I'll follow up here with any updates. I can't promise if/when the team will be able to update the server to accept these malformed requests though.

@MikkBenelis wrote:
...
The only difference is this extra new line at the end. I can't control it since it is an external library. ...

Hm..🤔 Honestly I'm not familiar with that environment you're using, but as far as I can see it's .NET based. In such a case (if I'm not in confusion), why at all additional engine would be needed at all. .Net is able to perform HTTP request successful! Use the available underlying engine instead of reimplemented buggy one that you don't know where is coming from. 😉 I believe this will be possible.

Good luck.

PS: For long term solution try replace your requests responsible code with non buggy or ask the supplier to debug its code (e.g. signal for the bug).

By the way, usually most libraries return one result matching to one corresponding request. If by mistake unexpected request (correct or incorrect) comes up, result is either first or last one. As seems in your case the result got back is the last one (carrying the error) - opposite to the OP. 🙋

Of course, this is rather speculation.

@ЗдравкоThe environment I'm using supports two languages: C# (.NET) and GDScript (or both). I'm using GDScript-only version for this application which as I understand relies on its own HTTP client implementation probably with libcurl and mbedtls libraries. I already reported it, but had to find some details since everything except Dropbox worked fine.

I lost a link so not sure what was the first request of the OP, but if I remember clear it was PUT, POST or something with the body while the second one was regular GET. While testing with netcat I found that this extra line is a line where request body starts and since it is empty, this line is empty as well (looks like an extra line, but it is actually an empty body). Pretty sure that the HTTP specification allows this new line since all the rest test subjects are working fine. I found only that the GET body should be ignored since it has no meaning, but it doesn't say that I can't send it. Hope it'll be allowed by Dropbox (at least an empty GET body), but seems like I need to search for alternatives for now...

@MikkBenelis wrote:
... I understand relies on its own HTTP client implementation probably with libcurl and mbedtls libraries. ...

@MikkBenelis, mbedtls has nothing to do with HTTP formation; it, if used, is responsible for encryption only. libcurl, if used, works perfectly fine! I personally tested it and it works. Even more the curl command built on top of libcurl is referred tool for testing Dropbox API - again working flawlessly. 🤔🤷 Are you sure libcurl is in use? 🧐 I planned to advise you use exactly this library if possible - just wasn't sure if the environment has such an option. Don't suppose, instead make it sure! 🙋

@MikkBenelis wrote:
...
I lost a link so not sure what was the first request of the OP, but if I remember clear it was PUT, POST or something with the body while the second one was regular GET. ...

The thread is here. Dropbox API primary uses POST requests. Your question is not related to API actually, but to regular download code - something you should not think at all usually. There must be some error in your environment or you have miss-configured something.

@MikkBenelis wrote:
... I found that this extra line is a line where request body starts and since it is empty, this line is empty as well (looks like an extra line, but it is actually an empty body). ...

Ha..😁 @MikkBenelis, As a backend developer, you should not use/state something like that. Is a line (empty or not) some empty body (recall - empty body means empty content, including string)? Is a new line string an empty string? 🙂 Take a bit more care.

In addition you may take a look on MDN definition for HTTP request or the same on RFC 2616. Can you see somewhere there that either on place of empty body has to be a new line or non empty body has to end with extra new line that is not part of the content length? 😉

Thanks for your comments, you are absolutely right, I'll try to be more carefull with it 🙂

For testing purposes I sent a GET request with the body and Dropbox accepted it without any issues from both curl and Postman so this new line is not a problem anymore. Moreover, now netcat shows me identical requests even without the body neither for Postman or Godot requests (kept the User-Agent header so I can distinguish these requests).

I tried to save this output to a file to check if there are any invisible characters, but no, they are both CR LF ended identical texts without invisibles. Now I'm completely lost... Can you suggest me something to test? The only Dropbox is failing now and I don't understand why.

Hm.. 🤔 @MikkBenelis, as far as I can see there is no any extra new line in both requests on your screenshot. You said that have one found. Can you clarify what exactly you found out and where it is? It may be useful to post a link to binary dump of your netcat output - there might be something else if not extra new line. No everything is clearly visible as simplest text or terminal screenshot.

Let's see...

Test scenario:

1. Send a GET request to this URL: https://www.dropbox.com/scl/fi/fc5rwkff82yvy66ktagv7/test.txt?rlkey=wqzs2qbbfazbgj3ttcaik9pq2&dl=1

3. Follow the redirect URL and get a 400 Bad Request response. Note: the URL above expires quite early, it'll return the 410 response now, but it is expected and not the actual issue.

Issue and workaround:

@MikkBenelis wrote:
...
Not sure why this character is a part of the URL and if it is an HTTP rule, ...

@MikkBenelis, I believe you're receiving email notifying you about my current post. Take a closer look on the link pointing my post. Can you see there something similar to that you are not sure is it HTTP rule? 🧐🙋 Something else you may try - navigate to the same page starting from www.dropboxforum.com and directly using the link pointing my post. Note that in the first case the page top comes up, but in the second one - directly my post. 🤔🤷 In addition take a look on the HTTP requests themself. Is there any difference between them?... In such a case, why does different page view appear? 😯 In fact it's the same page, received with the same request! 🤫

Ok, the last URL construct (named anchor), that starts with hash(#) and may have others following symbols, is absolutely valid construct. 🙂 That construct carries kind of parameter(s) targeting HTTP client, not server - some additional information that may affect client work, including (but not limited to) how the resulting view appears. Such information doesn't appear in any HTTP request (or should not at least). Server receives the classical params only - between question mark, if any, and hash (or till the URL end). The hash symbol (and all after that) should misses in the request. 😉

@MikkBenelis wrote:
..., Dropbox bug or bug of the underlaying library I'm using, but at least now I have a clean understanding of the issue ...

Hm...🤔 ...and is it a Dropbox bug or a bug in your underlying library? What did you conclude based on where that "cut" should happen? 😉

Good luck.

@ЗдравкоI knew that it is usually used to scroll the page to a specific section or highlight some region of interest if a user agent supports it, but did not know that it should never be sent back to server... My conclusion that there is a bug in both Dropbox and the URI specification itself 😁

Imagine you had a link www.abc.com/qwe#asd, then it was moved to www.abc.com/qwert#asdfg and you want to apply a 301 redirect to it. You can't do it with Dropbox, it'll respond with a 400 error. This part is called Fragment, here is the related RFC section https://datatracker.ietf.org/doc/html/rfc3986#section-3.5

Why is it a Dropbox bug? IMHO, it should not send a URI with the number sign character without anything after it, especially for such redirects. Also, I think that this part should be excluded on a server if it has no special meaning there, but it should not respond with the 400 error. Most of the servers (just tried a couple of Google links) can use or exclude it and the request will be processed normally.