upload API only works for very small files

https://www.raspberrypi.org/forums/viewtopic.php?t=223026

@ags65 We looked into this, and it seems like you may be running in to this issue:

As noted there, you can try:

To stop sending DFs, you can change net.ipv4.ip_no_pmtu_disc value to 1 in sysctl.
I would also try reducing the MTU on the interface in order to advertise a smaller MSS to the TCP peer.

All comments

Hi @ags65,

That's a old curl feature currently almost deprecated. The idea is when relatively big file have to be passed, credentials to be check in advance, so prevents get notification for invalid credentials after passing everything. Generally something useful. But seems your curl version doesn't behave as expected. Actual files data have to be passed after the confirmation, but seems passed before that. Probably your network connection is very slow and that's why response delay is more than 1 second (the default threshold). You can try set this threshold to something appropriate in your case or try disable this check by explicitly set "Expect" header with empty value, for example.

Hope this gives the right direction.

Hi,

Thank you very much for your help, but the problem persists:

my network connection is fast, the raspberry pi is connected by ethernet cable to the router and my internet connection works with optic fibre cable.

cURL version is the last in raspbian: 7.64.0

the cURL command with option -H 'Expect:' does not send the "Expect: 100-continue" to dropbox but directly send the file. But the error is the same:

> POST /2/files/upload HTTP/1.1
> Host: content.dropboxapi.com
> User-Agent: curl/7.64.0
> Accept: */*
> Authorization: Bearer xxxxxxx
> Dropbox-API-Arg: {"path": "/README.md","mode": "add","autorename": true,"mute": false,"strict_conflict": false}
> Content-Type: application/octet-stream
> Content-Length: 8900
>
* upload completely sent off: 8900 out of 8900 bytes
* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Connection #0 to host content.dropboxapi.com left intact
curl: (52) Empty reply from server

If there is a problem in my system, why it does not affect with small files to upload ?

From dropbox, is there a way to see the log of connection attempts?

Thank you for your help

Wow... That's strange. :thinking:

Make your own investigation, what's going on. Try crosscompile working desktop version and see if the behaviour changes. If I'm in your shoes, this would be the first check. Try if same file transfer gonna completes on desktop version.

We're not currently experiencing any known service disruption that would cause empty replies from the Dropbox API. Might there be something on your network connection preventing your app from connecting to the Dropbox servers? For example, might there be some firewall, proxy, or caching layer that is either blocking or failing on larger requests?

Hi,

I sent the very same command from Ubuntu to upload a file of similar size and it works perfectly.

The problem have to be in the raspberry pi system or cURL command version.

I will report any progress I make on this subject.

Thank you very much.

Hi,

Following with the subject, the problem arises when the file size is greater than the mtu (1500 bytes) and cURL send the more than one data packet.

I'm not un expert but what I see in Wireshark is that dropbox answer back the first packet with the ACK but inform parameters SLE and SRE with packets that I dont know what they are.

After that the raspberry try several times the retransmission of the first packet and finally dropbox send the reset that ends the connection

I dont know if someone used to deal with network problems can explain what's happening...

[IMAGE LINK IS BROKEN AND CANNOT BE DISPLAYED]

That what I can see is you had send 4 bytes less than Dropbox expects and the confirmations given are for packets that the server doesn't signal for... Bug in the software!

I've compiled in the raspberry pi the same version of cURL that works in my ubuntu (7.65.3) and the problem remains:

* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Dropbox, Inc; CN=content.dropboxapi.com
* start date: Jan 29 00:00:00 2020 GMT
* expire date: Mar 2 12:00:00 2022 GMT
* subjectAltName: host "content.dropboxapi.com" matched cert's "content.dropboxapi.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
> POST /2/files/upload HTTP/1.1
> Host: content.dropboxapi.com
> User-Agent: curl/7.65.3
> Accept: */*
> Authorization: Bearer xxxxxxxxxxx
> Dropbox-API-Arg: {"path": "/Cromosoma.cc","mode": "add","autorename": true,"mute": false,"strict_conflict": false}
> Content-Type: application/octet-stream
> Content-Length: 2863
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host content.dropboxapi.com left intact
curl: (52) Empty reply from server

Hmm... You can try another diagnose way. Check if the transaction completes successfully using something like:

curl -v -X POST https://httpbin.org/post -H "Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -H "Dropbox-API-Arg: {\"path\": \"/Cromosoma.cc\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}" -H "Content-Type: application/octet-stream" -H "Expect:" --data-binary @/home/pi/Dropbox-Uploader/Cromosoma.cc

The response would show what actually got send. Evaluate it.

Let's hope result will drive you to the right conclusion.

interesting...

the cURL command works in httpbin.org and the data seems to be OK:

Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 34.230.193.231:443...
* TCP_NODELAY set
* Connected to httpbin.org (34.230.193.231) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=httpbin.org
* start date: Jan 18 00:00:00 2020 GMT
* expire date: Feb 18 12:00:00 2021 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
> POST /post HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.65.3
> Accept: */*
> Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Dropbox-API-Arg: {"path": "/Cromosoma.cc","mode": "add","autorename": true,"mute": false,"strict_conflict": false}
> Content-Type: application/octet-stream
> Content-Length: 2863
>
* upload completely sent off: 2863 out of 2863 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 06 Mar 2020 16:28:55 GMT
< Content-Type: application/json
< Content-Length: 4463
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
<
{
"args": {},
"data": "data&colon;application/octet-stream;base64,LyoqIEBma............g==",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Authorization": "Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"Content-Length": "2863",
"Content-Type": "application/octet-stream",
"Dropbox-Api-Arg": "{\"path\": \"/Cromosoma.cc\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}",
"Host": "httpbin.org",
"User-Agent": "curl/7.65.3",
"X-Amzn-Trace-Id": "Root=1-5e627a47-49885dec98b9e392fb837b9a"
},
"json": null,
"origin": "83.47.100.114",
"url": "https://httpbin.org/post"
}
* Connection #0 to host httpbin.org left intact

Wow... If content of 'data' field match exactly to the posted file... hands up... I haven't more ideas. Isn't there something missing at the end? The content there is JSON-ASCII encoded! Decode it before compare if need (or opposite). For example, encode the file you passed using:

python3 -c "import json;import sys;f=open(sys.argv[1],'r');print(json.dumps(f.read()));f.close()" /home/pi/Dropbox-Uploader/Cromosoma.cc

And compare the result to 'data' field content above.

Yes, the content of "data" field match to the file:

https://httpbin.org:

 "data": "/** @file Cromosoma.cc\n    @brie...... << crom[i];\n    }\n    cout << endl;\n}\n",

print(json.dumps):

"/** @file Cromosoma.cc\n    @brie...... << crom[i];\n    }\n    cout << endl;\n}\n"

no missing characters at the end.

I'm up to here. Probably @Greg-DB can propose some additional checks.

Unfortunately I don't know what the issue may be here. We'll try to look into it, but I can't make any promises as it seems to be client-specific, based on the observation that it doesn't occur on your other computer.

Thank you Greg, I understand.

I will communicate if I make new advances.

https://www.raspberrypi.org/forums/viewtopic.php?t=223026

@ags65 We looked into this, and it seems like you may be running in to this issue:

As noted there, you can try:

To stop sending DFs, you can change net.ipv4.ip_no_pmtu_disc value to 1 in sysctl.
I would also try reducing the MTU on the interface in order to advertise a smaller MSS to the TCP peer.