Child Item

Best Practices for Chunked Upload in Node.js

Hello,

I have had a lot of success so far using the HTTP endpoints to make a web app, but when I tried to switch from a regular upload to a chunked upload I've run into all sorts of problems. Primarily, it seems like I keep getting a brand new session id each time I make a request, which means that I never get a file that is more than 0 bytes. Here's what I have:

request.post('https://content.dropboxapi.com/2/files/upload_session/start', {
		    headers: { Authorization: 'Bearer ' + token, 'Content-Type': 'application/octet-stream'
			     }, body: readableStream.read(),
		}, function(err, httpResponse, bodymsg,res) {
		    if (err) {
			console.log("Error in uploading "+err);
		    }
		    if(!err) {
			console.log(bodymsg);
			sessionid=JSON.parse(bodymsg).session_id;
			var offst=0;
			readableStream.on('readable', function() {  
			    while ((chunk=readableStream.read()) != null) {
				data += chunk;
				request.put('https://content.dropboxapi.com/2/files/upload_session/append_v2', {
				    headers: { Authorization: 'Bearer ' + token, 'Content-Type': 'application/octet-stream'
					     }, body: chunk,
				    form : {
					cursor:{
					    session_id : sessionid,
					    offset : offst
					}
				    }
				}, function(err, httpResponse, bodymsg,res) {
				    if(err){
					console.log(err)
				    }
				    console.log(httpResponse);
				    console.log('Made it?');
				})
			    }
			})
		    }
		});

I know I need to have a request to end the upload, but I have not even been able to append to my upload yet. I get a session ID, but then I get a brand new one it seems during the next call. Has anyone done this or can point me towards where I can get some help?

Find more posts tagged with

API

Abuse

Comments

Greg-DB

Can you elaborate on what you mean when you say you "get a brand new [session ID] it seems during the next call"? Only /files/upload_session/start returns session IDs. Can you log and share the responses you're getting?

Kevin L.28

When I say that, I mean that when I console.log the session ID I notice that it was changing each time and I never saw the same one twice.

For example,

AAAAAAAAGJXQJ-U67pHOfg
    65536
    {"expires": "Sun, 10 Jul 2016 16:40:19 +0000", "upload_id": "AAAAAAAAGJXQJ-U67pHOfg", "offset": 65536}
[16:40:19.709Z]  INFO wt: Chunked upload called for: /tmp/get-pip.py
[16:40:19.709Z]  INFO wt: 
    AAAAAAAAGJkcP0WgJC2wAQ
    16384
    {"expires": "Sun, 10 Jul 2016 16:40:19 +0000", "upload_id": "AAAAAAAAGJkcP0WgJC2wAQ", "offset": 16384}

What I realized was that my chunkedUpload() function was being called over and over again for some unknown reason, and that I was actually getting correct files in my Dropbox so long as they were small enough to fit in a single chunk, so I believe my code is likely mostly correct here but the problem is external to the upload function.

Thus it makes sense why I was getting more than one ID, the upload_session/start endpoint is getting hit multiple times and I need to find a way in my code to make sure it only attempts this once per file.

Greg-DB

I see, yes, only the start endpoint gives you a new session ID, so it does look like your method is getting called more than once. You only need to call successfully start once per file though.

If you're having trouble with the API once you sort that part out, let us know.

Steve M.

A couple other issues I can spot:

Your code doesn't appear to ever call /upload_session/finish. You'll need to detect when you've uploaded the entire file and use that endpoint to close out the upload session.
A while loop can't work here. You need to wait for each call to /upload_session/append_v2 to complete before calling again. Your code kicks off a bunch of concurrent HTTP requests, so I'd expect you to see error responses from the server on at least some of them.
You should be updating the value of `offset` when /upload_session/append_v2 returns. (This is part of what you'll do as you turn the while loop into a recursive call instead.) Otherwise, all your calls to /upload_session/append_v2 will try to use the same offset, and all but one call (the first one to make it through) will fail.
You may also need to put a size limit on your calls to `read`. I believe your current code has the potential of reading too much data and exceeding the maximum size allowed by /upload_session/start and /upload_session/append_v2.