Child Item

File uploading slow

Hi,

I'm using the javscript sdk, and trying to upload multiple files (one after another) with dbx.filesUpload. Due to the nature of my projectm I have to upload multiple files when the user sends a request. Each file is taking around a second, which results in long waiting times for the user when they need, say, 50 files.

My code:

async function uploadMultipleFiles (storageFilePaths,packFilePaths,packRoot) {
	try {
		for (i in storageFilePaths) {
			console.log(await dbx.filesUpload({ path: packRoot+packFilePaths[i], contents: fs.readFileSync(storageFilePaths[i])}))
		}
	} catch (err) {
		throw err
	}
}

Full code is here. I have to use await; see this thread. Would there be a quicker way to do this without getting too many requests? Would it be quicker to copy the files from another place in my dropbox rather than upload them like this?

Find more posts tagged with

API

Abuse

Comments

Greg-DB

There are a number of variables that will affect overall performance of transfers like this, some of which can't be controlled unfortunately. There are some other options that may be helpful though, depending on the specifics of your scenario.

If the files do already exist in the Dropbox account and you just need to make copies of them, using filesCopyBatchV2 may be faster.

Or, if the files already exist in another Dropbox account, and you have authorization to both of them, you may want to try using filesCopyReferenceGet and filesCopyReferenceSave instead.

Otherwise, if you have a significant number of files to upload, you may want to use the "Batch Upload" method described in the Performance Guide. That way, you can upload the file data for several files at a time in parallel, using filesUploadSessionStart and filesUploadSessionAppendV2 as needed, and then commit them all at once when those are done using filesUploadSessionFinishBatch.

Beatso

Thanks for your help. Using the batch upload method, how to I "group files that are being uploaded into one or more batches"? Also, do I use filesUploadSessionStart for each file, or do I use it once, and append for every other file? Is there anywhere where I could find an example of uploading multiple files with a batch upload?

Edit: I found your example here. My code is as follows:

var entries = []
for (i in storageFilePathsToUpload) {
	let fileData = fs.readFileSync(storageFilePathsToUpload[i])
	dbx.filesUploadSessionStart({
		contents: fileData,
		close: true,
	})
	.then(function (response) {
		entries.push({cursor:{session_id:response.session_id,offset:fileData.length},commit:{path:packPath+packFilePathsToUpload[i]}})
		console.log("entriesinprog")
	})
	.catch(function (err) {
		console.log(err);
	});
}

console.log(entries)
dbx.filesUploadSessionFinishBatch({entries:entries})
.then(function(response){
	dbx.filesUploadSessionFinishBatchCheck({async_job_id: response.async_job_id})
	.then(function(response){
		console.log(response)
	})
	.catch(function(err){
		console.log(err)
	})

	}).catch(err => {
		console.log(err)
	})

It's not working because it's not executing in the right order. The filesUploadSessionFinishBatchCheck is logging its response before the filesUploadSessionStart has finished uploading. Why is this, and how could I only finish the batch once the for loop has finished? (I'm not in an async function). It might sort of relate to this thread, but I obviously can't use .then on a for loop.

Thanks once again for your help.

Edit 2: I got the above working, and now my code looks like this:

async function startSessions () {
	entries = []
	for (let [index,val] of storageFilePathsToUpload.entries()) {
		let fileData = fs.readFileSync(val)
		await dbx.filesUploadSessionStart({
			contents: fileData,
			close: true,
		})
		.then(function (response) {
			entries.push({cursor:{session_id:response.session_id,offset:fileData.length},commit:{path:packPath+packFilePathsToUpload[index]}})
		})
		.catch(function (err) {
			console.error(err)
		})
	}
}


startSessions().then(()=>{
	console.log(entries)
	dbx.filesUploadSessionFinishBatch({entries:entries})
	.then(function(response){
		dbx.filesUploadSessionFinishBatchCheck({async_job_id: response.async_job_id})
		.then(function(output){
			console.log(output) // THIS RETURNS { '.tag': 'in_progress' }
			// get share link and send it
			getShareLink("/test").then((response)=>{res.send(response)}).catch((error)=>{res.send("error");console.error(error)})
		})
		.catch(function(err){
			console.error(err)
		})
	}).catch(err => {
		console.error(err)
	})

}).catch((err)=>{console.error(err)})

When the output of filesUploadSessionFinishBatchCheck is logged, it gives { '.tag': 'in_progress' }. Is there a way to not run the code until the batch is finished?

Greg-DB

Your app needs to call filesUploadSessionFinishBatchCheck occasionally to determine when the batch has finished, and only then proceed with whatever other operations it needs to run based on the result (similar to your other thread, except that you may need to call a few times). That batch operation is happening on the Dropbox servers, so your app won't know by itself when that's done without checking with the servers via that method.

So, you should have your app poll filesUploadSessionFinishBatchCheck occasionally, say, every few seconds, until FilesUploadSessionFinishBatchJobStatus is no longer 'in_progress'. Then you can check the FilesUploadSessionFinishBatchResult in FilesUploadSessionFinishBatchJobStatus.complete to see the results and continue from there.