Child Item

Custom Tool - Slow File Upload Speed

We have developed a tool to help our customers move data from Windows File System to Dropbox.

The upload is done in batches of 8 files using the Dropbox APIs - UploadSessionStartAsync, UploadSessionFinishBatchAsync and UploadSessionFinishBatchCheckAsync. Uploads were tested keeping the chunk size between 10-40 MB.

The upload speed we experienced using our tool varied from 1 Mbps to 10 Mbps (Megabits per second) which was very slow.
We want our tool to upload data in tune of several TBs which would take unreasonable time to complete.

Our experience with a third party tool - CFP (Cloud Fast Path) was very good as we experienced speeds of up to 210 Mbps (Megabits per second) on the same machine.

We tested both tools on a machine in our network as well as on an Azure VM.

How can we optimize and improve our tool to achieve faster upload speeds ?

Thanks,
Gagan

Find more posts tagged with

API

Abuse

Accepted answers

Gags

Hello @Greg-DB

Thank you for the reply !!!

We used the new Dropbox API - UploadSessionStartBatchAsync and this helped in reducing the "too_many_requests" errors.

We tested 2.2 TB data sample containing 427,002 files for migration using our custom app.
Last time we faced 26,000 failures which has been reduced to 1,753 in the latest test.

What more can be done to further reduce failures due to "too_many_requests" error response from Dropbox.
We do wait and retry as shown in the last code snippet.

Thanks,
Gagan

All comments

Greg-DB

Check out the Performance Guide for some optimizations you can make to upload in a performant manner.

Additionally, for uploading large files, check out the new "concurrent" UploadSessionType option, which lets you upload multiple pieces of a single file/upload session in parallel. There's also a new UploadSessionStartBatchAsync for starting multiple upload sessions at once. These haven't been added to the Performance Guide yet, but they can also be helpful.

gagsbh80

Hello @Greg-DB

Thank-you for your reply.

I went through your suggestions and the Performance guide.
I used the new "concurrent" UploadSessionType option and that increased the upload speeds.

After the above change, for 1.5 GB and 6481 Files on Azure VM we got Sync Time = 10 mins which earlier was above 50 minutes.
For 1 TB and 446 Files, Sync Time = 2 hours 47 mins which was very fast.

Upload speeds were between 300-700 Mbps and sometimes even 1.5 Gbps which is way above what we got earlier i.e. 1-7 Mbps.
However, this did not last long as few hours later when I tested again with same data, upload speeds were back to 1-7 Mbps.

What could be the reason for the fallback to lower upload speeds of 1-7 Mbps ?
Does Dropbox throttles multiple file uploads coming from the same IP or App ?
However, I did not encounter any upload failures or Rate Limit exceptions.

I know Dropbox has a rate limit policy which can be found out using https://www.dropbox.com/developers/documentation/http/teams#team-features-get_values
For the team we tested with the Upload Rate Limit was reported as 1,000,000,000 (1 billion).

We tested uploading files > 6000 using Batch upload APIs (with UploadSessionType:concurrent) for one of the Team account member. The upload speeds were again between 1-7 Mbps when we tested with another member of the same Team.
In real-life scenarios, the number of files can be as large as 200,000 files.

We are using a Dropbox Teams App for one-time Admin authentication and token generation.
I came across this forum post: Forum Post

Our Dropbox App is Development stage as of now and we are not Dropbox partners. Can that be one of the reasons ?

Can you please help us figure out the real reason for the reduction in upload speeds after we got great speeds for some hours.

Thanks,
Gagan

Greg-DB

Dropbox does not throttle connection speeds like that. While Dropbox has a rate limiting system in place, that operates on the basis of calls per time period, and any rate limited calls would result in a specific error, not limited connection speed.

Also, an app's development/production status does not affect connection speed, nor does whether or not it is a partner app.

As for your actual connection speed, I'm afraid I can't offer much insight based on this. Your connection speed to Dropbox depends on the routing you get between your ISP and our servers, and may be slower than your ISP's rated speeds. Sometimes resetting or retrying your connection gets you a different route and better speeds, but that is outside of our control. Some ISPs also throttle sustained connections so if you see an initial high connection speed followed by lower speeds, that could be the reason.

Beyond that, you may want to add some logging to see exactly what operations are being performed (or not) when your overall speed drops. It'spossible this is due to how you're ordering/parallelizing the calls. For instance, you might be running multiple sessions in parallel initially, but then "run out" of things to run in parallel, for example. If something on the Dropbox API doesn't appear to be working properly though, please share whatever output you can showing the issue so we can take a look.

gagsbh80

Hello @Greg-DB

We tested with Upload Session Finish list count of Max = 1000 in Upload SessionFinishBatchAsync and we started getting better speeds. We are continuing our testing and will let you know if we need further help.

Thanks,

Gagan

Gags

Hello @Greg-DB

Upload speeds are not major concern as of now but failures are.

We did a 2 TB migration using our custom tool and encountered "too_many_requests" failure as the end stages of the migration.

Thousands of files reported such failures which won't be acceptable to users.

We upload many files in parallel but using UploadSessionStartAsync, UploadSessionAppendV2Async & UploadSessionFinishBatchAsync APIs.

We do a wait and retry for Rate Failures as recommended by Dropbox and so we thought we should be good.

However, many files still fail due to "too_many_requests".

I am writing the relevant C# code snipped we have implemented for your reference.

Can you please give us some direction to minimize the Rate Failure exceptions and hence reduce upload failures.

        public async Task SendUploadFragment(DropboxUploadSession uploadSession, byte[] fragmentBuffer, long offset)
        {
            string errorlog = ConfigurationManager.AppSettings["ErrorLog"];
            int retry = 3;
            DropboxApi.DropboxClient clientadmin = null;
            DropboxApi.DropboxTeamClient teamclient = null;

            try
            {
                for (int i = 1; i <= retry; i++)
                {
                    try
                    {
                        Item data = new Item();

                        if (!String.IsNullOrEmpty(this.CurrentToken.RefreshToken))
                        {
                            string appkey = SyncProDropboxAppKey;
                            string appsecret = SyncProDropboxAppSecret;
                            teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.RefreshToken, appkey, appsecret);
                        }
                        else
                        {
                            teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.AccessToken);
                        }

                        clientadmin = teamclient.AsMember(this.TeamMemberId);

                        using (var memStream = new MemoryStream(fragmentBuffer, 0, fragmentBuffer.Length))
                        {
							//1. When No Upload session has been created so far
                            if (String.IsNullOrEmpty(uploadSession.SessionId))
                            {							  
							  try
                              {
							        //If multiple fragment session
									if (!uploadSession.islastfragment)
									{
										UploadSessionStartArg args = new UploadSessionStartArg();
										UploadSessionType sessiontype = new UploadSessionType();
										var result = await clientadmin.Files.UploadSessionStartAsync(false, sessiontype.AsConcurrent, memStream);
										uploadSession.SessionId = result.SessionId;
									}
									else
									{ 
									 //If single fragment session
											UploadSessionType sessiontype = new UploadSessionType();

											var result = await clientadmin.Files.UploadSessionStartAsync(true, sessiontype.AsConcurrent, memStream);
											uploadSession.SessionId = result.SessionId;

											var cursor2 = new UploadSessionCursor(uploadSession.SessionId, (ulong)(offset + fragmentBuffer.Length));
											UploadSessionFinishArg arg = new UploadSessionFinishArg(cursor2, new CommitInfo(uploadSession.Item.PathDisplay));

											args.Add(arg);
											break;
									 }
                                }
								catch (Exception ex)
                                {     
							            //Log Exception
                                        if (ex is DropboxApi.RateLimitException)
                                        {
                                            try
                                            {
                                                Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
                                            }
                                            catch (Exception ex1)
                                            {

                                            }
                                        }

                                        Thread.Sleep(1000);
                                        throw ex;                                     
                                }                              
                            }
                            else
                            {
                                var cursor = new UploadSessionCursor(uploadSession.SessionId, (ulong)offset);

								//If last fragment of session
                                if (uploadSession.islastfragment)
                                {
                                    try
                                    {
                                        await clientadmin.Files.UploadSessionAppendV2Async(cursor, true, memStream);
                                        var cursor2 = new UploadSessionCursor(uploadSession.SessionId, (ulong)(offset + fragmentBuffer.Length));
                                        UploadSessionFinishArg arg = new UploadSessionFinishArg(cursor2, new CommitInfo(uploadSession.Item.PathDisplay));
                                        args.Add(arg);
                                    
                                        break;
                                    }
                                    catch (Exception ex)
                                    {      
										//Log Exception									
                                        if (ex is DropboxApi.RateLimitException)
                                        {
                                            try
                                            {                                                
                                                Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
                                            }
                                            catch (Exception ex1)
                                            {

                                            }
                                        }

                                        Thread.Sleep(1000);                                        
                                    }                                    
                                }
                                else  //If not the last fragment of Session
                                {
                                    try
                                    {
                                        await clientadmin.Files.UploadSessionAppendV2Async(cursor, false, memStream);
                                        break;
                                    }
                                    catch (Exception ex)
                                    {   
										//Log Exception
                                        if (ex is DropboxApi.RateLimitException)
                                        {
                                            try
                                            {                                                
                                                Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
                                            }
                                            catch (Exception ex1)
                                            {

                                            }
                                        }

                                        Thread.Sleep(1000);                                     
                                    }
                                }
                            }
                        }
                        break;
                    }
                    catch (Exception ex)
                    {
                        string isgeneratelog = Global.IsErrorLog == false ? "0" : "1";                        
                        if (ex is DropboxApi.RateLimitException)
                        {
                            try
                            {                                
                                Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
                            }
                            catch (Exception ex1)
                            {                                
                            }
                        }

                        Thread.Sleep(1000);                      
                    }
                    finally
                    {
                        try
                        {
                            if (teamclient != null)
                            {
                                teamclient.Dispose();
                                teamclient = null;
                            }

                            if (clientadmin != null)
                            {
                                clientadmin.Dispose();
                                clientadmin = null;
                            }
                        }
                        catch (Exception ex)
                        {

                        }
                    }
                }
            }
            catch (Exception ex)
            {
                string isgeneratelog = Global.IsErrorLog == false ? "0" : "1";
                if (isgeneratelog == "1")
                {
                    try
                    {
                        System.IO.File.AppendAllText(errorlog, "\r\nDateTime: " + DateTime.Now.ToString() + "\r\nPathDisplay: " + uploadSession.Item.PathDisplay +

                        "\r\n" + ex.Message + "\r\n" + ex.StackTrace + "\r\n");
                    }
                    catch (Exception ex1)
                    {

                    }
                }

                throw ex;
            }
            finally
            {
                clientadmin = null;
                fragmentBuffer = null;
                teamclient = null;
            }
        }



		public async Task<string> UploadBatchFinish()
        {
            DropboxApi.DropboxTeamClient teamclient = null;
            DropboxApi.DropboxClient clientadmin = null;

            try
            {
                
                if (!String.IsNullOrEmpty(this.CurrentToken.RefreshToken))
                {
                    string appkey = SyncProDropboxAppKey;
                    string appsecret = SyncProDropboxAppSecret;
                    teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.RefreshToken, appkey, appsecret);
                }
                else
                {
                    teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.AccessToken);
                }

                clientadmin = teamclient.AsMember(this.TeamMemberId);
               
                UploadSessionFinishBatchArg batchArg = new UploadSessionFinishBatchArg(args);
                var result = await clientadmin.Files.UploadSessionFinishBatchAsync(batchArg);
               
                if (result != null && result.AsAsyncJobId != null && result.AsAsyncJobId.Value != null)
                {
                    return result.AsAsyncJobId.Value;
                }               
            }
            catch (Exception ex)
            {                
                throw ex;
            }
            finally
            {
                teamclient = null;
                clientadmin = null;
            }

            return "";
        }

Thanks,

Gagan

Greg-DB

The 'too_many_requests' error indicates that your app is hitting the actual Dropbox rate limiting system. We don't have specific rate numbers documented, nor can we increase the rate limits for any particular, app, user, or team. Apps should be written to handle these rate limit responses automatically, respecting the Retry-After value, and you should try to minimize the number of calls however you can. I recommend referring to the error documentation and Error Handling Guide for more information.

For example, I see you're using UploadSessionStartAsync and you mentioned you're uploading many files in parallel, so it would be better to use UploadSessionStartBatchAsync instead, as that would let you start multiple upload sessions in one call, reducing how many calls you need to make.

Gags

Thank-you @Greg-DB

I will try to incorporate your suggestions in my code logic.

I would use UploadSessionStartBatchAsync to start multiple upload sessions in one call. Can I use it to start max-allowed 1000 upload sessions or is that an ill-advised decision. Do I need to start with lesser upload sessions - say 500 session using one call to UploadSessionStartBatchAsync to avoid Rate Limit issue ?

Also, once I start 'x' number of upload sessions using UploadSessionStartBatchAsync , can I again start another 'y' no. of upload sessions using UploadSessionStartBatchAsync before calling UploadSessionFinishBatchAsync for the first batch of 'x' upload sessions ?

In other words, should I wait for UploadSessionFinishBatchAsync for the first batch of 'x' upload to be completed before I make another call to UploadSessionStartBatchAsync

Thanks,

Gagan

Thanks,

Gagan

Greg-DB

You can use UploadSessionStartBatchAsync to start up to 1000 upload sessions. If you have at least 1000 upload sessions to start, you can do so in one call; you do not need to lower that. Calling UploadSessionStartBatchAsync once only counts as one call, regardless of what you set numSessions to.

And you can call UploadSessionStartBatchAsync multiple times before calling UploadSessionFinishBatchAsync if needed. You don't need to wait for one batch to finish before starting another. Just keep in mind that you should only finish one batch per namespace at a time to avoid lock contention.

Gags

Hello @Greg-DB

Thank you for the reply !!!

We used the new Dropbox API - UploadSessionStartBatchAsync and this helped in reducing the "too_many_requests" errors.

We tested 2.2 TB data sample containing 427,002 files for migration using our custom app.
Last time we faced 26,000 failures which has been reduced to 1,753 in the latest test.

What more can be done to further reduce failures due to "too_many_requests" error response from Dropbox.
We do wait and retry as shown in the last code snippet.

Thanks,
Gagan

Greg-DB

It sounds like you're using all of the available batch endpoints for this use cases, so the only remaining change that comes to mind would be to increase the chunk size, so that you use fewer requests for sending the file data overall.

You can use a chunk size of up to 150 MB, but be aware that increasing the chunk size increases how long each request takes and consequently increases the chances of the network connection failing before it completes. And further, using a larger chunk size means more data needs to be sent on a retry.