Child Item

Uploading big file using Objective-C API V2 very SLOW...

Hello,

recently our app updates the Dropbox related code from API V1 to API V2. Usually we need to transfer files to Dropbox, it is OK for small files, but for video files of size about 1.5 GB the progress is very slow, and often the upload is stopped due to request timeout....

We use the following API calls:

1. Start the upload

[client.filesRoutes uploadSessionStartData:data];

2. Repeatedly upload new chunk of data

[client.filesRoutes uploadSessionAppendV2Data:cursor inputData:chunk];

3. Finish upload for the last chunk

[client.filesRoutes uploadSessionFinishData:cursor commit:info inputData:chunk];

And the log messages are as follows:

2017-01-20 14:25:11.336692 MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0000

2017-01-20 14:25:11.343177MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0000

2017-01-20 14:25:11.349521MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0001

2017-01-20 14:25:11.353344MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0001

// Skip...

2017-01-20 14:25:55.059215MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0003

2017-01-20 14:25:55.070222MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0003

2017-01-20 14:26:17.700571MyApp[19184:6471639] start: bytesWritten = 65536 progress = 0.0003

2017-01-20 14:26:17.704239MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0003

2017-01-20 14:26:34.987083MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0004

2017-01-20 14:26:34.992280MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0004

2017-01-20 14:26:35.005880MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0004

// No data downloaded for about 23 sec

2017-01-20 14:26:58.848907MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0004

2017-01-20 14:26:58.855699MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0004

2017-01-20 14:26:58.871372MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0005

2017-01-20 14:26:58.873328MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0005

2017-01-20 14:27:15.431969MyApp[19184:6471639] start: bytesWritten = 65536 progress = 0.0005

2017-01-20 14:27:15.436307MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0006

2017-01-20 14:27:44.313052MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0006

2017-01-20 14:27:44.318937MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0006

2017-01-20 14:27:44.323024MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0006

2017-01-20 14:27:44.327415MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0006

2017-01-20 14:28:10.012220MyApp[19184:6471639] start: bytesWritten = 65536 progress = 0.0007

2017-01-20 14:28:10.015194 MyApp[19184:6471639] start: bytesWritten = 32768 progress = 0.0007

// Skip...

2017-01-20 14:59:18.994143 MyApp[19184:6471639] __70-[....]_block_invoke [Line 92] start: requestError = DropboxClientError[{

NSError = "Error Domain=NSURLErrorDomain Code=-1001 \"\U8981\U6c42\U903e\U6642\U3002\" UserInfo={NSUnderlyingError=0x17025bb70 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 \"(null)\" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, NSErrorFailingURLStringKey=https://api-content.dropbox.com/2/files/upload_session/start, NSErrorFailingURLKey=https://api-content.dropbox.com/2/files/upload_session/start, _kCFStreamErrorDomainKey=4, _kCFStreamErrorCodeKey=-2102, NSLocalizedDescription=\U8981\U6c42\U903e\U6642\U3002}";

}];

2017-01-20 14:59:18.994833 MyApp[19184:6471639] __74-[...]_block_invoke_2 [Line 428] Failed To Upload /Horrible.Bosses.2011.DVDRip.XVID.AC3.HQ.Hive-CM8.avi

Error = DBRequestErrorClient

2017-01-20 14:59:18.995478 MyApp[19184:6475977] [] __tcp_connection_write_eof_block_invoke Write close callback received error: [89] Operation canceled

It can be seen from the log messages that the time the data arrived is not very smooth, and finally the operation is canceld.

My questions:

Is it pure network traffic problem?(Our network is relatively fast.)

Or is there something in API V2's upload API that causes the upload so slow?

Thanks for your attention.

Best regards,

Logan

Find more posts tagged with

API

Abuse

Comments

Greg-DB

Hi Logan, are you using a chunk size of 32768 bytes? You should probably use something much larger than that, at least 4 MB, or even significantly larger than that.

The optimal size depends on a variety of factors, so you may want to try a few different ones and see what works best for your app. Using a small chunk size can be useful, as it minimizes how much data you need to re-upload for failed requests, and reduces the chance of any particular request of failing, but it causes more overhead. A larger chunk size can reduce the overhead, which can improve overall transfer speed.

C. Logan

Hi Greg,

thanks for reply!

1. I have tried to pass the chunk of data with size 5 MB and 100 MB, but the result was the same. It seems that the bytesWritten from the progress block is always 32768 and in few cases also 65536, as shown in the log message. How can I modify the data chunk size for each transfer? Currently I use the following code:

NSData *data = [dropboxObj.fileHandle readDataOfLength:length]; // length = 5 MB or 100 MB or other values
uploadTask = [client.filesRoutes uploadSessionStartData:data];

2. Just a few minutes ago I tried to upload again, but there is no message from the progress block, and after one or two minutes the operation is canceled. My network is normal. So I guess there might be something wrong with the connection to Dropbox, or is there still anything in API V2 that I should know
when doing the upload ?

Thanks for your attention!

Logan

Greg-DB

1. The chunk size should just be the size of the data you supply for any given call, and that's what should get reported back in the progress callback, so it's unexpected that you'd still get 32768. Can you share the rest of the relevant code, including the definition of length, the progresc callback definitifion, etc.?

2. That also doesn't sound like the expected behavior. If you share the rest of the code and the steps to reproduce it we'll be happy to look into it.

C. Logan

// My code can be summarized as follows:

// sorry for the indentation, it seems the code indentation is gone.

const NSUInteger DB_CHUNK_SIZE = 5*1024*1024;

DropboxClient *client = [DropboxClientsManager authorizedClient];

// create dropboxObj to pass parameters
TGDropboxObject *dropboxObj = [[TGDropboxObject alloc] init];
dropboxObj.progressBlock = progress;
dropboxObj.completionBlock = completion;
dropboxObj.fileSize = [self getFileSize:localPath fromInternal:fromInternal];
dropboxObj.chunkSize = DB_CHUNK_SIZE;
dropboxObj.curTotalBytes = 0;
dropboxObj.srcPath = localPath;
dropboxObj.dstPath = remotePath;
dropboxObj.fromInternal = fromInternal;
if (fromInternal) {
dropboxObj.fileHandle = [NSFileHandle fileHandleForReadingAtPath:localPath];
}
// skip……

NSData *data = [self readDataOfLength:dropboxObj.chunkSize fromObj:dropboxObj];// read one chunk of data

DBUploadTask<DBFILESUploadSessionStartResult *, DBNilObject *> *uploadTask;

uploadTask = [client.filesRoutes uploadSessionStartData:data];
[uploadTask response:^(DBFILESUploadSessionStartResult * _Nullable result, DBNilObject * _Nullable obj, DBRequestError * _Nullable requestError) {

if (result) { // success for the first chunk
dropboxObj.sessionId = result.sessionId;

DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:result.sessionId offset:[NSNumber numberWithUnsignedLongLong:[self getOffsetFromObj:dropboxObj]]];

[self uploadNextFromObj:dropboxObj cursor:cursor];// do upload for subsequent chunks
}
else { // failure
[self closeFileFromObj:dropboxObj];

// skip…

dropboxObj.completionBlock(nil, error);
}
}];

// progress block for uploadTask
[uploadTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
dropboxObj.curTotalBytes += bytesWritten;
CGFloat progress = (CGFloat)dropboxObj.curTotalBytes / (CGFloat)dropboxObj.fileSize;
NSLog(@start: bytesWritten = %lld progress = %.4f, bytesWritten, progress);
dropboxObj.progressBlock(progress);
}];

// The method for uploading subsequent chunks:

//-------------------------------------------------------
- (void)uploadNextFromObj:(TGDropboxObject *)dropboxObj cursor:(DBFILESUploadSessionCursor *)cursor {

DropboxClient *client = [DropboxClientsManager authorizedClient];
NSData *const chunk = [self readDataOfLength:dropboxObj.chunkSize fromObj:dropboxObj];
const BOOL isLastChunk = chunk.length < dropboxObj.chunkSize;

if (isLastChunk == NO) {
DBUploadTask<DBNilObject *, DBFILESUploadSessionLookupError *> *continueTask;

continueTask = [client.filesRoutes uploadSessionAppendV2Data:cursor inputData:chunk];
[continueTask response:^(DBNilObject * _Nullable obj, DBFILESUploadSessionLookupError * _Nullable lookupError, DBRequestError * _Nullable requestError) {

if (lookupError == nil) { // success
DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:dropboxObj.sessionId offset:[NSNumber numberWithUnsignedLongLong:[self getOffsetFromObj:dropboxObj]]];

[self uploadNextFromObj:dropboxObj cursor:cursor];

} else { // failure
[self closeFileFromObj:dropboxObj];

// skip…

dropboxObj.completionBlock(nil, error);
}
}];

// progress block for continueTask
[continueTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
dropboxObj.curTotalBytes += bytesWritten;
CGFloat progress = (CGFloat)dropboxObj.curTotalBytes / (CGFloat)dropboxObj.fileSize;
NSLog(@continue: bytesWritten = %lld progress = %.4f, bytesWritten, progress);
dropboxObj.progressBlock(progress);
}];
}
else { // last chunk
DBUploadTask<DBFILESFileMetadata *, DBFILESUploadSessionFinishError *> *finishTask;
DBFILESCommitInfo *info = [[DBFILESCommitInfo alloc] initWithPath:dropboxObj.dstPath];

finishTask = [client.filesRoutes uploadSessionFinishData:cursor commit:info inputData:chunk];
[finishTask response:^(DBFILESFileMetadata * _Nullable metaData, DBFILESUploadSessionFinishError * _Nullable finishError, DBRequestError * _Nullable requestError) {
DLog(@metaData: %@", metaData);
[self closeFileFromObj:dropboxObj];

// skip…

dropboxObj.completionBlock(metaData, error);
}];
[finishTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
dropboxObj.curTotalBytes += bytesWritten;
CGFloat progress = (CGFloat)dropboxObj.curTotalBytes / (CGFloat)dropboxObj.fileSize;
NSLog(@finish: bytesWritten = %lld progress = %.4f, bytesWritten, progress);
dropboxObj.progressBlock(progress);
}];
}
}

// If there is anything unclear, just tell me, I can provide further infomation.

Greg-DB

Thanks! Your definition DB_CHUNK_SIZE of seems like it should be fine, but I don't see exactly where it's actually being used. Can you share the definitions of TGDropboxObject, readDataOfLength, getOffsetFromObj, closeFileFromObj, and anything else necessary so I can try to run this? Thanks in advance!

C. Logan

You cannot run the code without our external device. But the following code does the same thing and produces the the same result.

You need to provide the source file to be uploaded. The destination file is hard coded (/temp/test.mp4), you can modify it to another file path. The unused parameters in the API uploadInternal:files:index:fromDir:toDropboxDir:opType:resArray: can be ignored.

If there is anything unclear, feel free to point out.

const NSUInteger DB_BLOCK_SIZE = 5*1024*1024; // 102400000;

@interface DestVC ()

@property DBUploadTask<DBFILESUploadSessionStartResult *, DBNilObject *> * uploadTask;

@property DBUploadTask<DBNilObject *, DBFILESUploadSessionLookupError *> * continueTask;

@property DBUploadTask<DBFILESFileMetadata *, DBFILESUploadSessionFinishError *> *finishTask;

@property unsigned long long offset;

@property NSFileHandle *file;

@property NSData *data;

@property NSNumber *fileSize;

@property DropboxClient *client;

@property NSString *sessionId;

@end

@implementation DestVC

- (void)uploadInternal:(NSArray *)items files:(NSArray *)files index:(int)index fromDir:(NSString *)fromDir toDropboxDir:(NSString *)toDir opType:(OperationType)opType resArray:(NSMutableArray *)resArray {

NSString *file = [files objectAtIndex:index];

NSString *srcPath = [fromDir stringByAppendingPathComponent:file];// define your own source file to be uploaded

// NSString *dstPath = [toDir stringByAppendingPathComponent:file];// define your own destination file in Dropbox

_client = [DropboxClientsManager authorizedClient];

if (!_client) {

[DropboxClientsManager authorizeFromController:[UIApplication sharedApplication]

controller:self

openURL:^(NSURL *url) {

[[UIApplication sharedApplication] openURL:url];

}

browserAuth:YES];

} else {

_fileSize = [FCFileManager sizeOfFileAtPath:srcPath];// get file size

_file = [NSFileHandle fileHandleForReadingAtPath:srcPath];

_data = [_file readDataOfLength:DB_BLOCK_SIZE];

_uploadTask = [_client.filesRoutes uploadSessionStartData:_data];

[_uploadTask response:^(DBFILESUploadSessionStartResult * _Nullable result, DBNilObject * _Nullable obj, DBRequestError * _Nullable error) {

DLog(@result: %@", result);

DLog(@routeError: %@", obj);

DLog(@error: %@", error);

if (error == nil) {

_sessionId = result.sessionId;

DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:result.sessionId offset:[NSNumber numberWithUnsignedLongLong:_file.offsetInFile]];

[self uploadNext:cursor];

} else {

[_file closeFile];

}

}];

[_uploadTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {

NSLog(@Start: bytesWritten = %lld, bytesWritten);

}];

}

return;

}

- (void)uploadNext:(DBFILESUploadSessionCursor *)cursor {

if ([_fileSize unsignedLongLongValue] - _file.offsetInFile > DB_BLOCK_SIZE) {

_data = [_file readDataOfLength:DB_BLOCK_SIZE];

} else {

unsigned long long rest = [_fileSize unsignedLongLongValue] - _file.offsetInFile;

_data = [_file readDataOfLength:rest];

}

if (_data.length < DB_BLOCK_SIZE) {

DBFILESCommitInfo *info = [[DBFILESCommitInfo alloc] initWithPath:@/temp/test.mp4];// define your destination file

_finishTask = [_client.filesRoutes uploadSessionFinishData:cursor commit:info inputData:_data];

[_finishTask response:^(DBFILESFileMetadata * _Nullable meta, DBFILESUploadSessionFinishError * _Nullable finishError, DBRequestError * _Nullable error) {

DLog(@meta: %@", meta);

DLog(@finishError: %@", finishError);

DLog(@error: %@", error);

}];

return;

}

_continueTask = [_client.filesRoutes uploadSessionAppendV2Data:cursor inputData:_data];

[_continueTask response:^(DBNilObject * _Nullable obj, DBFILESUploadSessionLookupError * _Nullable lookupError, DBRequestError * _Nullable error) {

DLog(@obj: %@", obj);

DLog(@lookupError: %@", lookupError);

DLog(@error: %@", error);

if (error == nil) {

DLog(@_file.offsetInFile: %lld, _file.offsetInFile);

DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:_sessionId offset:[NSNumber numberWithUnsignedLongLong:_file.offsetInFile]];

[self uploadNext:cursor];

} else {

[_file closeFile];

}

}];

[_continueTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {

NSLog(@Continue: bytesWritten = %lld, bytesWritten);

}];

}

@end

C. Logan

Remark: File size can also be obtained through following code instead of using FCFileManager:

NSDictionary *attributes = [[NSFileManager defaultManager] attributesOfItemAtPath:path error:NULL];

fileSize = [attributes fileSize];

Greg-DB

Thanks! That's helpful

A few notes:

- I misunderstood what was being logged in your original post, and testing the code I can confirm the issue isn't the chunk size being 32768. That's the progress callback is showing incremental progress within each single chunk upload call. Your chunk size is working at 5 MB.

- In your latest code though, note that the comment is wrong. The actual code sets a 5 MB chunk size, but the comment indicates 100 MB.

- Testing this code, I have a better idea of where the overhead is. We're looking into it and I'll follow up here once I have more information.

C. Logan

Thanks for the reply!

The comment is just for testing different value of chunk size. I have tested 5 MB and 100 MB, but the result is the same.

If you have tested the upload using the code I provided, did you get the same result? I mean, is the upload blocked/stopped or canceled after a few seconds?

It would be great if you could help us out from the problem described above.

Hope to hear from you soon when you have more information about how to correctly use the API V2 and get rid of the problem.

Greg-DB

I was able to reproduce the slow uploads, due to what appears to be a performance regression on our side. I wasn't able to reproduce exactly the same error you're getting in some cases, but I believe these are both related to the issue we found. We're looking into it.

Pooja_S

Hi Greg,

I am also facing the same issue using API V2. File uplaod seems to be very slow. I have tried with 1.5 Gb file with chunk size 4MB. It is going on more than 6 hrs.

Good to hear your team is already looking into it. Looking forward for the solution.

Thanks,

Pooja

mikewu1211

Hi,

Any update?

Greg-DB

The team has improved this somewhat, but is still working on bringing this back down more.

Bmcdonald1

We are having the same slowness issues on upload. On create folder API it is taking 10 seconds. In our product we transfer whole directory trees. Transferring 100 folders takes 100 * 10 seconds to create the folders. And we get rate limited on 90% of the requests.

Greg-DB

Issues affecting create_folder would be different than the upload_session performance issues on this thread.

I'm not seeing the issue with create_folder you reported myself though. I see you also opened a ticket, so we'll look into it and follow up with you there specifically as it sounds like that may be account-specific.

Greg-DB

The team has finished some further work on this, and is reporting the performance should be back to about previous levels. There is certainly still room left for improvement, and the team has an open feature request to improve performance in general, but I don't have a timeline for if/when further improvements would be implemented.

drowles

Hi - I'm trying to upload some files and it's taking FOREVER. I've been at this for a week. Right now - I have 3 files left that total 24 Gigs....and it SAYS that it will take 16 hours (or it says 24 hours, or 3 days, or more.)

At a 3.7 MBPS rate - I should be able to upload 22 Megs per minute - or 2.2 Gigs per hour, I think - which means this should take me 12 hours...now it says 17 hours....am I wrong in this assumption? Thanks.

Greg-DB

@drowles It sounds like you're referring to using the Dropbox desktop client. This thread was about using a particular programmatic interface, not the Dropbox desktop client. For help with the desktop client, please post in the "Desktop app issues" forum, or contact support.

drowles

WIll do - thanks much.

DS6

I am having the same trouble of extemely slow upload speeds, no matter I set chunk size to 5 MB or 100 MB. It's only 191 MB file I am uploading. Please help, there is no official iOS sample code in Objective C or Swift though. Progress callbacks always arrive every 32 KB.

DS6

Also, many a times upload simply hangs without throwing any error. Setting large block size to 100 MB causes such problems more often.

Greg-DB

@DS6 Please open a new thread with the details of your issue, e.g., the relevant code, etc. so we can help you specifically without spamming the other people on this old thread. Thanks!