Child Item

Too Many Write Operations when Uploading Files

I am getting too_many_write_operations/ at times while uploading a hierarchy of folders each containing a single file.
I am using the UploadSessionStartAsync, UploadSessionAppendV2Async, UploadSessionFinishAsync and UploadAsync APIs to upload files using fragment buffer.

Sometimes all the files get uploaded correctly but in some test runs some files do not get uploaded due to too_many_write_operations error. How do I prevent "too_many_write_operations/" error ?

Error Message:
PathDisplay: /Sandy_Source/Starting folder/SUB1/SUB2/SUB3/SUB4/TriggeredAlarm_Service_Log.txt
Retry Count: 1
too_many_write_operations/...
at Dropbox.Api.DropboxRequestHandler.<RequestJsonString>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Dropbox.Api.DropboxRequestHandler.<RequestJsonStringWithRetry>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Dropbox.Api.DropboxRequestHandler.<Dropbox-Api-Stone-ITransport-SendUploadRequestAsync>d__13`3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at SyncPro.Adapters.Dropbox.DropboxClient.<SendUploadFragment>d__49.MoveNext()

Sample Source Code:

public async Task SendUploadFragment(DropboxUploadSession uploadSession, byte[] fragmentBuffer, long offset)
{
for (int i = 1; i <= retry; i++)
{
try
{
teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.RefreshToken, appkey, appsecret);

DropboxApi.DropboxClient clientadmin = teamclient.AsMember(this.UserProfile.TeamMemberId);

using (var memStream = new MemoryStream(fragmentBuffer, 0, fragmentBuffer.Length))
{
if (String.IsNullOrEmpty(uploadSession.SessionId))
{
if (!uploadSession.islastfragment)
{
UploadSessionStartArg args = new UploadSessionStartArg();
var result = await clientadmin.Files.UploadSessionStartAsync(args, memStream);
uploadSession.SessionId = result.SessionId;
}
else
{
FileMetadata fileMetadata = null;

try
{
fileMetadata = await clientadmin.Files.UploadAsync(new CommitInfo(uploadSession.Item.PathDisplay), memStream);
break;
}
catch (Exception ex)
{
if (ex is DropboxApi.RateLimitException)
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
}
}
}
else
{
var cursor = new UploadSessionCursor(uploadSession.SessionId, (ulong)offset);

if (uploadSession.islastfragment)
{
FileMetadata fileMetadata = null;
try
{
fileMetadata = await clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(uploadSession.Item.PathDisplay), memStream);
break;
}
catch (Exception ex)
{
if (ex is DropboxApi.RateLimitException)
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
}
}
}
else
{
try
{
await clientadmin.Files.UploadSessionAppendV2Async(cursor, false, memStream);
break;
}
catch (Exception ex)
{
if (ex is DropboxApi.RateLimitException)
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
}
}
}
}
break;
}
catch (Exception ex)
{
if (ex is DropboxApi.RateLimitException)
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}

if (i == retry)
{
throw ex;
}
}
}
}

Thanks,

Gagan

Find more posts tagged with

API

Abuse

Accepted answers

Greg-DB

@gagsbh You should set close=true on the last call to on upload session where you need to upload data for that upload session. That might actually be on a UploadSessionStartAsync call (for small files), or a UploadSessionAppendV2Async call, or a UploadSessionFinishAsync call, if not using UploadSessionFinishBatchAsync. If you are using UploadSessionFinishBatchAsync, then you do need to set close=true on the last UploadSessionStartAsync or UploadSessionAppendV2Async for that upload session.

Every upload session is used to upload one file, and has one upload session ID. So, the cursor for any given upload session should have the upload session ID for that upload session, and an offset that indicates how much has been uploaded so far for that upload session.

And yes, UploadSessionFinishBatchAsync takes an IEnumerable<UploadSessionFinishArg>, which should contain one UploadSessionFinishArg per upload session (each of which needs to be closed) that you wish to finish.

All comments

Greg-DB

If you're making multiple changes at the same time in the same account or shared folder, you can run in to this 'too_many_write_operations' error, which is "lock contention". That's not explicit rate limiting, but rather a result of how Dropbox works on the back-end. This is a technical inability to make a modification in the account or shared folder at the time of the API call. This error indicates that there was simultaneous activity in the account or shared/team folder preventing your app from making the state-modifying call (e.g., adding, editing, moving, copying, sharing, or deleting files/folders) it is attempting. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here. The app will need to be written to automatically handle this error.

In short, to avoid this error, you should avoid making multiple concurrent state modifications and use batch endpoints where possible, such as UploadSessionFinishBatchAsync. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you should also implement error handling and automatic retrying as needed.

gagsbh

Thanks @Greg-DB for the information

I will look into my code and let you know if I have any more queries.

Thanks,

Gagan

gagsbh

Hello @Greg-DB ,

I reviewed my code and found that multiple concurrent threads were trying to upload files that was causing "too_many_writes" error.
Once I made the uploads serial, the issue got resolved.

I would like to make the upload of several files happen is parallel so I like to implement "UploadSessionFinishBatchAsync".
I use the following APIs:
- UploadSessionStartAsync that takes memory stream as parameter:
clientadmin.Files.UploadSessionStartAsync(args, memStream);
- UploadSessionAppendV2Async that also takes memory stream as parameter:
clientadmin.Files.UploadSessionAppendV2Async(cursor, false, memStream);
- UploadSessionFinishAsync that also takes memory stream as parameter:
clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(Item.PathDisplay), memStream);

In case most of the files are small in size and the memory stream is used completely in UploadSessionStartAsync(args, memStream), then what do I pass in memstream parameter in clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(Item.PathDisplay), memStream);

In other words, in case of single fragment for small files, can I use UploadSessionStartAsync.
If yes, what do I pass for memory stream parameter when I end session with UploadSessionFinishAsync API.

If I cannot use UploadSessionStartAsync for single fragment small files then I need to use UploadAsync API to upload small files.
How can I use "UploadSessionFinishBatchAsync" with clientadmin.Files.UploadAsync(new CommitInfo(item.PathDisplay), memStream).

I mean how do I prevent "too_many_write" error while uploading multiple files concurrently in case I have to use UploadAsync API.

Thanks,
Gagan

Здравко

@gagsbh wrote:
...
I would like to make the upload of several files happen is parallel so I like to implement "UploadSessionFinishBatchAsync".
...
In case most of the files are small in size and the memory stream is used completely in UploadSessionStartAsync(args, memStream), then what do I pass in memstream parameter in clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(Item.PathDisplay), memStream);
...

Hi @gagsbh,

Happy New Year. 🙂

Seems you are mixing different ideas! When you are using UploadSessionFinishAsync, a big file is going to be upload and put in Dropbox namespace at the end. After this call, upload session is closed and can't be used (there is no reason for such use actually); the file upload has finished. For multiple files upload with multiple upload sessions, instead of UploadSessionFinishAsync you can use UploadSessionFinishBatchAsync to finish all sessions at once, not one by one, so contentions appearing becomes less likely.

@gagsbh wrote:
...
In other words, in case of single fragment for small files, can I use UploadSessionStartAsync.
If yes, what do I pass for memory stream parameter when I end session with UploadSessionFinishAsync API.
...

Yes, you can use UploadSessionStartAsync. Even more it's mandatory. Do you know other way to start upload session?! 🧐🙂 By the way if you want to use batch mode, as I mentioned above, you haven't to use UploadSessionFinishAsync! It's NOT for use in batch mode!!! UploadSessionFinishBatchAsync UploadSessionFinishAsync are mutually exclusive! 👆

@gagsbh wrote:
...
If I cannot use UploadSessionStartAsync for single fragment small files then I need to use UploadAsync API to upload small files.
...

No, you can use either UploadSessionStartAsync or UploadAsync for small files (less than 150MB). You can't use UploadAsync for large files, where UploadSessionStartAsync can be used only. Just notice that UploadAsync doesn't start session, so can't be used in batch mode!

@gagsbh wrote:
...
How can I use "UploadSessionFinishBatchAsync" with clientadmin.Files.UploadAsync(new CommitInfo(item.PathDisplay), memStream).
...

You can NOT! As mentioned before, UploadSessionFinishBatchAsync needs set of sessions, but UploadAsync doesn't start any session.

@gagsbh wrote:
...
I mean how do I prevent "too_many_write" error while uploading multiple files concurrently in case I have to use UploadAsync API.
...

As @Greg-DB has mentioned above, you can't prevent entirely contentions ("too_many_write" error). It's matter to decrease appearance probability! You should be still able handle such a exception; take care for such in your code and be ready. Using UploadAsync is definitely not the way.

For you case (multiple small files) start a session with UploadSessionStartAsync for every file and upload it on start together with closing that uploads (use the corresponding flag). After you have all files sessions use them in a single UploadSessionFinishBatchAsync call. 😉 That's it. Instead multiple finishing for every single file, will be just a single one for all files in the set.

Hope this clarifies matter.

Greg-DB

That's correct, thanks and happy new year @Здравко!

gagsbh

Happy New Year to @Здравко and @Greg-DB !

Thank-you for the clarification.

I now understand that I need to use UploadSessionStartAsync with Close = true instead of UploadAsync.

I have another query related to the last fragment call to UploadSessionAppendV2Async

As I understand, I need to make the last call to UploadSessionAppendV2Async with Close = true parameter.

var cursor = new UploadSessionCursor(uploadSession.SessionId, offset);
await clientadmin.Files.UploadSessionAppendV2Async(cursor, true, memStream);

Do I using the same cursor as used above while initializing UploadSessionFinishArg ?

And do I then add UploadSessionFinishArg to List<UploadSessionFinishArg> as below?

UploadSessionFinishArg arg = new UploadSessionFinishArg(cursor, new CommitInfo(uploadSession.Item.PathDisplay));
args.Add(arg);

where args is if type declared globally

public List<UploadSessionFinishArg> args = new List<UploadSessionFinishArg>();

And, once all upload sessions have been added to args, I call the UploadSessionFinishBatchAsync

UploadSessionFinishBatchArg batchArg = new UploadSessionFinishBatchArg(args);
var result = await clientadmin.Files.UploadSessionFinishBatchAsync(batchArg);
if(result != null && result.AsAsyncJobId != null && result.AsAsyncJobId.Value != null)
{
return result.AsAsyncJobId.Value;
}

Finally, check if Job is complete and obtain and process FileMetadata Entries

UploadSessionFinishBatchJobStatus resultstatus = await clientadmin.Files.UploadSessionFinishBatchCheckAsync();

if (resultstatus.IsComplete)
{
  foreach (var filemetadata in resultstatus.AsComplete.Value.Entries)
  {
    //Process FileMetatData
  }
}

Since, I did not find a Dot Net SDK sample for the UploadSessionFinishBatchAsync API, I need your feedback on the above steps that I plan to implement for Batch Uploads using "UploadSessionFinishBatchAsync" API.

Thanks,

Gagan

Greg-DB