Child Item

too many write operations when uploading files

I am using Elixir wrapper to DropboxAPI to upload files, which is doing simple upload using `files/upload` route.

  def upload(client, path, file, mode \\ "add", autorename \\ true, mute \\ false) do
    dropbox_headers = %{
      :path => path,
      :mode => mode,
      :autorename => autorename,
      :mute => mute
    }

    headers = %{
      "Dropbox-API-Arg" => Poison.encode!(dropbox_headers),
      "Content-Type" => "application/octet-stream"
    }

    upload_request(
      client,
      Application.get_env(:elixir_dropbox, :upload_url),
      "files/upload",
      file,
      headers
    )
  end

But I am having an issue lately while uploading so many files parallelly, I am getting an error as `too many write operations`

I have been reading and looking into https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-start

But I am not following it totally, What I read from it, I can upload multiple files in one session, When I start a session while uploading one file, I think it works for sending chunks of a single file not, different files.

But https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-finish_batch

says, This route helps you commit many files at once into a user's Dropbox.

But how?? How I can send multiple files to Dropbox account not

Find more posts tagged with

API

Abuse

Comments

TaylorKrusen

Hello,

The data ingress guide should be of some help here. Specifically, there are some instructions towards the bottom that explain how to handle many files in parallel.

I'll paraphrase a bit here:

Organize files into batches of under 1,000
Call the files/upload_session/start endpoint once per file
Call the files/upload_session/append_v2 endpoint as needed for each file
The last call to files/upload_session/append_v2 for each file must contain a `close: true` parameter
Call the files/upload_session/finish_batch endpoint to commit the files.
- Note: you may need to call this multiple times if you're doing more than 1,000 files or you'll run into the lock contention issue again.
- This will return an `async_job_id`. You can use that to check the status of your upload with the files/upload_session/finish_batch/check endpoint.

Hope that helps!

ijunaid8989

Okay I have few questions:

How to organize files into batches of under 1000? like is there any structure for it? which is not written in documentation?

and how you can pas that batch of files to the endpoint?

First I made a new client with the access token

iex(3)> client = ElixirDropbox.Client.new("PWK08cePo_kAAAAAAAB2oRsQSyG")
%ElixirDropbox.Client{
access_token: "PWK08cePo_kAAAAAAAB2oRsQSyG"
}

and uploaded a file

iex(4)> ElixirDropbox.Files.UploadSession.start(client, false, "mix.lock")
%{"session_id" => "AAAAAAAYclutwUwG5odRew"}

with close: false,

iex(5)> ElixirDropbox.Files.UploadSession.append(client, "AAAAAAAYclutwUwG5odRew", false, "mix.exs")
{{:status_code, 409},
 {:ok,
  %{
    "error" => %{".tag" => "incorrect_offset", "correct_offset" => 18597},
    "error_summary" => "incorrect_offset/."
  }}}

I uploaded another file now, with the session id, and it gave me error as above?

What is the proper way of sending the batches of files?

https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-start I don't see any such information here?

Greg-DB

To clarify a bit, you need one "upload session" per file. Each upload session gets its own session ID. When adding data to a single upload session, you need to identify how much you've already added to that upload session by using the "offset" value. This helps make sure you're uploading the right pieces. If you specify the wrong current offset for an upload session, you'll get that "incorrect_offset" error.

You can then "finish" up to 1000 files together (i.e., up to 1000 upload sessions) using /2/files/upload_session/finish_batch.

Check the /2/files/upload_session/finish_batch documentation for information on how to format the parameters for it. Be sure to click on "UploadSessionFinishArg", and so on, to expand the documentation for the nested fields.

I also recommend using the API v2 Explorer to help guide you in understanding how to build the parameters. Click the "Add" button to add an entry and fill in the fields, for instance, and then click "Show Code" to see what the resulting code would be for the call.

I don't believe we have a sample for using upload sessions with /2/files/upload_session/finish_batch, but there are some examples of just using upload sessions (with some of the SDKs) here, which should be a good starting point for the logic:

ijunaid8989

So at first, I ran this in the command line

curl -X POST https://content.dropboxapi.com/2/files/upload_session/start \
    --header "Authorization: Bearer PWK08cePo_kAAAAAAAB2oRsQSyGQbQ2qy-SC69zC7G" \
    --header "Dropbox-API-Arg: {\"close\": false}" \
    --header "Content-Type: application/octet-stream" \
    --data-binary @local_file.txt

It returned:

{"session_id": "AAAAAAAbQhipmPJOosGS-Q"}

I don't see any offset values here but let's take a look at next request of append.

curl -X POST https://content.dropboxapi.com/2/files/upload_session/append_v2 \
    --header "Authorization: Bearer C69zip6zi9bwHgqhYHNYQbrpZ5C7G" \
    --header "Dropbox-API-Arg: {\"cursor\": {\"session_id\": \"AAAAAAAbQhipmPJOosGS-Q\",\"offset\": 0},\"close\": false}" \
    --header "Content-Type: application/octet-stream" \
    --data-binary @local_file.txt

It returned:

{"error_summary": "incorrect_offset/.", "error": {".tag": "incorrect_offset", "correct_offset": 37}}

Now I sent the offset value 37 and it gave me null as

curl -X POST https://content.dropboxapi.com/2/files/upload_session/append_v2     --header "Authorization: Bearer SC69zip6zi9bwHgqhYHNYQbrpZ5C7G"     --header "Dropbox-API-Arg: {\"cursor\": {\"session_id\": \"AAAAAAAbQhipmPJOosGS-Q\",\"offset\": 37},\"close\": false}"     --header "Content-Type: application/octet-stream"     --data-binary @local_file.txt

it returned: `null`

Now I did finish_batch,

curl -X POST https://api.dropboxapi.com/2/files/upload_session/finish_batch \
>     --header "Authorization: Bearer SC69zip6zi9bwHgqhYHNYQbrpZ5C7G" \
>     --header "Content-Type: application/json" \
>     --data "{\"entries\": [{\"cursor\": {\"session_id\": \"AAAAAAAbQhipmPJOosGS-Q\",\"offset\": 0},\"commit\": {\"path\": \"/Homework/math/Matrices.txt\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}}]}"

It returned to me with async_job_id,

{".tag": "async_job_id", "async_job_id": "dbjid:AACkTPVL8VvaGyrUdEPui0uHg-mzJuroOoZ5yQvP3HB-4JxhpBGNlRrt8XOH6XiOJIpUKlNjUbHtJgUPGWKhQnXm"}

on the batch check,

curl -X POST https://api.dropboxapi.com/2/files/upload_session/finish_batch/check     --header "Authorization: Bearer SC69zip6zi9bwHgqhYHNYQbrpZ5C7G"     --header "Content-Type: application/json"     --data "{\"async_job_id\": \"dbjid:AACkTPVL8VvaGyrUdEPui0uHg-mzJuroOoZ5yQvP3HB-4JxhpBGNlRrt8XOH6XiOJIpUKlNjUbHtJgUPGWKhQnXm\"}

It returned as

{".tag": "complete", "entries": [{".tag": "failure", "failure": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 74}}}]}

My question is, from where I can have the offset value? its nowhere from the very start?

I followed the complete documentation guide, and I am not able to even upload one file with upload session.

Greg-DB

The 'offset' value is "The amount of data that has been uploaded so far. We use this to make sure upload data isn't lost or duplicated in the event of a network error.".

That is to say, it should be the number of bytes that you have already uploaded so far for that particular upload session. You should keep track of this client-side as you send the file data up to Dropbox.

ijunaid8989

Okay thanks now,

on upload_session/start and then doing upload_session/finish

my file is getting uploaded.

the file was of 37 kbs. which I set in offset in upload_session/finish, but

how upload_session/finish_batch can over all the present sessions? when it only takes one session id? and one one file name?

I can now upload the file to my account using only

upload_session/start and upload_session/finish

would that be enough ? for uploading multiple files at once? to dropbox?

I am actually in a loop which being spawned and sending files to dropbox account.

Now I only need to so session start and in the finish add the file size and that it? would that stop sending me too many write operations message?

ijunaid8989

and If I can the append_v2 with close true, it gives me error as

double the offset, it should be, file as 37KBs and it gives an error as offset should be 74

Greg-DB

If you want to commit multiple file uploads at once, and avoid the you should use /2/files/upload_session/finish_batch. That endpoint does not only take a single file name/upload session ID. Its UploadSessionFinishBatchArg.entries parameter takes a list of multiple UploadSessionFinishArg. You can find information on that in the documentation here:

https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-finish_batch

It says "List of (UploadSessionFinishArg, max_items=1000)", indicating that you can supply up to 1000 per call. Each UploadSessionFinishArg should contain the information for a single file and upload session.

When appending data to an upload session, you shouldn't repeatedly send the same data if it's already been successfully sent.

ijunaid8989

When I am going to need this append option anyways?

you mean if the file is bigger, I first send some of its kbs and then some of the others?

In case I have a 2MB file? Can I send it once in the first upload session start request. and then Should I still need to do the append? or not?

ijunaid8989

Also I can send data only through upload session start and finish..

Finish batch never worked for me to upload all data at once.

Greg-DB

Each of the /start, /append, and /finish endpoints, but not /finish_batch, can accept file data.

You would need to use /append if there is more data than you can send in the /start and /finish requests.

Upload sessions work by having you send only a portion of the file's data per request.

Exactly how much data you send per request is up to you. For example, a maximum of 8 MB per request is reasonable, but it can vary by use case. So, if you were using a maximum of 8 MB per request, and the file is only 2 MB, you would only need a single /start request to send all of the file data (and wouldn't need to call /append for that particular file). You could then use /finish without sending any more data to commit that file, or use /finish_batch to commit that file (as well as others).

ijunaid8989

Okay thanks for all your help and replies, right now what I am doing is:

    client = ElixirDropbox.Client.new(System.get_env["DROP_BOX_TOKEN"])
    {:ok, file_size} = get_file_size(image_path)
    %{"session_id" => session_id} = ElixirDropbox.Files.UploadSession.start(client, false, image_path)
    ElixirDropbox.Files.UploadSession.finish(client, session_id, upload_image_path, image_path, file_size) |> handle_upload_response

I still got one error as

"error_summary\": \"too_many_write_operations/\", \"error\": {\"reason\": {\".tag\": \"too_many_write_operations\"

and 5 errors are API timed out.

I have no idea why still there is too many write operations error? where as I am right now first starting the session, then finishing it in a proper way?

but these requests almost goes in parallel, should I need to change something as well? more..

Greg-DB

In this code I see you're using the 'finish' method, presumably once per file, to finish the uploads. If you send off multiple of these at the same time, you will cause lock contention.

You should use a single 'finish_batch' to finish multiple file uploads in a single call to avoid this. (Or, only send the multiple 'finish' calls one at a time.)

ijunaid8989

I uploaded using session start and I got an session_id.

AAAAAAAb0vwx3ypjoH836Q

and then instead of doing append I just finished the batch of this file

curl -X POST https://api.dropboxapi.com/2/files/upload_session/finish_batch \
    --header "Authorization: Bearer -SC69zip6zi9bwHgqhYHNYQbrpZ5C7G" \
    --header "Content-Type: application/json" \
    --data "{\"entries\": [{\"cursor\": {\"session_id\": \"AAAAAAAb0vwx3ypjoH836Q\",\"offset\": 34520},\"commit\": {\"path\": \"/REGULAR/licence\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}}]}"

This gave me results as

{".tag": "async_job_id", "async_job_id": "dbjid:AACg_hehcOljTPYxh6wQhigGQhtc3-eCFneDKK--vzA5V9wBLNi74iNeNqWUQDFOnx1_k-sViaZJpJvXRG0UTlLw"}

and whereas on normal session start and finish I just get the file uploaded moreover when I check batch as

curl -X POST https://api.dropboxapi.com/2/files/upload_session/finish_batch/check \
>     --header "Authorization: Bearer -SC69zip6zi9bwHgqhYHNYQbrpZ5C7G" \
>     --header "Content-Type: application/json" \
>     --data "{\"async_job_id\": \"dbjid:AACg_hehcOljTPYxh6wQhigGQhtc3-eCFneDKK--vzA5V9wBLNi74iNeNqWUQDFOnx1_k-sViaZJpJvXRG0UTlLw\"}"

it returns as

{".tag": "complete", "entries": [{".tag": "failure", "failure": {".tag": "lookup_failed", "lookup_failed": {".tag": "not_closed"}}}]}

So now where I am wrong? the same file is getting uploaded through session start and finish but giving the error in this way?

Greg-DB

@ijunaid8989 All of the upload sessions that you wish to finish using /finish_batch need to be "closed" before you call /finish_batch. That message is indicating that the upload session was not closed.

You should close the upload session with the last call you make to upload the final data for that upload session, whether that's on /start or /append, using the "close" parameter.

ijunaid8989

Thanks for all your replies and I have fixed this issue, my only question is right now, what could be the reason for timeouts from dropbox api?

I have timeout errors a few, while upload session start.

Greg-DB

Can you share the code and error output you're getting for these timeouts so we can take a look? Thanks!