Sending cursor parameter to delta to get complete listing of files

J C.17

I'm trying to get a complete listing of files from a dropbox folder using the /delta endpoint. The folder contains around 3400 files, and it seems a call to /delta returns 2000 items, with has_more set to true. I'm assuming then, that I would need to make 1 more call to get the remaining 1400 files.

However, no matter how many times I call /delta using the returned cursor, it always returns 2000 entries with a has_more set to true.

I'm assuming this means I'm using the cursor incorrectly. Can someone point me to some example code for this?

I'm using the HTTP endpoints. And making calls using jquery ajax. It looks something like this:

has_more = true;
deltacursor = "";
while (has_more) {
  var settings = {
    "async": true,
    "crossDomain": true,
    "url": deltaurl,
    "method": "POST",
    "headers": headers,
    "data": {
      "path_prefix": "/acip",
      "cursor": deltacursor
    },
    "dataType": "json"
  };
  $.ajax(settings).done(function (data) {
    deltacursor = data.cursor;
    has_more = data.has_more;
  }).fail(function(error) {
    console.log("In error");
  });
}

 

Steve M.

I think the issue you're having relates generally to async programming and understanding the order in which things happen.

Consider the following code:

console.log('About to make the request.');
$.ajax(...).done(function () {
  console.log('Request completed.');
});
console.log('After making the request.');

If you ran this code, what would you expect to see in the console?

You should see the messages in this order:

About to make the request.
After making the request.
Request completed.

This is because the network request is asynchronous. Immediately after starting the request, the code continues executing on the next line. Only after the request completes does the callback get invoked. So now imagine wrapping the request in a while loop, as in your code:

while (true) {
  console.log('Making a request...');
  $.ajax(...).done(function () {
    console.log('Request completed.');
  });
}

You would see the following output:

Making a request...
Making a request...
Making a request...
Making a request...
Making a request...
Making a request...
Making a request...
Making a request...
...

This is because as soon as the first request is made, your code continues executing on the next line, which means you reenter the loop and make another request, etc. The issue with your current code is that this means you're making each request to the delta endpoint with the original (empty) cursor, since the requests are getting made before the first call even finishes.

Rather than writing a while loop, you need to come up with a way to make the next request only after the first request has completed. The code that runs after the first request completes is in the "done" handler. A naive approach would be as follows:

$.ajax(...).done(function (data) {
  console.log('First request completed.');
  if (data.has_more) {
    console.log('Starting second request.');
    $.ajax(...).done(function (data) {
      console.log('Second request completed.');
      if (data.has_more) { ... }
    });
  }
});

But as you can see, this approach doesn't really work. You do need something like a loop, but a traditional "while" loop won't help in this world of asynchronous calls and callbacks.

Try instead creating a reusable function that gets called once to make the first call and then gets called every time there's more data:

function fetchDeltaEntries(cursor) {
  cursor ||= '';

  console.log('In fetchDeltaEntries with this cursor: ' + cursor);

  $.ajax({
    url: deltaurl,
    method: 'POST',
    headers: headers,
    data: {
      path_prefix: '/acip',
      cursor: cursor,
    },
  }).done(function (data) {
    // In reality, do here whatever you want to do with the entries.
    console.log('Got ' + data.entries.length + ' entries.');

    if (data.has_more) {
      // Fetch more entries, using the received cursor.
      fetchDeltaEntries(data.cursor);
    } else {
      console.log('No more entries to retrieve.');
    }
  });
}

// Call once with no cursor to start the process.
fetchDeltaEntries();

I haven't tested this code at all, so sorry if there are typos or other mistakes in it! But you should see output like this:

In fetchDeltaEntries with this cursor:
Got 2000 entries.
In fetchDeltaEntries with this cursor: abc123xyz
Got 1400 entries.
No more entries to retrieve.

This is the order we want to see. We start a call, then we process the results, and then (if there's more to do) we start the next call.