Child Item

Error 429 Rate limit - Too many requests

Hi,

We are dealing with an application that's job is to retrieve around 10-20 images from dropbox, and pass their shared links (2 per image, one for the image and one for it's thumbnail) to a Canva based UI application. This UI application then places the retrieved images in respective positions on the Canva design.

The code to retrieve those URLs is below

const downloadUrls = await Promise.all(rankedImages.slice(startIndex,endIndex).map(async (rankedImage) => {

const imgPath = `${dbxFolder}/` + rankedImage.image;

const thumbnailPath = `${dbxFolder}/` + rankedImage.thumbnail;

const sharedLinkImg = await get_dropbox_shared_link(dbx, imgPath);

const sharedLinkThumbnail = await get_dropbox_shared_link(dbx, thumbnailPath);

console.log("Retrieved shared links ...", imgPath);

return [sharedLinkImg, sharedLinkThumbnail];

}));`

As you can see, we're operating with a small slice from startIndex to endIndex which is generally 10+ images. I'm not sure why we would hit a rate limit with such small number of API calls in general. How do we find out what are the limits. I also don't see any possible APIs to deal with the shared links in a batch manner.

Please advice,

Thanks in advance,

foranuj

Find more posts tagged with

Developers

Comments

Здравко

@foranuj wrote:
...
As you can see, we're operating with a small slice from startIndex to endIndex which is generally 10+ images. I'm not sure why we would hit a rate limit with such small number of API calls in general. ...

Hi @foranuj, Your issue is not related to how many are the API calls, but how many of them are executed at the same time. When particular namespace got accessed, all calls there are strictly serialized! It's tolerated few calls get in waiting to be serialized, but when those calls become too much, then such rate limiting happens. Try serialize your calls client side and I hope such rate limiting would decrease. You should always be ready to handle such rate limit though! It can happen, cosa some other application try access to the same namespace at the same time, by any chance. Just a recall: currently you're calling all 10+ calls at the same time - something that force server side conflict between calls! When such limit happens just respect the noted delay in response header, if there is such, or organize your own exponential backoff and perform the call anew.

Hope this helps.

Greg-DB

@foranuj The Dropbox API does have a general rate limiting system that applies to all account types, but we don't have specific rate numbers documented for that, and we cannot increase the limits for any particular app, user, or team. Apps should be written to handle these rate limit responses automatically. Also as Здравко said, note that not all responses with a 429 status code indicate explicit rate limiting, and may instead be a matter of namespace lock contention, but in any case that you get a response with 429 status code the best practice is to retry the request, respecting the Retry-After header if given in the response, or using an exponential back-off, if not. I recommend referring to the error documentation and Error Handling Guide for more information.

Also, I see you have two calls to your get_dropbox_shared_link method, differing only in the path using rankedImage.image versus rankedImage.thumbnail. Make sure those are different path values; if they happen to be identical, the rate limiting system may rate limit the duplicate calls, in which case you should eliminate the duplicate calls.

Additionally, you didn't show what your get_dropbox_shared_link method does exactly, but if it's calling /2/sharing/list_shared_links, try setting direct_only: true, if that's sufficient for your use case, as that can be more efficient. Refer to the linked documentation for more information on that parameter.

foranuj

@Greg-DB wrote:

I've got an image and it's thumbnail both stored at those paths and I need to pass both those download/shared links. So I do get request different links. I can add a small wait/sleep in that request and hopefully that'll take care of the issue.

Thanks for both your inputs;

export async function get_dropbox_shared_link(dbx, dbx_path){

try {

// Try to create a shared link

const sharedLink = await dbx.sharingCreateSharedLinkWithSettings({

path: dbx_path

});

return sharedLink.result.url;

} catch (error) {

// Convert error to string if it's not already

const errorString = error.toString();

if (errorString.includes('409')) {

// Shared link already exists, get the existing link

const links = await dbx.sharingListSharedLinks({ path: dbx_path });

if (links.result.links.length > 0) {

return links.result.links[0].url;

} else {

return "";

}

else{

console.log("We have an error, ", errorString);

return "";

}

Здравко

@foranuj wrote:
... So I do get request different links. I can add a small wait/sleep in that request and hopefully that'll take care of the issue.
...

Hi again @foranuj,

No, adding wait, just so, wouldn't do anything. You have to add back-off only when needed, i.e. when you got rate limit (as discussed above), otherwise - no. Don't confuse by Greg's advise to avoid double request for the same link. While they are different, as you said - no such problem.

@foranuj wrote:
...
const links = await dbx.sharingListSharedLinks({ path: dbx_path });
...

It's a good idea to avoid requesting all possible shared link if you are interested in direct links only, as Greg advised you. Even more, in such a case your code can get in confusion (i.e. it's potentially buggy). Let's say you have a link to your containing folder and are trying to create link to a new file inside, but unsuccessfully for some reason. In spite unlikely, you can get as result of your code that folder' link instead of image link. Again it's unlikely, but not impossible! To avoid such situation for sure and to speed up a bit, follow the Greg's advise.

Probably my description was not clear enough. Your main issue is that you're bombarding Dropbox server with your requests, starting with all pairs of link at the same time. Normally, the result, in such a situation, is such as you saw it. To avoid this, avoid the simultaneous starting and iterate the images range instead, for instance. 😉

Hope it's a bit more clear now.

Add: You can optimize your code a bit. It's likely when shared link already exists on error indicating link existence the same already existing link to becomes passed together with the error. In your code you completely ignore this! If the link you're interested in is available already, you may skip requesting the same link. This has potential to speed up significantly your code. Simply don't perform unneeded request whenever possible (again - very often it's possible - just a simple check before the request).

foranuj

Здравко

I think I now get the idea and the issues with the above code pieces. I have a different image ranking and generating process that actually puts those images on dropbox in the first place and constructs a json file indicating a few other things. I will create the shared link there, once and then place that link in the json itself. That'll simplify my code significantly and I won't have to retrieve anything other than the actual json file from dropbox.

Thanks once again for taking the time to explain the underlying issues with this line of operation.

Best always.