Child Item

If the cursor is too long, list_folder_continue api always encounter 504 timeout

Hi,

We've designed an app and found that if we create more than 5000 team folders under team root (eg. 1st level folder). The cursor returned by list folder api (eg. https://api.dropboxapi.com/2/files/list_folder) will be around 90000 bytes long.

If we use this cursor to send list folder continue api (eg. https://api.dropboxapi.com/2/files/list_folder/continue), then it will always be responded with 504 gateway timeout.

We've found that the cursor may grow with the number of team folders (eg. see https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/Cursor-max-size/td-p/253505). But we didn't expect that there should always be timeout if there are more than 5000 team folders. (eg. We know that there a soft limit related to this, see https://help.dropbox.com/zh-tw/teams-admins/admin/member-space-limits. But more than 20 seconds still takes too long). Is there any way to list folders recursively faster without timeout?

Thank you very much.

Best regards,

Ethan Fang

Find more posts tagged with

API

Abuse

Comments

Greg-DB

Thanks for the report. It looks this error may be related to the large number of files and/or a large amount of file activity in the affected account. That can vary over time due to account activity and other such factors though. We'll look into it.

You may be able to workaround this issue by setting a smaller limit and/or setting recursive:false on /2/files/list_folder (though that may require updating your implementation accordingly), if you aren't doing so already.

ethanfang

Hi,
We've attempted to set the limit to a low number (eg. 20), it still leads to timeout after 1m30s.
However, we really need to set recursive as true. Or there will be lots of api calls. (eg. 5000 folders, so at least 5000 calls)

Thank you very much~!
Best regards,

Ethan Fang

Здравко

Hi @ethanfang,

The problem in your case is that you have lots of namespaces. Every single namespace gets locked by self. Since you need to lock all namespaces together, looks like the task takes too long. You need to split list processing for different namespaces to avoid such a deadlock. On other side, as you mentioned too many calls take too long to establish connections (Dropbox SDKs doesn't support multi-call on single connection). As a workaround you can perform such a request serie on you own on a single connection and so speed up the processing. I just created a simple example in Python:

#!/usr/bin/python3

from http.client import HTTPSConnection, _CS_IDLE
from urllib.parse import urlparse, urlunparse
from json import JSONDecoder, JSONEncoder

token = '<your access token here>'
paths = ['', '/path1', 'id:..', ...] # id's or paths to all your namespaces

def requestListResponse(conn, path):
  body = JSONEncoder().encode({"path": path,
                               "include_mounted_folders": False,
                               "recursive": True}).encode()
  headers = {'Host': conn.host,
             'Content-Length': len(body),
             'Connection': 'Keep-Alive',
             'Authorization': 'Bearer ' + token,
             'Content-Type': 'application/json'}
  conn._HTTPConnection__state = _CS_IDLE
  conn.request("POST", "/2/files/list_folder", body, headers)
  return conn.response_class(conn.sock, method=conn._method)

def requestListContResponse(conn, cursor):
  body = '{"cursor":"' f"{cursor}" '"}'
  headers = {'Host': conn.host,
             'Content-Length': len(body),
             'Connection': 'Keep-Alive',
             'Authorization': 'Bearer ' + token,
             'Content-Type': 'application/json'}
  conn._HTTPConnection__state = _CS_IDLE
  conn.request("POST", "/2/files/list_folder/continue", body, headers)
  return conn.response_class(conn.sock, method=conn._method)

def handleResponse(resp):
  resp.begin()
  if 200 != resp.status:
    raise RuntimeError(f"Status: {resp.status}\nHeader info:\n{resp.info()}\n"
                       f"\n{resp.read().decode()}")
  return JSONDecoder().decode(resp.read().decode())

def listFolders(paths):
  conn = HTTPSConnection("api.dropboxapi.com")
  responses = []
  for path in paths:
    responses.append(requestListResponse(conn, path))
    while len(responses) > 4:
      resp = responses.pop(0)
      result = handleResponse(resp)
      for entry in result['entries']:
        yield entry
      if result['has_more']:
        responses.append(requestListContResponse(conn, result['cursor']))
  while len(responses):
    resp = responses.pop(0)
    result = handleResponse(resp)
    for entry in result['entries']:
      yield entry
    if result['has_more']:
      responses.append(requestListContResponse(conn, result['cursor']))

for e in listFolders(paths):
  print(f"{e['.tag']}: {e['name']} with path -> {e['path_lower']}")

Hope the above demonstrates the idea clearly.

ethanfang

Hi，Greg

We've tried to create 6000 folders under a team folder under team space root (eg./team_folder1/folder1,......,/team_folder1/folder6000). In this case, we can use list_folder and list_folder_continue api to list until the end within seconds.

But if all the 6000 folders are created under team space root, then it will takes long and 504 timeout. I guess the problem is not totally with large amount of files, but with 1st level team space folder.

We've attempted adding "limit" parameter, but it still timeout, and we can't set recursive to false, since we want to list all the folders with 1 api. If we do so, then 6000 folders would require 6000 api.

Is there any other way to solve the large number team folders performance issue?

Thank you,

Best regards,

Ethan Fang

Greg-DB

@ethanfang Thanks for following up. No, unfortunately I don't have any other workarounds to suggest for this.