Child Item

Downloading large amount of files to my web application

Hi,

I'm currently using the Dropbox download endpoint in a Vue framework, but I'm experiencing slow loading times for photos due to the large volume I'm downloading. Is there an alternative endpoint, such as batchDownload, that I can use to improve the speed? Or should I consider changing my approach to using the download endpoint?

Thank you for your assistance.

Find more posts tagged with

Developers

Comments

Greg-DB

If the files are in one or a few folders, you can use filesDownloadZip to download the entire folder(s) and unzip them locally, instead of downloading each file individually.

Otherwise, you may want to try running several filesDownload calls at a time in parallel, if you aren't already.

Здравко

@daynightdev, in addition to Greg's proposal, you may considered usage of links (either permanent or temporary) and let client's browser to manage downloads itself. 😉

Good luck.

daynightdev

I apologize for not providing the purpose behind my file downloads. I download these files to display them on my web application. I attempted to use 'GetTemporaryLink,' but I encountered an error: '429 Too Many Requests.' Can you provide insights on how to resolve this '429 Too Many Requests' issue? Should I consider debouncing or delaying my requests?

Здравко

Yes, in such a case you may considered exponential back off If no header denoting proposed delay. Unfortunately, there is not 'bunch' version to get links at once.

Greg-DB

@daynightdev Здравко is correct; when you get a 429, the best practice is to retry the request, respecting the Retry-After header if given in the response, or using an exponential back-off, if not. I recommend referring to the error documentation and Error Handling Guide for more information.

The Dropbox API does not offer a batch version of that method, but I'll pass this along as a feature request. I can't promise if or when that might be implemented though.

Здравко

@daynightdev and all other interested,

There are many undocumented Dropbox calls. Some of them may be useful in different cases (like the current one). One workaround, I know about, is usage of a call that deals with shared links (unfortunately that is inapplicable to temporary link). To use it, reorganization of files that need to be accessed may be needed - all of them have to be accessible in one folder and a link to this folder should exist or will be created. The workaround implies the Dropbox class update (named here 'DropboxLinkList' and implemented in Python) as follows:

#!/bin/python3
###############################################################################
# Addition of undocumented link listing functionality to regular Dropbox client
# -----------------------------------------------------------------------------
# file: dropbox_client_link_list.py
# Author: Здравко
#   www.dropboxforum.com/t5/user/viewprofilepage/user-id/422790
###############################################################################

from dropbox import Dropbox
from dropbox.exceptions import BadInputError, HttpError, ApiError
from urllib.parse import urlparse, parse_qs
import json

class DropboxLinkList(Dropbox):
  """
  Use this class to make requests to the Dropbox API using a user's access
  token. Methods of this class are meant to act on the corresponding user's
  Dropbox. In addition :meth:`sharing_list_shared_link_folder` may be called.
  """
  def sharing_list_shared_link_folder(self, path_link):
    """
    Returns list of links (and other data) pointing to all direct entries
    (files/folders) residing in the pointed folder. When ``path_link`` refers
    to the target using path in the current context existing link is used or
    new one is created if there is no existing.
    
    Route attributes:
      No specific scopes are needed for this method usage,
      but :meth:`sharing_create_shared_link_with_settings` will be needed when
      path is in use to refer target. (i.e. sharing.write) and scope of
      :meth:`sharing_list_shared_links` may be needed too (i.e. sharing.read).
    
    :param str path_link: Either path pointing folder (in current context)
      containing all entries that need to be listed or Dropbox shared link to
      folder which content needs to be listed.
    
    The type of return result is not documented by Dropbox, but usable links
      are referred in '.shared_link_infos[].url' of the response.
    """
    if not path_link.startswith("https://"):
      try:
        metadata = self.sharing_create_shared_link_with_settings(path_link)
      except ApiError as e:
        er = e.error
        if not er.is_shared_link_already_exists():
          raise
        er = er.get_shared_link_already_exists()
        if er != None and er.is_metadata():
          metadata = er.get_metadata()
        else:
          metadata = sharing_list_shared_links(path_link, direct_only=True
                                               ).links[0]
        path_link = metadata.url
    if path_link.startswith("https://www.dropbox.com/sh/"):
      link_type = 's'
    elif path_link.startswith("https://www.dropbox.com/scl/fo/"):
      link_type = 'c'
    else:
      raise BadInputError('', f"Unsupported link type: {path_link}")
    url = urlparse(path_link)
    path = url.path.split('/')
    body = {'t': 'T', 'link_type': link_type}
    if link_type == 's':
      body['link_key'] = path[2]
      body['secure_hash'] = path[3]
      body['sub_path'] = "/".join(path[4:])
    else:
      body['link_key'] = path[3]
      body['secure_hash'] = path[4]
      body['sub_path'] = "/".join(path[5:])
      rlkey = parse_qs(url.query)
      if 'rlkey' not in rlkey:
        raise BadInputError('',
          f"Expected 'rlkey' query param, but is missing: {path_link}")
      body['rlkey'] = rlkey['rlkey'][0]
    url = 'https://www.dropbox.com/list_shared_link_folder_entries'
    headers = {'Content-Type': 'application/x-www-form-urlencoded',
               'Cookie': 't=T'}
    body = "&".join(f"{k}={v}" for (k,v) in body.items())
    r = self._session.post(url, headers=headers, data=body, stream=False,
                           verify=True, timeout=self._timeout)
    if r.status_code != 200:
      raise HttpError('', r.status_code, r.content.decode())
    assert r.headers.get('content-type') == 'application/json', (
                'Expected content-type to be application/json, got %r' %
                r.headers.get('content-type'))
    return json.loads(r.content.decode())

As mentioned within the code, existing link of a folder may be listed or directly referring to the folder keeping needed files (with path, id, etc.). Let's say somebody keeps lot of files in a folder with path '/ForListing' that need to be pointed with a link (every one). Instead of looping through all of them and get links one by one (generating lot of API calls), the mentioned folder may be enumerated directly with just few calls (2 or 3 at most):

import .dropbox_client_link_list

# Proper init here - the same like original Dropbox class
dbx = dropbox_client_link_list.DropboxLinkList(...)
container_folder = '/ForListing'

...

for lm in dbx.sharing_list_shared_link_folder(container_folder)['shared_link_infos']:
    print(lm['url'])

...

The above will list links for all entries (files/folders) inside 'ForListing' folder. They will not be generated - they exist (pre-generated) while files reside within pointed folder. If needed files are hierarchically spread in directory subtree of the pointed folder (not just flat in a single folder), recursive enumeration may be needed, as follows:

...

def linkList(path_link, prefix=''):
    entries = dbx.sharing_list_shared_link_folder(path_link)
    for lm, tm in itertools.zip_longest(entries['shared_link_infos'], entries['entries']):
        url = lm['url']
        is_dir = tm['is_dir']
        print(f"{'D: ' if is_dir else 'F: '}{prefix}{url}")
        if is_dir:
            linkList(url, prefix + "  ")

linkList(container_folder)

Here one more call for each subfolder will be needed. The chance for API call rate overrun to be meet here is negligible. The above code-snippets are for illustration only. The idea is simple enough, so easy enough to be implemented/'translated' into any other programming language. 😉

Good luck.

Greg-DB

Please note that undocumented endpoints are not meant for third party use, and are subject to change without notice.