Dropbox Python API and Google Cloud Composer

ohienmhen

I am attempting to orchestrate downloads using the Dropbox Python API and Google Cloud's Cloud Composer. I am trying to download the files off my dropbox account, unzip them and them upload them to my Bigquery warehouse for analysis. Does anyone have a more efficient way of accomplishing this? I am currently stuck at downloading my files off Dropbox using the API. It seems as if my calls are succesful however the files do not appear in my local drive. I have tried every conceivable combination for the download path including using cloud storage url's and local file paths. A sample of the code to download the file is as follows:

 

def download_files_in_folder():

            """Download a file from Dropbox to the local machine."""

            dbx = connect_to_dropbox()

            files = dbx.files_list_folder(dbxPath)

         

            for entry in files.entries:

                print(entry.path_lower)

                dbx.files_download_to_file(r"Users\damian.ohienmhen\\"+entry.name,entry.path_lower)

                print('downloaded file')

 

Can anyone help me?

 

Addendum: I attempted to run the code using only the Google Cloud Command line and ran it as a standalone python script. The following code worked below:

 

for entry in files.entries:

                print(entry.path_lower)

                dbx.files_download_to_file(r"/home/dohmhen/"+entry.name+entry.name,entry.path_lower)

                print('downloaded file')

Is there a way to specify a file download path while running it in the cloud composer environment? This same script fails in the environment with the following message:

 

FileNotFoundError: [Errno 2] No such file or directory: '/home/dohmhen/2022-06-29_7-20-04 pm_hk_1656544804.zip2022-06-29_7-20-04 pm_hk_1656544804.zip'

Greg-DB

Using the files_download_to_file method is a valid way to download a file from Dropbox to your local filesystem. When calling that, the first parameter 'download_path' should be the local filesystem path where you want to save the downloaded file. Regardless of where you're running this, that first parameter is how you specify where on the local filesystem to save the file. (If you'd prefer to handle the data directly, that is, without having the SDK save it to the local filesystem for you, you can use the methods without "to_file", e.g., files_download, as shown in this example.)

 

Refer to any output/error you are getting when you run this to see if there's an issue. If you're not getting any exception thrown, that should indicate that the download succeeded.

 

In your first sample, it looks like you may not be supplying a correct path value for the path you expect though. For instance, you don't have a leading slash so this would be a relative path, while you perhaps intended for it to be an absolute path for your home folder. It looks like there may be an issue with the escaping slashes as well.

 

In your second sample, it looks like writing to the local filesystem failed, perhaps because "/home/dohmhen" doesn't exist on that machine. (It looks like you're also writing entry.name to the path twice.)

 

In any case, if you want this to save a file to your home folder on the local filesystem, you'd likely be better off with something like this:

 

dbx.files_download_to_file(os.path.join(os.path.expanduser("~"), entry.name), entry.path_lower)

 

 

As for optimizing this, if you want to download an entire folder, you can use the files_download_zip_to_file method (or files_download_zip) instead. That will allow you to download the folder as a zip in one call, instead of making one call per file.