Child Item

How do I get all files and folders on Dropbox Business account

I have asked this question in the past and did get answers but I am still struggling to figure this out.

Basically, I want to export the entire structure of the given Dropbox Business account to a local DB. I want to export each and every folder/file to the DB.

Do I list this namespaces and use the namespace id as path to traverse each folder? Would that cover all the folders that are there? How would I de-duplicate the traversed folders? How do I segregate between Private, Shared and Team folders?

Can you recommend an approach and provide some guidance?

Find more posts tagged with

API

Abuse

Comments

Greg-DB

(For anyone else interested, the earlier thread can be found here.)

Do I list this namespaces and use the namespace id as path to traverse each folder?

Yes, specifically, it sounds like you're referring to doing this by listing all namespaces across the team using /2/team/namespaces/list[/continue], which is a good way to approach this.

Would that cover all the folders that are there?

Yes, that should cover everything.

How would I de-duplicate the traversed folders?

You can use the namespace ID (e.g., "shared folder ID") to de-duplicate entries. These are consistent across users.

How do I segregate between Private, Shared and Team folders?

You can use the NamespaceType value (i.e., under NamespaceMetadata.namespace_type returned by /2/team/namespaces/list[/continue]) to check the type.

rk90

Thanks for the response!

Does the namespaces_list returns entries for all folders or just the top-level folders only. Listing namespace and checking the namespace_type is pretty straight forward, but as we traverse through the folders it becomes difficult to find out type of the sub-folders.

I list folders recursively, in folder listing response, I see several entries that have sharing_info which can be used to get the shared_folder_id, but I have noticed that when I list member folders, the team_folders the member has access to also gets listed and returns sharing_info. So, in that user's context the team_folder is a shared_folder. How do I deal with this situation? I also found that several folders do not return sharing_info what type of folders those are? member's personal folder? Unmounted folders also don't return info sharing_info I guess.

Also when I list namespace NamespaceMetadata.name is 'Root' for all member folders.

Lastly if the Dropbox Business account I am operating on has a team_space, then what approach should I take to list all files and folders. I want to use a consistent approach across Dropbox accounts (wheteher it has team space or not).

I would appreciate any pointers and suggestions.

I would also like to thank you for your help so far. You are a champ!

Greg-DB

Does the namespaces_list returns entries for all folders or just the top-level folders only.

The /2/team/namespaces/list[/continue] endpoints return a list of all of the "namespaces". This will not be a complete list of all folders. Namespaces include team folders, shared folders, team members' home folders, and team members' app folders (but not normal folders insides those folders).

To then list the contents of any particular folder, you should use /2/files/list_folder[/continue], with recursive=true if you want all nested entries as well.

in that user's context the team_folder is a shared_folder. How do I deal with this situation?

Team folders are very similar to shared folders. I'm not sure I understand your question "How do I deal with this situation?" though. Can you elaborate? What specifically is causing you trouble?

several folders do not return sharing_info what type of folders those are?

If a folder does not have sharing_info, it is not itself a shared folder, nor contained in a shared folder. (E.g., it's a private folder in a member's account.)

Unmounted folders also don't return info sharing_info I guess.

Unmounted folders aren't listed by /2/files/list_folder[/continue] anyway.

the Dropbox Business account I am operating on has a team_space, then what approach should I take to list all files and folders.

I recommend reviewing the Namespace Guide and Content Access Guide, if you haven't already, as they should offer some useful reference on this.

rk90

I have reviewed Namespace Guide and Content Access Guide. The namespace guide explains the difference between a team space and non-team space accounts, but is listing namespace will include all namespaces (not recursive but top-level) on both types (Team space and non-team space) of accounts?

I am still not clear on the concept of team and shared folders (I have reviewed online resources). If team folders and shared folders both have a shared_folder_id and are almost similar, then why are they identified differenty. Nevertheless, my main objective is to walk through each and every folder recursively and for that I need a list of top-level folders which I assume can be obtained by listing namespace.

Greg-DB

listing namespace will include all namespaces (not recursive but top-level) on both types (Team space and non-team space) of accounts?

Yes, /2/team/namespaces/list[/continue] will list all namespaces for a team, whether or not the team uses the team space configuration.

If team folders and shared folders both have a shared_folder_id and are almost similar, then why are they identified differenty.

While team folders and shared folders are similar and share some properties, there do not behave exactly the same, so they are identified differently. For instance, team folders are shared with the entire team, while shared folders can be shared with arbitrary/specific accounts.

rk90

This means if I have 5 members on Business account and I create a Team folder, then all those 5 members get access to the Team folder. There is no concept of folder permissions on team folders? What I am failing to understand is all users I create in Dropbox Business are team members and they have access to all team folders, then where does this shared folder comes from.

Greg-DB

Team folders are for sharing content with one's team. You can find more information on team folders in the help center here.

Shared folders can be used to share folders with arbitrary users. You can find more information on shared folders in the help center here.

rk90

Whenever I think I have an idea about how I am going to get what I want, I bump into something that makes me wonder if I should really do this project.

I list namespaces and the namespace.name for team member folder is 'Root'. I list folders recursively and there are thousands of deleted files/folders that don't have 'path_display' attribute and I don't know how would I get the size of those deleted files/folders without knowing thier path or an identifier. Then there are several entries in folder list response that do not have 'path_display' attribute. All I see in those entries are ( {'name': '<item name>', 'parent_shared_folder_id': '<id>, 'id': 'id:<id>'} ). I don't know how am I supposed to construct the absolute path of such entries unless Dropbox expects us to make another request to another endpoint to get the absolute path of such entries.

I have seen poorly designed APIs, but Dropbox APIs are worst. I wonder if the developers knew what they were doing. Dropbox chose not to give an endpoint that can be queried to get the unique listing of all files/folders rather they chose to give something like impersonation where you impersonate each user or an admin and list the same folders and do the sorting yourself. No API to get the size of folder (list folders recursively and get calculate the size). The list goes on.

You guys forward feature requests but those get moved to the trash bin because Dropbox does not want to make things easy.

Greg-DB

Thanks for the feedback! We appreciate it. I'm sharing this with the team.

Note that if a Metadata object doesn't contain `path_lower` or `path_display`, it indicates that "the file or folder is not mounted". I.e., it's in a shared folder that isn't currently added to the account, so it doesn't have a path in the account.

And it sounds like specifically, you'd like the following, so I've enumerated the following feature requests for the team:

The ability to easily get the total size for a folder's contents
The ability to get the size for a deleted file

rk90

This is not true, there are several folders that are inside team folders and show up within the team folder on the UI, but when I list those folders via API I don't see `path_display`. Moreover, there are several deleted files/folders in the response that just have `name` but no `path_display` or `path_lower`.

I am frustrated because of the disambiguity and unnecessary optional fields in the response. Most importantly if someone was given a task to generate the list of all files/folders on a given Dropbox account, it will take them forever because Dropbox doesn't have an endpoint that can be called to list each and every folder. What Dropbox expects us to do is call `list_folder/{continue}`. Consider 1000 members that have almost identical folders (i.e. a large shared folder shared with all of them). When I list each member's folder recursively I will be unneccesarily listing that same shared folder 1000 times. If I want to skip listing that folder again, then I need to set `recursive=False` and check each folder in response to see if it was listed previously, unnecssary processing. APIs are suppossed to make life easier not difficult.

Greg-DB

there are several folders that are inside team folders and show up within the team folder on the UI, but when I list those folders via API I don't see `path_display`

Do you mean you're calling for a namespace directly as an admin without a root like this?

curl -X POST https://api.dropboxapi.com/2/files/list_folder \
    --header "Authorization: Bearer <ACCESS_TOKEN>" \
    --header "Dropbox-API-Select-Admin: <ADMIN_USER_ID>" \
    --header "Content-Type: application/json" \
    --data "{\"path\": \"ns:1990893360\"}"

# {
#   "entries": [
#     {
#       ".tag": "folder",
#       "name": "test",
#       "parent_shared_folder_id": "1990893360",
#       "id": "id:hro-cp_zmJAAAAAAAAAACg",
#       "sharing_info": {
#         "read_only": false,
#         "parent_shared_folder_id": "1990893360",
#         "traverse_only": false,
#         "no_access": false
#       }
#     },
#     {
#       ".tag": "folder",
#       "name": "test folder in team folder",
#       "parent_shared_folder_id": "1990893360",
#       "id": "id:hro-cp_zmJAAAAAAAAAADw",
#       "shared_folder_id": "3980274560",
#       "sharing_info": {
#         "read_only": false,
#         "parent_shared_folder_id": "1990893360",
#         "shared_folder_id": "3980274560",
#         "traverse_only": false,
#         "no_access": false
#       }
#     }
#   ],
#   "cursor": "...",
#   "has_more": false
# }

The paths aren't returned because this call isn't rooted anywhere; i.e., there's no root for the paths to be mounted relative to.

Instead, you can supply the path root using the 'Dropbox-API-Path-Root' header (from the Namespace Guide), like this:

curl -X POST https://api.dropboxapi.com/2/files/list_folder \
    --header "Authorization: Bearer <ACCESS_TOKEN>" \
    --header "Dropbox-API-Select-Admin: <ADMIN_USER_ID>" \
    --header 'Dropbox-API-Path-Root: {".tag": "root", "root": "1990815600"}' \
    --header "Content-Type: application/json" \
    --data "{\"path\": \"ns:1990893360\"}"

# {
#   "entries": [
#     {
#       ".tag": "folder",
#       "name": "test",
#       "path_lower": "/team folder 1/test",
#       "path_display": "/team folder 1/test",
#       "parent_shared_folder_id": "1990893360",
#       "id": "id:hro-cp_zmJAAAAAAAAAACg",
#       "sharing_info": {
#         "read_only": false,
#         "parent_shared_folder_id": "1990893360",
#         "traverse_only": false,
#         "no_access": false
#       }
#     },
#     {
#       ".tag": "folder",
#       "name": "test folder in team folder",
#       "path_lower": "/team folder 1/test folder in team folder",
#       "path_display": "/team folder 1/test folder in team folder",
#       "parent_shared_folder_id": "1990893360",
#       "id": "id:hro-cp_zmJAAAAAAAAAADw",
#       "shared_folder_id": "3980274560",
#       "sharing_info": {
#         "read_only": false,
#         "parent_shared_folder_id": "1990893360",
#         "shared_folder_id": "3980274560",
#         "traverse_only": false,
#         "no_access": false
#       }
#     }
#   ],
#   "cursor": "...",
#   "has_more": false
# }

rk90

Yes, I am not supplying the 'with root path' header. What about the deleted files/folders? There are several that don't return the path_display.

Greg-DB

Can you share a bit of sample code/output for that case so I can take a look? Thanks in advance!

rk90

for namespace_list in session.post('2/team/namespaces/list'): 
    for namespace in namespace_list: 
        for listing in session.post('2/files/list_folder', headers={'Dropbox-API-Select-Admin': <team_member_id>}, json={'path': f'ns:{namespace[namespace_id]}', 'recursive': True, 'include_deleted': True}):
            for item in listing:
                print(item)

Above is an example of the approach I am taking to list all folders. Part of the problem is if a folder is shared with all members, then when I list 'team_member_folders' I am listing the same folder multiple times.

Greg-DB

Thanks! The omission of the path on the deleted entries in this case would be for the same reason as mentioned in my previous post yesterday; i.e., without the path root there's no root for the paths to be mounted relative to.

For shared folders, the shared folder ID will be consistent across users, so you can keep a list of the IDs you've seen already, and skip them when you come across them again.