Child Item

listFolder with includeMediaInfo not listing all available files

We have a problem while integrating the v2 iOS API on our photography App.

If we call listFolder for the root folder with includeMediaInfo set to @yes, fewer results are returned (e.g. recently batch uploaded photos are not included in the response).

If we set includeMediaInfo:@NO, all files are listed.

Find more posts tagged with

API

Abuse

Comments

Greg-DB

This secondary issue should also be fixed now.

Greg-DB

Thanks! We're working on it.

BranchSolutions

I've opened a API ticket.

However if I setup the request with

include_media_info = false

the response returns the proper file list with all media info ... strange ...

Greg-DB

@cdemiris99 (or anyone else), please open an API ticket with affected user ID samples so we can track down any remaining issues:

https://www.dropbox.com/developers/contact

Thanks!

cdemiris99

So, it seemed to be working yesterday, but we're seeing problems again. Frankly it now seems like media files are processing fairly quickly, but other files aren't showing up. For now I'm removing the include_media_info flag for our next update. We'll try it again once it seems to be resolved better.

Greg-DB

@BranchSolutions Please open an API ticket with the affected user ID(s) so we can check on this for you:

https://www.dropbox.com/developers/contact

Janne L.

I just tested this and works for me.

BranchSolutions

Sorry, but for me the API isn't working.

I've uploaded a new file 1 hour ago and is still missing from file list (it's the only file present in the folder).

cdemiris99

Working great for me in first tests. Thank you very much!

Greg-DB

This is fixed now. Calls to list_folder/continue will now return the latest entries. Please let us know if you're still seeing any issues.

Greg-DB

The team worked on a fix, but unfortunately it looks like it will take a little longer to get deployed than originally anticipated. The fix should be live tomorrow.

Greg-DB

It looks like we should be able to get this fixed today. I'll follow up here once I have an update.

And thanks for your feedback on the difference between get_metadata and list_folder. I'm sending it along to the team.

Your solution of using get_metadata in addition to list_folder with include_media_info=false would work. You can also keep two different list_folder cursors, one with include_media_info=false and include_media_info=true and track them both if you'd prefer.

cdemiris99

Thanks for responding.

We are using /files/list_folder, so that explains why I'm not seeing them. It hasn't been a real issue in the past as long as files were being processed in a reasonable amount of time (and hopefully once this is fixed it we'll be ok for now).

However, it seems inefficient if I now have to do list_folder without the metadata then have to query individually for each file to get metadata (or merge two separate list_folder calls like another user mentioned), so I really do hope it can be changed in the future to have list_folder simply provide the same list regardless of the include_media_info flag, and simply provide media info if available.

ramonpoca

We've implemented a fix by double-querying listFolder first with no mediaInfo and next with mediaInfo and "merging" the mediaInfo obtained on the second request, if any.

This is obviously not optimal neither for us nor for the API server.

In related news, a JPEG file I uploaded on friday via batch upload still has no metadata. Looks like a problem in the metadata processing backend?

Janne L.

Hi!

I'm having similar issues. For example I have few files in 'Camera Uploads' , oldest pics and videos are ~ 3 days old. Here are some Python examples.

Media info not included:

>>> result = dbx.files_list_folder('/Camera Uploads')

>>> print len(result.entries)

Media info included:

>>> result = dbx.files_list_folder('/Camera Uploads', include_media_info=True)
>>> print len(result.entries)
0

If I target one file at a time:

>>> dbx.files_get_metadata('/Camera Uploads/2017-06-09 17.25.40.jpg', include_media_info=True)

media_info=MediaInfo(u'pending', None)

The media info is still pending after three days? It seems that there some issues with your media metadata system? API v1 was much more faster in this case, almost instant.

De L.1

@Greg-DB thanks for your response and for looking into this issue. I really hope that you can figure out soon what stopped the media information processing!

You also explained nicely why /files/list_folder is responding differently than /files/get_metadata. However, it doesn't really make sense to us. We think that /files/list_folder should always return all files and folders on the Dropbox even if the media information isn't yet processed. The response definitely shouldn't be deferred until everything is known (we actually haven't seen any cloud API doing this). In our humble opinion this isn't a 'Feature request' but just an essential part of this kind of API. Let's give an example:

Lets say that we have a Video App where new videos are uploaded for our clients. Our clients use the App to see and use those files. When we upload new media files, the client is informed that new files are added. However, when the clients App wants to see the new files via a /files/list_folder/continue call with a curser obtained from /files/list_folder/get_latest_cursor with include_media_info = true, it doesn't return any of the newly added files. Now the client has to wait (even "on the scale of hours") till all media information is processed before it can see any of the new files via this call.

We are not sure how we are supposed to solve this situation. The App is able to cope with files having the media information pending perfectly well. We just try to get this information later on for files missing this information, and in the mean time we deal with the dimensions and duration in another way. But the app has to receive this pending tag to know that there is something there. As far as I can see, the only solution we currently have, is to use a /files/list_folder/get_latest_cursor call with include_media_info = false, and do a seperate /files/get_metadata call on each and every media file returned by /files/list_folder/continue to see the status of the media information.

What is your view on this?

Greg-DB

If you're using /files/get_metadata with include_media_info=true, you should get 'pending' media information. I just tried it, and it is working for me. Can you share the HTTP request and response for an API call like that that isn't working for you? Thanks in advance!

Also, the API is designed to show pending media info from /files/get_metadata but not /files/list_folder. The former can be accessed on demand, but the latter is only supposed to return one entry per file, so they are deferred until all of the requested (media) information is available. I'll send this along as a feedback though, to see if we can do this in a different way in the future.

In any case, the delayed files/folders appear to be due to an issue on our side. We're looking into it, and will follow up here one we have an update.

cdemiris99

Second the question about why files aren't returned if they haven't been processed yet. Seems like they should be returned without the additional metadata until the metadata is ready. Files seem to be taking a lot longer to process. Just tried a small 4MB mp4 file, and an hour later it's still not appearing. We're starting to get customer complaints about their files not showing up.

De L.1

I just tried it, but doing a call to /files/get_metadata with include_media_info = true to a file which doesn't have mediametadata yet available doesn't return a mediadata object at all. So pending isn't set in that case.

I am also currious why the /files/get_metadata call and the results of /file/list_folder don't return the same FileMetadata object for the file which doesn't have mediametadata yet. Could you explain why that is the case?

And what about the issue of not showing folders when doing a /files/list_folder call with include_media_info = true?

Greg-DB

For reference, the 'pending' state is used when metadata is directly requested for a file for which the media metadata is not yet available, such as when using /files/get_metadata, but not from /files/list_folder.

If you're seeing significant delays like this when using include_media_info=true, please open an API ticket with the affected user ID(s) so we can look into it:

https://www.dropbox.com/developers/contact

Thanks in advance!

De L.1

I have the exact same issue here with files not showing up after a hour or more. However, I also have this problem with the folders that contain the media files.

If I create a new folder on Dropbox, the folder will immediately show up when I do a /listFolder call with includeMediaInfo on the parent folder. However, when I upload a media file to the new folder, the folder suddenly disappears from the reponse when doing the same API call.

That is a little bit strange, as the folder itself will never have any media information... I hope that this is a bug and not the expected behavior! I would like to see the folder entries at all time when doing a includeMediaInfo call.

Furthermore, it would be nice to list the files via the /listFolder call as well even if the don't have the media information yet. I would expect to have the same kind of FileMetadata response as when doing a /get_metadata on an entry that doesn't have the media info yet. By the way, what does the pending attribute actually do in the MediaInfo object? Isn't that supposed to be set and used in this case?

Individual IT S

just tested again. Files that were placed into the folder yesterday 'server_modified' => string '2017-06-09T16:48:56Z' are still now showing up with "include_media_info" => true but all is fine if setting "include_media_info" => false

Greg-DB

For most users most of the time, it's on the scale of seconds or minutes, but it can be longer sometimes, on the scale of hours.

In some cases, backend issues on our side can also slow this down.

Individual IT S

"variable amount of time"
is that minutes or hours?
I've tested on 2 accounts with basically no activity.

Greg-DB

When using includeMediaInfo, there can be a delay on results being returned while the media information is prepared.

They will get listed once the information is available, but this can take a variable amount of time, especially if there are many files or a lot of activity in the account.

Individual IT S

have a posted a simmilar problem here: https://www.dropboxforum.com/t5/API-support/new-files-not-returned-by-files-list-folder/m-p/224988#M12174

and now reading your post I tried changing my code and it does work. The API does return all new and old files correctly.

I would say that is a problem in the API.