Child Item

Search_v2 API Changes

Hi folks,

I just wanted to give some feedback regarding the new search_v2 API, which came as a bit of surprise to me.

Firstly it seems that the creation of this new API was not announced, at least not anywhere that I can find. I'm a registered Dropbox developer, but I don't seem to have received any notification. I did not receive any e-mail as a registered developer, it is not in the blog, and there's no mention in a quick search of these forums (not that I was a member before now). The application I developer provides Dropbox as a side-feature and I do not actively use it myself so I have not been notified as a developer of the app because I have not called the old API recently.

Secondly I question why these changes have not been implemented as a v3 API, rather than creating a breaking change to the existing API. Versioned APIs should be stable within the version.

Thirdly I question the naming of the new API - `files/search_v2` and `files/search/continue_v2` - a break from the existing naming scheme of simply appending `/continue` to receive additional pages of results. This means that the API is no longer consistent within itself (it previously was). You have to know which endpoint is a v2 endpoint, and to treat it differently.

Finally, in October it was announced that the old `/files/search` API was going to be removed and we were given less than 6 months to replace it with the new API. That's great... if you have a release cadence which fits into that timeframe, but not all projects do - let alone the latest version being installed in time.

I would urge you to reconsider removal of the 'old' v1 search API, especially in such a short timeframe, but really at all. Don't break APIs!

Find more posts tagged with

API

Abuse

Accepted answers

Greg-DB

Good news, the team has finished reviewing this and has decided to extend this, bringing the retirement date for /files/search to October 28. Hope this helps!

All comments

Greg-DB

Thanks for sharing this detailed feedback! We appreciate you taking the time to write this up. I'll send this along to the team.

andrewnicols

Greg.

Thank you for your reply.

Are you able to provide any further information on other planned changes to the API? I notice that there are other v2 endpoint options in the documentation, but these are also inconsistent - most use the same continue endpoint as the v1 API for example. I also fail to notice any information about whether more changes are planned, whether any further removals are planned, or any roadmap for the API as a whole.

Is there any timeframe for when the relevant team may respond to these concerns?

I will also note that this is not the first API change made to the v2 API which has broken applications.

Not every application which uses Dropbox is a SaaS solution fully under control of the original developer. Self-hosted services using Dropbox are put in a very difficult position with these changes. We cannot simply roll out a fix overnight. Some of our users take months to plan upgrades and can only do so at certain times of year.

Andrew

Greg-DB

Thanks for following up. I don't have any other news or planned changes to share. Dropbox will announce any other changes if/when they are planned. We don't have a public roadmap for the API, but I'll pass this along as a feature request as well. I can't promise if or when that might be implemented though.

Also, I checked on this change with the team on this, and we did send an email regarding the search endpoint migration, but it sounds like that unfortunately did not make it to you, perhaps either because it got filtered or because you weren't using the endpoint, like you mentioned. In any case, I recommend making sure the email address on the account that owns your API app(s) is valid and can receive emails from Dropbox. And, make sure you haven't disabled the "API announcements" category on https://www.dropbox.com/account/notifications .

In addition to email notifications, you can find news about the API on the blog here: https://dropbox.tech/developers . There's also an RSS feed for that here: https://dropbox.tech/developers/feed . For instance, here's a blog post about the new search functionality.

Anyway, Dropbox is now offering new search functionality, so the team chose to introduce a new version of the search endpoint on API v2 (/2/files/search_v2) to replace the older one on API v2 (/2/files/search). Dropbox generally versions specific endpoints like this when introducing changes to specific functionality, instead of versioning the entire API (e.g., from Dropbox API v2 to a hypothetical Dropbox API v3). I understand this is not your preferred pattern for changes like this so I've raised this with the team, but right now there aren't any plans to change the versioning scheme used on the Dropbox API. I've likewise sent your feedback along regarding the "continue" endpoint format.

Finally, I've also shared this request to reconsider the retirement of /2/files/search, but right now it is still planned to be retired on February 28th, 2021. We'll follow up if that changes. If you are unable to migrate your app to use the new version by then, feel free to open an API ticket to see if we can offer some solution.

andrewnicols

Hi Greg,

Thank you for the much more detailed and informative response.

The issue that I, and I'm sure others, face is that we are the developers of software which users typically self-host. As a result we add features for our users to make use of, but we do not necessarily use them ourselves. If we don't use them, then we don't use the endpoint and it seems that we do not get notified. Nor do we have access to their system.

Furthermore, because our users self-host, while we will patch this in time (I'm testing the patch now), our next point release won't be until the 8th March 2021, which is after the retirement date. Because of the nature of our software (Online Learning), many institutions will not actually install the update until their next school break. So while our next point release is the 8th March, many of our users will not be in a position to install it until April at the earliest. Even if were to release a version sooner, many of our users cannot upgrade during their school term time, especially with the critical role of online learning at the present time.

Sadly, because of self-hosted nature of the software, we're unable to request an exemption like you've suggested for others because we do not control their application access.

I will also note that this is not the first instance of this affecting us. We also have an issue whereby when we wrote our integration Dropbox did not require any scopes in the authentication request (in fact your documentation still suggests that this is acceptable and describes the `scope` with type `String?` and requesting all if not specified). However this changed in September and now new API keys require that scopes be specified. There was no warning beyond "but may be turned off soon". Note: the links at the bottom of the blog post are just underlined text.

I do appreciate that you've passed this on to the relevant team. I'm aware that more and more applications are only offered through SaaS or managed offerings, but many projects are not, and this will almost certainly affect the Open Source community more than others.

Regarding versioning, I think that introducing a new API endpoint is not a particular problem, but that retiring endpoints most certainly is. Breaking changes within the lifetime of a major version is generally not well received and has a tendency to cause problems - especially given the short timeframes involved in your recent changes.

I would really urge you to reconsider dropping support for the v2 search v1 API on the 28th February - at least defer it until October 15th when the announcement will have been 12 months old and you will have given non-SaaS solutions and their users a chance to manage this change properly.

Thank you and best wishes,

Andrew

Greg-DB

Thanks for the additional information and feedback.

Also, when you say you "do not control their application access", do you mean that they each register their own Dropbox API app and that you do not know the relevant app key for each one? I ask because any extensions would be granted by app key, so that would complicate granting those if so. I'll highlight this with the team in that case.

And, yes, we do now only support creating new API apps as scoped apps (as scoped apps can access all of the same functionality as non-scoped apps). I see you opened a new thread on the scopes parameter behavior in particular though so I'll follow up with you there regarding that.

ccle-operations

Just following up as a user of the Moodle.

Greg you mentioned "any extensions would be granted by app key", how do we request the extension and what do we ask for?

Greg-DB

@ccle-operations If you are unable to migrate your app's code by the retirement date, please feel free to open an API ticket here and include the app key(s) for your app(s) and let us know how long of an extension you would need. I can't make any promises myself, but the team will review the request to see if we can offer an extension.

andrewnicols

Hi Greg,

Yes - that is correct. We develop an Open Source learning management system (Moodle). Anyone is free to download Moodle from our website/github, install it on their own server with their own domain and under their own control, and use it. We do have a SaaS offering, but the vast majority of our installations is people hosting their own sites (or paying someone else to).

We have a number of repository integrations, one of which is Dropbox. This allows students and teachers to uploadcontent from their own Dropbox account into Moodle. One example of this is a student who has completed written coursework submitting this for assessment.

Because each installation of Moodle can have its own DNS, its own hosting, etc. they must register their own Dropbox developer account, get their own developer key/secret, and provide their own OAuth2 callbacks which match their hosting/DNS. Moodle is unable to do this for them (even if we wanted to) because of the OAuth2 callback requirement and the fact that the sites control their hostname.

In addition, the administrator of each installation of Moodle site can choose what version to run, when to upgrade, etc. We have little-to-no control over when administrators do this. Moodle is used widely across the world so the academic year is about to start in the Southern Hemisphere (Feb - Dec), whilst in the Northern hemisphere we're half-way through the year (Sept - July). Many users will only upgrade outside of term times, and some will only upgrade between academic years.

We may be able to request extensions for the small number of sites that we do control, but the vast majority of our users manage their own integrations.

As I say, our next scheduled release of Moodle is on the 8th March, which is after the v1 search API has been retired, so unless users patch their installations between releases (unlikely for most), this change will break dropbox usage for existing installations for at least a week, but likely longer as many sites will not want to upgrade until their Easter break in April. Online learning is too important for many of these users to 'risk' an upgrade during term time, with so many of their students being home schooled due to Covid.

I hope this helps explain my concerns,

Andrew

Greg-DB

I see, thanks! That's helpful context.

andrewnicols

Hi @Greg-DB ,

I'm just wanting to confirm that you have now, indeed, canned the perfectly valid search API in favour of the problematic search_v2 API as announced? The end date you announced was 28th February and that date has now passed.

I'm very disappointed by the lack of meaningful response to the concerns that I have raised regarding the timeframe of this change.

Best wishes,

Andrew

Greg-DB

@andrewnicols Thanks for the note. We have not actually yet retired the /2/files/search endpoint. The team is checking the usage and reviewing the feedback such as yours and may decide to extend this. I'll let you know when I have an official update on that.

Greg-DB

Good news, the team has finished reviewing this and has decided to extend this, bringing the retirement date for /files/search to October 28. Hope this helps!