Child Item

Minimal client to "sync" files via API?

Hi all,

I have a question about how to use the Dropbox API to accomplish something: sync a bunch of files into a Dropbox folder.

Sounds easy, right? Well, this is going to be done from an embedded (microcontroller) target, so I have to do 100% of it myself.
I am almost not a backend dev, so I might be overlooking something, or thinking about this in the wrong way.

The proposed method for a minimal "sync" implementation is:
1) Iterate over local files using a set filename wildcard
2) For each file, call files/get_metadata
3) If file does not exist, call files/upload, go to next file
3) If file exists, check the hash & dates against the hash & dates of the existing local file
4) If hash is different, call files/upload to overwrite the remote file

Does that sound right? Am I misusing the API?

The application is that I have a data-gathering device which records to an SD card, but the SD card is significantly hard to access, due to legacy design.
(yes, I could possibly roll my own lambda function that pushes to S3 or something similar, but having files "magically" appear onto a Window PC as a result of this sync operation would be a great feature, in this application.)

And the stretch goal:
On my system, TLS is extremely expensive - 10+ seconds of compute time for handshaking.
Is there a batch-equivalent for getting metadata? I did not see one in the docs.
And/or does Dropbox API allow for a number of sequential HTTP transactions all across a single keep-alive socket?

Find more posts tagged with

API

Abuse

Comments

Здравко

@apullin wrote:
...
And/or does Dropbox API allow for a number of sequential HTTP transactions all across a single keep-alive socket?

Hi @apullin,

Do you mean something like pointed out here? If so, Yes, it is possible. Unfortunately, it's NOT a main target to Dropbox' SDK development (in spite you can provide your own connection mechanism). 🤦 That's it.

About the correct way to sync something: There isn't one correct such a way (especially for embedding)! You can use algorithm suitable to achieve your objectives (these objectives can vary between projects). Whatever you choose as algorithm, the Dropbox storage' accessing way is the same. 😉

Hope this helps.

Greg-DB

That's correct, the Dropbox API allows you to list, upload, and download files (among other operations), but we don't provide pre-built code for a full sync solution like that, so you'll need to write your algorithm to meet the needs of your use case. You can find everything for the API here, including the full terms here.

Note that as a matter of security, all Dropbox API calls do require TLS.

Also, unfortunately, the Dropbox API doesn't offer a way to retrieve the metadata for multiple specific items at once. I'll pass this along as a feature request, but I can't promise if or when that might be implemented. Depending on your use case though, e.g., if the items are under a common parent folder, one thing you can do is use /2/files/list_folder[/continue] to list everything under that parent path and then filter the returned entries for the desired items.

apullin

OK, great. If the expectation is that I should be doing this level of operation manually, then I will forge ahead on implementing that way.

In my application, it is really only one-way push:
If the device has the file, and the DB app folder does not, or if it has an old version, then upload.

Although, my backend colleague is now insisting that we wrap this all in a lambda function, so we can just define our own interface & machinery (like batching) cloud-side, and we know for certain that Lambda can sustain a back-and-forth sequence of HTTP requests on a single held-open socket.
So ... the first prototype might just use that route.

But I will *eventually* try going direct from device -> dropbox API.
(maybe ... device OTA via DB? 🤔 )

Out of curiosity: How does the Dropbox desktop client handle a full bi-directional sync on startup ... surely it doesn't do a hash and an API call (or RPC or whatever internal tooling you have) for every file, each time it is started?

Greg-DB

I don't have specific information about how the official Dropbox desktop application works to share, but it does involve a significant amount of code to monitor the local filesystem as well as changes on the Dropbox servers and sync the two together.