Child Item

Silent failures with files_upload and Python API

I am having trouble figuring out the right way to address this issue I am having with one of my apps.

I loop over some files and create some worker threads (just a few threads) to upload files to Dropbox using files_upload. This works well except some small percentage of time some users report some files don't get uploaded. There must be some hiccup in the network connection and files_upload hangs somewhere in urllib3 or dropbox maybe? I can sort of reproduce this by disconnecting my wifi during an upload for a few seconds, then reconnecting. If the connection isn't restablished after a certain period of time files_upload returns some how because my threads are freed but the code after my files_upload call never runs and the exception is never thrown/caught. I am using the ApiError exception.

Anyways, is there a proper way to handle this from the Dropbox API perspective? Or is this just Python and urllib3?

Find more posts tagged with

API

Abuse

Comments

scaron

OK, after some more digging, I tried passing a small timeout to the Dropbox client constructor. I then used urllib3 TimeoutError exception. And now I catch something I wasn't catching before.

I should mention I am using QThreadPool within my app and it has a default 30 second timeout for unused threads. So I think this is why I am not getting any exception or error. So, it sounds like I need to align my timeouts for threads and Dropbox client timeout. Then I can catch urllib3's TimeoutError to restart if needed. Does this sound like the right way to utilize the Dropbox client API?

Greg-DB

Yes, it sounds like you have the right handle on this now. For the Dropbox API calls themselves, they should either return a result, or raise some sort of exception (such as an API error, or some network error, etc.). They shouldn't fail silently.

It seems that the issue with them failing silently was due to the thread pool throwing away the threads that take a certain amount of time, so yes, you'd want to configure the thread pool accordingly for this use case. Exactly how you do so is up to you, but you could potentially set the thread expiration to be sufficiently large to accommodate however long the API call might take (before succeeding or failing in some way; and you can set a timeout on the Dropbox client as you mentioned), or perhaps disable the thread expiration entirely, if possible.

scaron

I am hesitant to pass None or disable timeouts entirely. I would rather restart the thread once I know a timeout has occurred. There is a whole slew of internet connectivity issues that could happen on the user end which I don't want to mask by just waiting infinitely.

Thanks for your reply!