Using php, I call list_folder to get a list of about 2000 files of about 800K each to download.
I then go into a loop calling get_temporary_link, retrieving the file contents and writing the file to local disk.
Out of 2000 requestes, about 3-5 times get_temporary_link will not return any json. (empty string)
Retrying the call the get_temporary_link after trying all files have been tried once has so far always worked.
The files were uploaded long before my download request and no one other than the file sender who is done and myself has access to the files so I don't expect the files would be in use and any indexing should have been long done.
What could cause this random failure ? Not always the same file. I dont see a pattern in which file will fail. I also have not had any network/interent problems.
As development of the app continued, rather than write the files downloaded to local disk, I started uploading them to a sql blob. Since doing that, which added a time delay over a local disk write, the failures of get_temporary_link have vanished. This makes me think there is a maximum request rate and the blob write lowered by request rate under the maximum.
If there is a maximum rate at which get_temporary_link can be called or there is something else I should check, please let me know.
While retry has been effective and saving to blob vs local disk is effective, if there is something I'm missing, I'd like to code for it and not have the errors at all.
thanks!
