Child Item

Dropbox Java API - java.net.SocketException: Too many open files

I am using Dropbox Java API to make a large amount of requests to Dropbox API to createFolder and upload files. Currently, I am doing this sequentially, that is, no parallel thread, etc. I am regularly getting the above error from Dropbox. The pseudo code basically looks like this:

DbxRequestConfig config = new DbxRequestConfig(CLIENT_IDENTIFIER, Locale.getDefault().toString());

DbxClient client = new DbxClient(requestConfig, accessToken);

for ( int di = ; di < (large number) ; ++di ) 
    DbxEntry.Folder folderEntry = client.createFolder(somePath[di]);

client.createFolder will throw DbxException, specifically on DbxException.NetworkIO.

Does the API close all the socket properly after each request?

Find more posts tagged with

API

Abuse

Comments

Greg-DB

Thanks for the report. For reference, can you let us know about how many requests you make before this happens, and post the full output for the exception? Thanks in advance!

DolphinA2

Below is the script we used to create a directory structure for testing. Because Dropbox does not have any API for bulk upload, we essentially traverse the directory tree, and upload file and create directories one at a time. The program would throw "java.net.SocketException: Too many open files" error. I would say, around 20+ directory creation. I checked out your Java API source code, my guess is that there are a few places in your DBRequestUtil.java that DropboxAPI team hasn't closed the InputStream properly. One of the example is readJsonFromResponse() function call, the result is simple returned with 'return reader.readFully(body)'. I have changed it to the following:

T t = reader.readFully(body);
if ( body != null )  body.close();
return t;

______ script for creating directories for testing ________

#!/usr/local/bin//tcsh -f
set level1 = ( a b c d e f g h i j )
set level2 = ( 1 2 3 4 5 6 7 8 9 0 )
set level3 = ( p q r s t u v w x y )
set size = 1024
set count = 128

foreach aa ( $level1 ) 
    mkdir $aa
    dd if=/dev/random of=$aa/file$aa.txt count=$count bs=$size
    foreach bb ( $level2 )
        mkdir $aa/$bb
        dd if=/dev/zero of=$aa/$bb/file$aa$bb.txt count=$count bs=$size
        foreach cc ( $level3 ) 
            mkdir $aa/$bb/$cc
            dd if=/dev/zero of=$aa/$bb/$cc/file$aa$bb$cc.txt count=$count bs=$size
        end
    end
end

Kannan G.1

We currently aren't able to reproduce the issue ourselves. Do you get the error when you only call createFolder in a loop? Or do you also have to call uploadFile?

The readJsonFromResponse() function is not responsible for closing the stream. There's higher-level code that should close the stream.

I found an open source "file leak detector" (http://file-leak-detector.kohsuke.org/). I've never used it before, but it looks straightforward. That might help locate the issue.

DolphinA2

The loop has both createFolder() and uploadFile(). It does not happen all the time, and it happens randomly in the same loop. But when it happens, it seems to be on the createFolder() only, and I haven't gotten it on uploadFile() call.

Let me dig into your suggested library below and see if I have a better understanding on what causes the java.net.SocketException and get back to you.