Child Item

reading parquet file using python sdk

Hi, I am trying to read a parquet file using pandas and vaex. I can sucessfully read a .csv but I get the following error message when I try to download the parquet file with dbx.files_download :

dropbox.exceptions.ApiError: ApiError('4ce7ef4f93544a4fa18c29478e1869a8', DownloadError('path', LookupError('not_file', None)))

Full code:

ACCESS_TOKEN = "My_Token"

# Initialize the Dropbox API client
dbx = dropbox.Dropbox(ACCESS_TOKEN)

# download csv file from dropbox
metadata, f_csv = dbx.files_download('/County_test.csv')

# this works for csv and pandas
with io.BytesIO(f_csv.content) as stream:
    df = pd.read_csv(stream, index_col=0)
print(df)

# this works for csv and vaex
with io.BytesIO(f_csv.content) as stream:
    df = vaex.read_csv(stream, index_col=0)
print(df)

# download parquet file from dropbox FAILS
metadata, f_parquet = dbx.files_download('/County_test.parquet')

# this part NOT tested yet
with io.BytesIO(f_parquet.content) as stream:
    df = pd.read_parquet(stream, index_col=0)
print(df)

Find more posts tagged with

API

Abuse

Accepted answers

Greg-DB

Thanks for following up and sharing that. For reference, the URL parameters for shared links like that are documented here.

All comments

Greg-DB

I see you're getting a 'not_file' error, which means: "We were expecting a file, but the given path refers to something that isn’t a file."

It looks like the ".parquet" is not a file, but rather a sort of folder, possibly referred to as a "package" or "bundle" in some environments.

That being the case, to download that you would instead need to use files_download_zip (or files_download_zip_to_file) and then unzip the downloaded zip file, or walk through the contents using files_list_folder/files_list_folder_continue and then download each individual nested item using files_download (or files_download_to_file). The first option of using files_download_zip is probably better/faster.

notoriusjack

Thanks for replying. I have tried with one of your solutions but it's really slow.

file_list = []
for entry in dbx.files_list_folder(path="/County.parquet").entries:
print(entry.path_lower)
    _, dwnld_file = dbx.files_download(entry.path_lower)
with io.BytesIO(dwnld_file.content) as stream:
        pd_df = pd.read_parquet(stream) #this works
        vdf = vaex.from_pandas(pd_df)
del pd_df
        file_list.append(vdf)
        conc_df = vaex.concat(file_list)
print(conc_df)

I have tried with dbx.files_download_zip but I can't find a way to read the data it returns 'utf-8' codec can't decode byte 0x82 in position 12: invalid start byte.

Greg-DB

Yes, using files_list_folder/files_list_folder_continue and files_download (or files_download_to_file) requires more API calls so it would be less performant.

The files_download_zip method would return the requested data the same way files_download would, except that it would be zip data that you would need to unzip to access the original folder. Also, note that you can use files_download_zip_to_file if you want to save the zip data to a file. Can you share the code you're having trouble with for that, and indicate which line fails with that error?

notoriusjack

When I try:

md, zipFile = dbx.files_download_zip('/County_test.parquet')
with ZipFile(zipFile, 'r') as zip:

with ZipFile(zipFile, 'r') as zip: fails with AttributeError: 'Response' object has no attribute 'seek'

I can't understand how to solve this

Greg-DB

The files_download_zip method works like the files_download method, in that the second value it returns is the response object. To access the data from the response object, you would access the 'content' field like you did in your other code snippet. So, in this case, it would be 'zipFile.content'.

Beyond that, refer to the documentation for ZipFile, BytesIO, pandas, etc., for information on using those. Those aren't made by Dropbox so I can't offer support for those in particular.

notoriusjack

Thank you for your support, I just managed to do it with dbx.files_download_zip but it takes more or less the same time to process.

Do you know if pandas or vaex support reading the data directly from a file in Dropbox?

md, zipFile = dbx.files_download_zip('/County.parquet')
file_list = []
with ZipFile(io.BytesIO(zipFile.content), 'r') as zip_ref:
for file in zip_ref.infolist():
if file.filename.endswith('.parquet'):
            pd_df = pd.read_parquet(zip_ref.open(file.filename))  # this works
            vdf = vaex.from_pandas(pd_df)
del pd_df
            file_list.append(vdf)
            conc_df = vaex.concat(file_list)
print(conc_df)

Greg-DB

I can't offer support for pandas or vaex themselves as they are not made by Dropbox. I suggest referring to the documentation for those for information on their capabilities.

notoriusjack

I understand, and I found out it's possible. I put this here as it might help someone else.

You can use copy link (set the permissions as you like) and use the URL inside pandas.read_csv or pandas.read_parquet to read the dataset.

However the copy link will have a 'dl' parameter equal to 0, you have to change it to 1 to make it work. Example:

# this does not work
df = pd.read_parquet('https://www.dropbox.com/s/somecode/part.0.parquet?dl=0')

# this works
df = pd.read_parquet('https://www.dropbox.com/s/somecode/part.0.parquet?dl=1')

Thank you again for helping out

Greg-DB

Thanks for following up and sharing that. For reference, the URL parameters for shared links like that are documented here.