Better understanding of offset with appendV2 and finish

Michael-jamf

Attempting to upload large files. The regular upload method works without an issue. When I try to use appendV2 and Finish I get that the offset was incorrect and the correct offset. This post does discuss how to do offsets but want to make it clearer.

From the docs. "Offset in bytes at which data should be appended. We use this to make sure upload data isn’t lost or duplicated in the event of a network error."

If I have a file with looping "abcd" * 13*1024*1024 it would give me a 54525952 Bytes or 54.5MB. My chunk size is 4MB (4194304 Bytes).

For this file I would start the upload with the full file .uploadSessionStart(input: abcdFileURL). The initial offset should be my chunksize. My cursor would be .UploadSessionCursor(sessionId, offset) and the append would be .uploadSessionAppendV2(cursor, input: abcdFileURL). Every append after should increment the offset by the chunksize, 4MB, 8MB ... 50MB, until the last offset is equal to or less than the chunksize. This would be the final file size in the finish. Which would be .uploadSessionFinish(cursor(sessionId, fileSize), input: abcdFileSize)
Does that sound correct logically how it should work?

When I attempt this I get that the offset should be the file size for append. When I test if the file is less than 150MB and set offset to file size with the start/append/finish (so that it basically acts like the regular .upload) it works.

Output when using normal append with increasing offset.

Offset Append Value: 155189248
difference 12582912
Offset Append Value: 159383552
difference 8388608
Offset Finish Value: 167772160
File Size: 167772160
Error appending data to file upload session: [request-id 3415da207401412eaf901ef5aef245a8] API route error - {
    ".tag" = "incorrect_offset";
    "correct_offset" = 167772160;
}
Error appending data to file upload session: sessionTaskFailed(error: Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x6000023d7d50 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalUploadTask <7490F290-DA4A-49A9-AA00-B378E94A46E0>.<35>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalUploadTask <7490F290-DA4A-49A9-AA00-B378E94A46E0>.<35>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=https://api-content.dropbox.com/2/files/upload_session/append_v2, NSErrorFailingURLKey=https://api-content.dropbox.com/2/files/upload_session/append_v2, _kCFStreamErrorDomainKey=4})2023-04-20 19:57:32.228273-0600 cloud-tools[61893:1384752] Task <C348ACD5-53C3-4908-9511-697D5FB68DEF>.<44> finished with error [-1001] Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x6000023d54d0 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-210
2, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalUploadTask <C348ACD5-53C3-4908-9511-697D5FB68DEF>.<44>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalUploadTask <C348ACD5-53C3-4908-9511-697D5FB68DEF>.<44>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=https://api-content.dropbox.com/2/files/upload_session/append_v2, NSErrorFailingURLKey=https://api-content.dropbox.com/2/files/upload_session/append_v2, _kCFStreamErrorDomainKey=4}

 

Output where I set the offset to the file size if less than 150MB.

Offset Finish Value: 54525952
File Size: 54525952
Finish Good: {
    "client_modified" = "2023-04-21T01:39:37Z";
    "content_hash" = 9fd01c7cc2807ed423dbd11a1a06b9fd77ad15e843fcbabf4f363da527e06175;
    id = "id:992ScyV19hQAAAAAAAAHbA";
    "is_downloadable" = 1;
    name = "testjamf.txt";
    "path_display" = "/testfile.txt";
    "path_lower" = "/testfile.txt";
    rev = 5f9cebae50714a5e0ed31;
    "server_modified" = "2023-04-21T01:39:37Z";
    size = 109051904;
}

My code that produces the errors.

 

dbxClient.files.uploadSessionStart(input: fileURL)
            .response(completionHandler: { [self] response, error in
                if let result = response {
                    print(result)
                    var offset: UInt64 = chunkSize
                    //Append chunks to file
                    while (offset < fileSize!) {
                        
                        if ((fileSize! - offset) <= chunkSize) {
                            print("Offset Value: \(offset)")
                            dbxClient.files.uploadSessionFinish(cursor: Files.UploadSessionCursor(sessionId: result.sessionId, offset: fileSize!), commit: Files.CommitInfo(path: dbFilePath), input: fileURL)
                                .response { response, error in
                                    if let result = response {
                                        print("Finish Good: \(result)")
                                    } else {
                                        print("Finish Error: \(error!)")
                                    }
                                }
                            offset = fileSize!
                        } else {
                            print("Offset Value: \(offset)")
                            dbxClient.files.uploadSessionAppendV2(cursor: Files.UploadSessionCursor(sessionId: result.sessionId, offset: offset), input: fileURL)
                                .response {response , error in
                                    if let response = response {
                                        print("File appended: \(response)")
                                    } else {
                                        print("Error appending data to file upload session: \(error!)")
                                    }
                                }
                            //                            .progress { progressData in print(progressData) }
                            offset += chunkSize
                        }
                    }
                } else {
                    // the call failed
                    print(error!)
                }
            })
        }

 

Michael-jamf

That makes sense. Thanks for clearing that up. I was thinking you provide the full file, then the offset was used to tell which part to upload. I also ran into an issue when the upload would not finish before attempting the next part so I made it recursive if the file part finished.

 

Anyone that find this. Here is my upload function for small and large files. It could be improved but it works.

 

func upload(file filePath: String, to dbxLocation: String) {
        if FileManager.default.fileExists(atPath: filePath) {
            print("File Exists")
        } else {
            return
        }
        let fileURL: URL = URL(filePath: filePath)
        var dbFilePath: String = ""
        if dbxLocation.last == "/" {
            dbFilePath = dbxLocation + fileURL.lastPathComponent
        } else {
            dbFilePath = dbxLocation + "/" + fileURL.lastPathComponent
        }
        print(dbFilePath)
        let fileSize = try? FileManager.default.attributesOfItem(atPath: filePath)[.size] as? UInt64
        let chunkSize: UInt64 = 4 * 1024 * 1024
        //        Check to see if file is smaller than chunksize
        if fileSize! < chunkSize {
            dbxClient.files.upload(path: dbFilePath, input: fileURL).response(completionHandler: { response, error in
                if let response = response {
                    print("File uploaded: \(response)")
                } else {
                    print("Error upload session: \(error!)")
                }
            })
            .progress { progressData in print(progressData) }
            print("small file")
        } else {
            let data: [Data] = try! self.split(file: fileURL, into: chunkSize)
            //start the upload session
            var offset: UInt64 = UInt64(data[0].count)
            dbxClient.files.uploadSessionStart(input: data[0])
                .response(completionHandler: { [self] response, error in
                    if let result = response {
                        print(result)
                        append(data: data, part: 1, sessionId: result.sessionId, offset: offset, chunkSize: chunkSize, fileSize: fileSize!, dbFilePath: dbFilePath)
                    } else {
                        // the call failed
                        print(error!)
                    }
                })
        }
    }
    
    func append(data: [Data], part: Int, sessionId: String, offset: UInt64, chunkSize: UInt64, fileSize: UInt64, dbFilePath: String) {
        let chunk = data[part]
        if ((fileSize - offset) <= chunkSize) {
            dbxClient.files.uploadSessionFinish(cursor: Files.UploadSessionCursor(sessionId: sessionId, offset: offset), commit: Files.CommitInfo(path: dbFilePath), input: chunk)
                .response { response, error in
                    if let result = response {
                        print("File Uploaded: \(result)")
                    } else {
                        print(fileSize)
                        print("Finish Error: \(error!)")
                    }
                }
        } else {
            dbxClient.files.uploadSessionAppendV2(cursor: Files.UploadSessionCursor(sessionId: sessionId, offset: offset), input: chunk)
                .response {response , error in
                    if let response = response {
                        self.append(data: data, part: part + 1, sessionId: sessionId, offset: offset + UInt64(data[part].count), chunkSize: chunkSize, fileSize: fileSize, dbFilePath: dbFilePath)
                    } else {
                        print("Error appending data \(part) to file upload session: \(error!)")
                    }
                }
        }
    }
        
    func split(file fileURL: URL, into chunkSize: UInt64) throws -> [Data] {
        let data = try Data(contentsOf: fileURL)
        let chunks = stride(from: 0, to: data.count, by: Int(chunkSize)).map {
            data.subdata(in: $0 ..< Swift.min($0 + Int(chunkSize), data.count))
        }
        return chunks
    }