Child Item

Architecture pattern used by Dropbox

As we have many software architecture patterns available like peer-to-peer, client server, Observer and a lot more, which one does Dropbox uses ? and why it has selected that specific architecture? Please list out its pros and cons as i am unable to find much info on these topics.

Find more posts tagged with

View

Desktop

OffTopic

Insightful

Accepted answers

Jane

Hey @Akhil_M_S @Akhil_M_S, I'd happily take it from where you left off with @Daphne; I hope my message finds you well today!

To answer your initial question Akhil, different features on different platforms tend to have slightly different models, but the most common pattern is the observer pattern. This is implemented as long-polling from the client to the server, which holds the canonical representation of the data. For example, we do this for a lot of sync features, but also for comments/file activity. Where we don't use long-polling, we do client/server polling.

Hope that helps with your class presentation and good luck with your project!

All comments

Daphne

Hey there @Akhil_M_S,

For some more info about the architecture overview, please check out this article here. If you have any other questions feel free to let me know. In terms of the pros and cons, this discussion will be open for others to input their opinions on the matter.

Hope this helps - Thanks!

Akhil_M_S

Hi @Daphne ,

Thanks for the link. I had gone through the same yesterday. But i didn't get the exact answer. Actually my question is regarding the architecture pattern used by Dropbox. For example when i go through the link you attached it feels its a Publisher Subscriber pattern, but some say it as a Observer pattern with central server pushing the changes to the client. Could you please help me with the same ? As i have chosen this topic to present in my class..

Thanks,

Akhil

Daphne

Hi there @Akhil_M_S,

I will look further into this for you to see what I can find! I’ll get back to you with an update when I have some more info on this.

Until then, have a great day!

Akhil_M_S

Hi @Daphne

Sure, will be waiting for you. Thank you so much.

Have a good day !

Jane

Hey @Akhil_M_S @Akhil_M_S, I'd happily take it from where you left off with @Daphne; I hope my message finds you well today!

Hope that helps with your class presentation and good luck with your project!

prasanthkp

Hi Daphne,

I was reading about the Dropbox architecture. (Streaming File Sync). And in my search for the backend database details I found this thread. In the Under the hood : architecutre overview , i understood that Metadata database is a RDBMS. Is it fair to understand that the block storage where the actual files is on the cloud (S3)?

Walter

Hi @prasanthkp; how are you today?

Sorry to jump in here, but quoting from the relevant resource I just wanted to let you know that (sic) once a file is added to your Dropbox, it's synced to our secure online servers. All files stored online by Dropbox are encrypted and kept in secure storage servers. Storage servers are located in data centers across the United States. Additionally, storage servers are available in Germany, Australia, and Japan for some Dropbox Business users.

For reference, here's a diagram of how the service exactly works:

Diagram of how the Dropbox service works

Let us know if you have any additional questions; we'll be happy to follow up!

prasanthkp

Hi Walter,

Am good. Hope all is well there.

Thanks for the quick reply. I appreciate that.

The reference diagram was helpful. That answer my question partially. My question was more towards the database. Based on the reference diagram,

Block stroage servers - Is this something like Amazon S3?
Metadata Database - I read in one of the blog ((Re)Introducing Edgestore), Dropbox uses MySql?

Is the above a right understanding?

I guess MySql was opted to maintain ACID. How is sharding done as scaling a RDBMS is a costly effort.

Thanks in advance.

Prasanth

Walter

Thanks for getting back to me Prasanth; how are you today?

I'll try to address your questions here but please feel free to also check our security whitepaper as most of your concerns are addressed within that document.

To cut to the chase now, note that (1) the actual contents of users’ files are stored in encrypted blocks with the Block Storage Servers while prior to transmission, the Dropbox client splits files into file blocks in preparation for the storage. The Block Storage Servers act as a Content-Addressable Storage (CAS) system, with each individual encrypted file block retrieved based on its hash value.

And (2) File metadata is stored in a MySQL-backed database service, and is sharded and replicated as needed to meet performance and high availability requirements.

And to further elaborate, file blocks are stored in one of two systems: Amazon Web Services (AWS) or Magic Pocket, our in-house storage system. Magic Pocket consists of both proprietary software and hardware and has been designed from the ground up to be reliable and secure. In both Magic Pocket and AWS, file blocks are encrypted at rest, and both systems meet high standards for reliability.

I hope this information helps and we're always around if you have any additional queries; just give us a nudge here @prasanthkp!

prasanthkp

Thanks Walter. I am doing good. How about you?

Your explanation helps. I appreciate that. I am reading the docs and will revert if I have more questions.

Cheers.

Prasanth

Walter

Hi again Prasanth! I'm fine as well; thanks for asking.

I'm glad to hear the information I provided helps and should you come up with any additional Dropbox questions, please feel free to let us know.

Until we speak again, have a wonderful weekend @prasanthkp!

prasanthkp

Hi Walter,

Hope you are doing great.

I went through the security whitepaper. Thanks for sending this. It had lots of information. I would like to understand more about how you shard the MySQL db. As you mentioned below in the previous post.

"And (2) File metadata is stored in a MySQL-backed database service, and is sharded and replicated as needed to meet performance and high availability requirements."

I assume you use hash partitioning.

How is sharding done?
How edgestore routes to the correct "shard"?
How is it replicated across different DC?
Also is it still benefical to go with MySQL than a NoSQL db?
Is there a detailed paper or blog around the sharding alone?

Thanks in advance.

Prasanth

Walter

Thanks for the follow up and happy Friday Prasanth!

When it comes to sharding, please take a look at the following resources; I think they make an excellent read:

https://blogs.dropbox.com/tech/2018/11/cross-shard-transactions-at-10-million-requests-per-second/
https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure/ (this is an older blog post from 2016)

If you're still having questions, let me know if it would be OK if I reached out to you via a ticket to your Dropbox associated email address so I can then forward your queries to the relevant team and we'll take it from there.

Thanks a bunch and have a lovely weekend @prasanthkp!

prasanthkp

Thanks Walter. Am reading through the blog post.