PROBLEM Dropbox does not currently provide an ignore file/folder feature that enables a locally-unique instance to exist on each machine. One important use of this is the “node_moduels" folder used for node javascript development. The node_modules folder sees thousands of file changes during the development cycle. If this folder is allowed to sync, the important files you need (outside of it), are so far down in the sync queue that you never have the important files when you need them on your other systems. SOLUTION A better way to sync projects with Dropbox while ignoring the node_modules folder is to use a “properly” formed symlink (alias). At this time npm and yarn support a property formed symlink. HOWEVER, IT IS CRITICAL THAT THE LINK (ALIAS) AND TARGET HAVE THE SAME NAME! From shell prompt (terminal): * cd YOUR_PROJECT_NAME * mkdir -p ~/Downloads/node_modules_for_YOUR_PROJECT_NAME * mv node_modules ~/Downloads/node_modules_for_YOUR_PROJECT_NAME * ln -s ~/Downloads/node_modules_for_YOUR_PROJECT_NAME/node_modules Now create the same target node_moduels folder on each of your other systems **AFTER** Dropbox syncs the ENTIRE project direcotry. The old node_modules folder should be removed and the symlink (alias) added before running these steps. * cd YOUR_PROJECT_NAME * mkdir -p ~/Downloads/node_modules_for_YOUR_PROJECT_NAME/node_modules * yarn install SUCCESS Dropbox WILL sync the symlink/alias file itself, but NOT the contents it points to. This seems to work well for file syncing across several of your own development systems, and then use git to push to the repo for others on the team. CAVEAT The tiny symlink (alias) file itself IS sync’d on all systems, but will point to a non-existent / empty folder on the other systems. This is most applicable for a single user, but can also be useful for teams if an individual variant of the ignored data is appropriate, such as for node_moduels. Hope this helps. 🙂 ..sebastian

Child Item

How to make Dropbox ignore node_modules folder with symbolic links (aliases)

PROBLEM

Dropbox does not currently provide an ignore file/folder feature that enables a locally-unique instance to exist on each machine. One important use of this is the “node_moduels" folder used for node javascript development. The node_modules folder sees thousands of file changes during the development cycle. If this folder is allowed to sync, the important files you need (outside of it), are so far down in the sync queue that you never have the important files when you need them on your other systems.

SOLUTION

A better way to sync projects with Dropbox while ignoring the node_modules folder is to use a “properly” formed symlink (alias). At this time npm and yarn support a property formed symlink. HOWEVER, IT IS CRITICAL THAT THE LINK (ALIAS) AND TARGET HAVE THE SAME NAME!

From shell prompt (terminal):

cd YOUR_PROJECT_NAME
mkdir -p ~/Downloads/node_modules_for_YOUR_PROJECT_NAME
mv node_modules ~/Downloads/node_modules_for_YOUR_PROJECT_NAME
ln -s ~/Downloads/node_modules_for_YOUR_PROJECT_NAME/node_modules

Now create the same target node_moduels folder on each of your other systems **AFTER** Dropbox syncs the ENTIRE project direcotry. The old node_modules folder should be removed and the symlink (alias) added before running these steps.

cd YOUR_PROJECT_NAME
mkdir -p ~/Downloads/node_modules_for_YOUR_PROJECT_NAME/node_modules
yarn install

SUCCESS

Dropbox WILL sync the symlink/alias file itself, but NOT the contents it points to. This seems to work well for file syncing across several of your own development systems, and then use git to push to the repo for others on the team.

CAVEAT

The tiny symlink (alias) file itself IS sync’d on all systems, but will point to a non-existent / empty folder on the other systems. This is most applicable for a single user, but can also be useful for teams if an individual variant of the ignored data is appropriate, such as for node_moduels.

Hope this helps.

🙂

..sebastian

Find more posts tagged with

Desktop

Sync

Insightful

Closed - Partially Delivered

Accepted answers

sebastian_

NOTE: The Initial Post on this thread IS the Solution. While the technique specifically targets the node_modules folder in Javascript / TypeScript projects, it can also be applied to any "folders" that need to be ignored (see bottom of this post for a drag-n-drop technique).

HOW IT WORKS

Because Dropbox does not synchronize folders referened by symlinks (aliases), you can move those folders into a NON-DROPBOX path (outside of dropbox), then create a symlink (alias) in the original Dropbox location.

RESULT

The ignored folder in Dropbox looks and behaves as normal folder, but in reality the data lives in a non-synchronized location on the storage device.

HOW TO IGNORE ANY FOLDER (THE DRAG-N-DROP WAY)

Move the folder to ignore "Outside of Dropbox" into another location (e.g. ~/Documents)
In Finder (Mac OS), Command+Option Drag the folder back into Dropbox to create an alias. The cursor should turn into a little curved arrow while dragging if you have the right keys pressed. Once dropped, an alias will be created in the original Dropbox location pointing to the moved folder.

NOTE: Using symlinks/aliases on Windows and Linux should behave the same way, but I have not verified it. Also, I've updated the original post to state that the tiny symlink (alias) file itself "IS" copied to all systems, but not the content of the folder.

..sebastian

All comments

miromannino

Thank you for posting this idea!

The only thing is... where is this working? In OSX Dropbox is still syncing the symlinks.

sebastian_

HOW IT WORKS

RESULT

The ignored folder in Dropbox looks and behaves as normal folder, but in reality the data lives in a non-synchronized location on the storage device.

HOW TO IGNORE ANY FOLDER (THE DRAG-N-DROP WAY)

Move the folder to ignore "Outside of Dropbox" into another location (e.g. ~/Documents)
In Finder (Mac OS), Command+Option Drag the folder back into Dropbox to create an alias. The cursor should turn into a little curved arrow while dragging if you have the right keys pressed. Once dropped, an alias will be created in the original Dropbox location pointing to the moved folder.

..sebastian

sebastian_

I received a response from Tech Support explaining how to use the xattr command in terminal to ignore a file/folder (link below).

There is a an upside and a downside to each approach (xattr and symlink). The xattr method ignores the target file/folder content and does not upload it, HOWEVER, upon EVERY change within node_modules, Dropbox reindexes and pegs the processor for long periods. The symlink method avoids indexing entirely, so it has a zero-impact when changes within node_modules occur. The symlink/alias file may point to a non-existent folder on a system that doesn't need it, so it's ideal for node_modules, but may not for all others.

You can find the xattr ignore Dropbox article here, plus ongoing conversation about the ignore topic here (pages 87, 88, etc...). Hopefully the upcoming xattr ignore method (currently in beta) will support a hard-ignore without reindexing before it's released.

lkljsjjjsjj

Are you a god?

I've been searching for a solution for several years! I wouldn't expect it to be as simple as a symlink. Kudos!

Arne1

It does not work for me (on Windows). The first npm install is fine, the second run will basically delete the symlink, recreate the directory, add some packages and then fail with a file not found error

sebastian_

@lkljsjjjsjj, Glad to hear it works for you!! I too pulled my hair out for years before figuring this configuration out.

@Arne1, Sorry to hear it isn't easily translating to Windows. Is the "second run" failure happening on the same system, or on a secondary system? I no longer have access to a Window platform or I'd be happy to troubleshoot it.

Windows Secondary System: “Theoretically", Dropbox should sync the symlink/alias file itself to all clients subscribed to the same Dropbox/share. Once the symlink/alias file itself is sync'd to your other system(s), you need to create the exact same target folder path (e.g. "C:\users\your_username\downloads\node_modules_for_projectname\node_modules").
Windows Secondary Run (on same system): "Theoretically" NPM should behave the same, regardless of platform, but maybe not. Have you tried yarn? What do the commands do after creating the node_modules symlink=>downloads_folder?

yarn install
yarn upgrade --latest

Unfortunately, I do not know the behavior of symlinks in cross-platform implementations (e.g. Mac Dropbox syncing w/ Windows and/or Linux Dropboxes).

** Can someone working with NPM on Windows help with a “Working and Tested” set of steps for Windows and post back? **

Arne1

Hi,

Thanks for the reply! It is the same machine.

It seems to work with yarn though, but I can't switch easily since other people work with the code (and lock file) too.

Arne1

After switching to yarn, I've noticed that composer (the PHP package manager) also doesn't like if its directory ("vendor" in this case) is a symlink.

So at the moment, this trick does not work with NPM or composer.

sebastian_

@Arne1 , That's a bummer. I’ve definitely seen various commands on the toolchain not respecting symlinks and takes a bit of effort to figure it out. You might consider posting a question to the tool's project page asking for help getting symlinks working. If/When you find a workaround for your toolchain, please post it back here to help the rest of the developer community.

Until we have a .dropboxignore (a “true ignore”, without reindexing), sharing workarounds (hacks) are the only course of action. If a .dropboxignore feature existed, I’d put far more of my projects into Dropbox.

Terry U.1

Ao I removed all node_modules folders on both my iMac and actually from within my Dropbox account.

However, since then my Dropbox won't complete, it has been on day after day and disabled sleep. It is stills tuck on 500K files. I had a very similar incident on my Macbook Pro when I removed a load of files via Selective Sycn and this too over a month to resolve, having to leave the machine on full time during this period without sleep. So, how can I resolve this.

At the moment, Dropbox is unusable. I am close to finding another provider who does sync files effectively, something that Dropbox was great at it, in it's early days. Now it seems to struggle with the basics of Syncing. Instead adding useless features like connecting Google Calendar and Outlook.... Why not focus on the core functionality of the syncing and ignoring of files. having been a paid subscriber since 2012 it is with disappointment and regret I am faced with having to look to another provider.

qrayg

I wrote an Apple Script for this specific problem (sorry Windows users):
https://gist.github.com/craigerskine/fe1278ca405d58cd61e349797f96d9d0

fernandorojo

Any shot you could upload a video showing how you did this? Slightly hard to follow with your usage of the Downloads folder. Thanks!

devinrhode2

Symlinks are probably the best general solution. For developers using Git, you have another option. If you are fanatic about creating great git history/commit log for your changes, then you should try using `git worktree add ../projectA-dupe` which will create a duplicate folder that's a sibling to your current git repo. It's actually another "working directory" in proper git terms. It's the same git repo, all the same branches, same stashes, all the git info is the same. Actually the .git dir for "projectA-dupe" basically just points to your original git repo's .git folder. The advantage here, the workflow, is to have your main working directory outside of dropbox entirely, and then add another "working directory" or "worktree" inside your dropbox folder, which will sync. Or, vice versa. You could have the main git repo be inside dropbox, but, setup an npm pre-install hook to fail installation if it sees "Dropbox" is in your current folder path. All developers are instructed to cd to the main git repo inside Dropbox, and then run `git worktree add ../../path/to/your/projectA`. Furthermore, you could have everyone keep their worktree inside dropbox, which means they could all collaborate without ever doing a git push/pull/fetch, by simply allowing dropbox to sync all the git objects. One developer could help another simply by cd'ing to their worktree inside of dropbox. They see the exact same thing, no need to stash changes, commit anything, etc. The symlink might as well be committed into the git repo, and point to a globally reliable path, like ~/universal_node_modules/projectA. Better than all that, probably use yarn v2, and inside .yarnrc.yml set cacheDirectory to be something like ~/Dropbox/.yarn-cache/projectA, so all worktree's can share the same cache.

sp1thas

I've recently implemented dropboxignore which is a simple shell script which facilitates you to generate .dropboxignore files based on your file patterns or even based on existing .gitignore files and ignore matched files from dropbox. Hope to find it useful for your case. Any feedback is more than welcome. dropboxignore is currently available only for Mac OS and Linux.

Riderpaul

This is just dumb. No system should ever back up node_modules. This makes me want to rethink my Dropbox subscription.

pdepedo

It's 2021 and Dropbox still has no ignore functionality. I'm wondering how difficult would that be, really. Just before you sync a folder, check for existence of a .dropboxignore file, and then make sure you don't sync the files listed there. Like, seriously. How long can this take to implement? A week? A month? What the f are you waiting for?

rvvincelli

Thank you so much! After so much time wasted fiddling with xattr andthe Dropbox selective sync!!

Josh W.14

I've fought with dropbox syncing node_modules, .git/, etc for years hoping the team would release a fix. I set up sym links, fs watchers to auto-set the ignore attribute for new folders, etc. To get it all working I have to turn off sync on all my computers, delete node_modules from dropbox.com, re-add the folder on each computer, re-add the ignore attribute, then re-enable syncing. Same for .git repos. If I ever need to blow away node_modules, I run `rm -rf node_modules/*`, otherwise I've got to redo the entire process. I've always loved Dropbox and always recommend it to family & colleagues, but last week I ran out of patience because all the Dropbox node_modules thrashing was destroying my IDE. I killed Dropbox & installed mega.io. It lets you setup a global ignore list where I added node_modules, .git, .idea, etc. It just works. I'll still recommend Dropbox for family, but for tech friends Dropbox is a nightmare.

Spl221

I agree that if dropbox team does not support the users with must-have features for ages, I would rather cancel my subscription and use a better service. Could anyone recommend something that is aware about ignore patterns and has an UI for Linux/Windows machines?