sunrat Posted November 28, 2017 Share Posted November 28, 2017 I have now got 2 computers running having brought my old one out of retirement. The old one (bedroom) is mainly for everyday use, and the newer one (loungeroom) is mainly for music production although it's bound to be used for games and watching video as it has my new Sony 43" 4K TV as its monitor. I need to sync files both ways between them as either may be used to add files, mainly to my music collection which currently is about 650GB, but also for backups of other data such as recording production sessions. Both are on the same network via wifi. I considered rsync which I have always used for backups to external drives but it seems more suited for one way transfers and needs ssh or similar over network. Then a search revealed Syncthing which seems like it could be suitable. Setup looks easy with only firewall settings to consider. The FAQ mentions things that are not synced which hopefully are not an issue: The following are not synchronized;File or Directory Owners and Groups (not preserved) Directory Modification Times (not preserved) So my questions are - has anyone experience using Syncthing? Does it look like the best tool for the job? It's open source and available in repos for Debian and Arch. https://syncthing.net/ Quote Link to comment Share on other sites More sharing options...
Hedon James Posted November 28, 2017 Share Posted November 28, 2017 I haven't used Syncthing, but everything I've read suggests that it would be a good tool for the job. FWIW, you are correct, IMO, that rsync is best suited for 1-way backups. It sounds like you might also be interested in Unison, which is a 2-way synchronization designed to "mirror" multiple locations. Is this what you're looking to accomplish: https://www.howtoforge.com/tutorial/unison-file-sync-between-two-servers-on-debian-jessie/ Assuming Unison also does the job, someone else will have to chime in on the pros vs. cons of each. I believe Syncthing is multi-platform, including Android, so that could be a plus, as I'm not sure Unison is available for cell phone platforms. Quote Link to comment Share on other sites More sharing options...
securitybreach Posted November 28, 2017 Share Posted November 28, 2017 I have a daily cron job that runs syncs data between drives using rsync https://wiki.archlinux.org/index.php/rsync#Automated_backup Basically it copies over anything that is on the host but not on the backup so that way both drives have the same exact data. This should work just fine between computers as well. There is no reason to run an additional application as you can set this as a cronjob and it will run automatically. Example of mine: http://forums.scotsnewsletter.com/index.php?showtopic=90612entry443202 1 Quote Link to comment Share on other sites More sharing options...
sunrat Posted November 28, 2017 Author Share Posted November 28, 2017 Thanks for your advice but it still looks like Syncthing will be the best for my use case. I was mainly just wondering if anyone had experience with it. I will try it and let you know how it goes. For phone data swapping I use KDE Connect. Perfect for what I need there. 1 Quote Link to comment Share on other sites More sharing options...
securitybreach Posted November 28, 2017 Share Posted November 28, 2017 Cool, let us know what you ended up using. Quote Link to comment Share on other sites More sharing options...
sunrat Posted April 6, 2019 Author Share Posted April 6, 2019 It took a little time to get around to it, but I finally set up Syncthing between the 2 computers. Did some reading of the docs, watched a couple of videos and then installed on both computers and had it up and running in about 10 minutes! So far only added the default test folders so I'll have to get up some more nerve to sync the actual hundreds of gigabytes that exist on both computers. This was prompted by a minor installation catastrophe on my production computer where I somehow wiped out the whole partition table on the SSD. No data lost as that's on separate drives and backed up as well, but I wiped out about 7 installed operating systems most of which needed some spring cleaning due to 3 years of experimental program installs and such. The most painful bit was hours reinstalling and tweaking Win10 and still have my favourite games to go. I'd just installed Forza Horizon 3 which took ages. Edit: It seems I didn't explain very well. The catastrophe was prior to installing Syncthing and caused by an OS installation failure as explained below. Syncthing works perfectly so far. 2 Quote Link to comment Share on other sites More sharing options...
securitybreach Posted April 6, 2019 Share Posted April 6, 2019 It took a little time to get around to it, but I finally set up Syncthing between the 2 computers. Did some reading of the docs, watched a couple of videos and then installed on both computers and had it up and running in about 10 minutes! So far only added the default test folders so I'll have to get up some more nerve to sync the actual hundreds of gigabytes that exist on both computers. This was prompted by a minor installation catastrophe on my production computer where I somehow wiped out the whole partition table on the SSD. No data lost as that's on separate drives and backed up as well, but I wiped out about 7 installed operating systems most of which needed some spring cleaning due to 3 years of experimental program installs and such. The most painful bit was hours reinstalling and tweaking Win10 and still have my favourite games to go. I'd just installed Forza Horizon 3 which took ages. Wow, I guess that didn't work as intended 1 Quote Link to comment Share on other sites More sharing options...
securitybreach Posted April 6, 2019 Share Posted April 6, 2019 So you synced your working drive with a blank one and it synced the wrong way? It would be a lot easier to simply add this to a daily/hourly cron job: rsync -ar --delete /Source /Target -a indicates that files should be archived, meaning that most of their characteristics are preserved (but not ACLs, hard links or extended attributes such as capabilities) --delete means files deleted on the source are to be deleted on the backup as well -r recurse into directories This works flawlessly and I have been using it for probably a decade now. https://wiki.archlin...index.php/Rsync Cerberus :: ~ » crontab -l00 15 * * * rsync -ar --delete /Various/comhack/Downloads /Various/comhack/Miscellaenous /Various 00 16 * * * rsync -ar --delete /MEDIA/Documents /Various/comhack/ 00 17 * * * rsync -ar --delete /Backup2/Movies/ /Various/Moves/ 00 17 * * * rsync -ar --delete /MEDIA/Other /Backup/ Quote Link to comment Share on other sites More sharing options...
securitybreach Posted April 6, 2019 Share Posted April 6, 2019 Sorry, I didn't include a link to the important part for you... Automated backup with SSH Quote Link to comment Share on other sites More sharing options...
sunrat Posted April 6, 2019 Author Share Posted April 6, 2019 The catastrophe was before Syncthing in case I was unclear. Some glitch while installing Debian Buster I think but compounded by some misconfiguration of the partitions which I had already had minor issues with before. It destroyed the partition table rather than wiping the drive. I installed Buster successfully now on the new setup. It felt good to start fresh with a clean slate. Syncthing so far appears to be working perfectly and browsing through the manual made me aware of just how powerful and configurable it is. The idea is to have both computers with identical data so I can change something on either one and have it sync to the other. Not to just have one as backup for the other. That's harder to do with rsync. Quote Link to comment Share on other sites More sharing options...
securitybreach Posted April 6, 2019 Share Posted April 6, 2019 The catastrophe was before Syncthing in case I was unclear. Some glitch while installing Debian Buster I think but compounded by some misconfiguration of the partitions which I had already had minor issues with before. It destroyed the partition table rather than wiping the drive. I installed Buster successfully now on the new setup. It felt good to start fresh with a clean slate. Syncthing so far appears to be working perfectly and browsing through the manual made me aware of just how powerful and configurable it is. Ah ok, I may check it out. Quote Link to comment Share on other sites More sharing options...
sunrat Posted April 6, 2019 Author Share Posted April 6, 2019 It's like having a Dropbox sync folder on 2 devices but without having to upload to a Dropbox server in between. Quote Link to comment Share on other sites More sharing options...
securitybreach Posted April 6, 2019 Share Posted April 6, 2019 It's like having a Dropbox sync folder on 2 devices but without having to upload to a Dropbox server in between. Well I host my own cloud storage using Nextcloud. Quote Link to comment Share on other sites More sharing options...
sunrat Posted April 6, 2019 Author Share Posted April 6, 2019 It's like having a Dropbox sync folder on 2 devices but without having to upload to a Dropbox server in between. Well I host my own cloud storage using Nextcloud. You wouldn't be needing Syncthing then. Having cloud storage is a step too far for me as I never need access to my data when I'm not home. It can be used securely over the internet as it uses strong encryption for blocks of data. It can also be used to sync any number of computers which has the added advantage of speeding up the process similar to bittorrent sharing. Available for Linux, Windows, Mac, Android, BSD, Solaris etc. Check it out: https://syncthing.net/ Quote Link to comment Share on other sites More sharing options...
securitybreach Posted April 7, 2019 Share Posted April 7, 2019 It's like having a Dropbox sync folder on 2 devices but without having to upload to a Dropbox server in between. Well I host my own cloud storage using Nextcloud. You wouldn't be needing Syncthing then. Having cloud storage is a step too far for me as I never need access to my data when I'm not home. And that is why I use rsync for local stuff.. Quote Link to comment Share on other sites More sharing options...
sunrat Posted April 7, 2019 Author Share Posted April 7, 2019 And that is why I use rsync for local stuff.. I use rsync to back up my home folder and it's perfect for that. But I wouldn't know how to set it up to automatically do a 2-way sync between 2 computers, or even if that's possible. I did the full sync of my 132GB data partitions yesterday and it went almost without a hitch. I had to set the .firefox and .thunderbird profile folders to be excluded as I used them while the sync was happening and they both create temp files which confuse the process. Otherwise all was perfect. It did take a few hours to do it as syncthing does a hash of every file on each computer. Subsequent syncs are almost instant though. I'll score it 9/10. Unbelievably easy to use after you learn the basics. 1 Quote Link to comment Share on other sites More sharing options...
sunrat Posted April 18, 2019 Author Share Posted April 18, 2019 Another 10 days of using Syncthing and it's definitely a winner. I synced up my 680GB music collection which went well. A couple of minor things to be aware of: - It tends to copy things faster than deleting other things so be careful you have enough disk space. I was moving many GB of files from one directory to another on the same drive which were simultaneously syncing to the other system and its disk got to 97% full. Worked out ok though. - It makes a database so it doesn't have to read all the synced files again for each scheduled scan. This got to about 1.5GB for 800GB of synced files. - All the directories I synced had been copied from one computer to the other many months ago so there were quite a few changes. Syncthing creates copies of files with the same name but different content and renames the older ones with a "syncthing-conflict-" prefix. Usually it suffices to search for these and delete them but at least you get the chance to compare and ok it manually. - I have just been starting it manually on each system to start sync but it can also be run as a daemon and autostarted at boot time with DE autostart or a systemd timer. It has already saved me quite a few instances of sneakernet! 1 Quote Link to comment Share on other sites More sharing options...
securitybreach Posted April 18, 2019 Share Posted April 18, 2019 Cool Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.