Jump to content


Syncing between two computers - Syncthing?


  • Please log in to reply
17 replies to this topic

#1 ONLINE   sunrat

sunrat

    Thread Kahuna

  • Forum Moderators
  • 5,946 posts

Posted 28 November 2017 - 09:20 AM

I have now got 2 computers running having brought my old one out of retirement. The old one (bedroom) is mainly for everyday use, and the newer one (loungeroom) is mainly for music production although it's bound to be used for games and watching video as it has my new Sony 43" 4K TV as its monitor.
I need to sync files both ways between them as either may be used to add files, mainly to my music collection which currently is about 650GB, but also for backups of other data such as recording production sessions. Both are on the same network via wifi. I considered rsync which I have always used for backups to external drives but it seems more suited for one way transfers and needs ssh or similar over network.
Then a search revealed Syncthing which seems like it could be suitable. Setup looks easy with only firewall settings to consider. The FAQ mentions things that are not synced which hopefully are not an issue:

Quote

The following are not synchronized;File or Directory Owners and Groups (not preserved)
Directory Modification Times (not preserved)

So my questions are - has anyone experience using Syncthing? Does it look like the best tool for the job? It's open source and available in repos for Debian and Arch.

https://syncthing.net/
registered Linux user number 324659  ||    The importance of Reading The *Fine* Manual! :D
Posted ImagePosted ImagePosted ImagePosted Image
For the things we have to learn before we can do them, we learn by doing them.

#2 ONLINE   Hedon James

Hedon James

    Topic Cop

  • Members
  • PipPipPipPipPipPipPip
  • 922 posts

Posted 28 November 2017 - 10:42 AM

I haven't used Syncthing, but everything I've read suggests that it would be a good tool for the job.  FWIW, you are correct, IMO, that rsync is best suited for 1-way backups.  It sounds like you might also be interested in Unison, which is a 2-way synchronization designed to "mirror" multiple locations.  Is this what you're looking to accomplish:

https://www.howtofor...-debian-jessie/

Assuming Unison also does the job, someone else will have to chime in on the pros vs. cons of each.  I believe Syncthing is multi-platform, including Android, so that could be a plus, as I'm not sure Unison is available for cell phone platforms.

#3 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 28 November 2017 - 11:21 AM

I have a daily cron job that runs syncs data between drives using rsync

https://wiki.archlin...utomated_backup

Basically it copies over anything that is on the host but not on the backup so that way both drives have the same exact data. This should work just fine between computers as well. There is no reason to run an additional application as you can set this as a cronjob and it will run automatically. Example of mine:  http://forums.scotsn...12
Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984

#4 ONLINE   sunrat

sunrat

    Thread Kahuna

  • Forum Moderators
  • 5,946 posts

Posted 28 November 2017 - 06:02 PM

Thanks for your advice but it still looks like Syncthing will be the best for my use case. I was mainly just wondering if anyone had experience with it.
I will try it and let you know how it goes.

For phone data swapping I use KDE Connect. Perfect for what I need there.
registered Linux user number 324659  ||    The importance of Reading The *Fine* Manual! :D
Posted ImagePosted ImagePosted ImagePosted Image
For the things we have to learn before we can do them, we learn by doing them.

#5 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 28 November 2017 - 06:05 PM

Cool, let us know what you ended up using.
Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984

#6 ONLINE   sunrat

sunrat

    Thread Kahuna

  • Forum Moderators
  • 5,946 posts

Posted 06 April 2019 - 09:21 AM

It took a little time to get around to it, but I finally set up Syncthing between the 2 computers. Did some reading of the docs, watched a couple of videos and then installed on both computers and had it up and running in about 10 minutes! So far only added the default test folders so I'll have to get up some more nerve to sync the actual hundreds of gigabytes that exist on both computers.

This was prompted by a minor installation catastrophe on my production computer where I somehow wiped out the whole partition table on the SSD. No data lost as that's on separate drives and backed up as well, but I wiped out about 7 installed operating systems most of which needed some spring cleaning due to 3 years of experimental program installs and such. The most painful bit was hours reinstalling and tweaking Win10 and still have my favourite games to go. I'd just installed Forza Horizon 3 which took ages. :(

Edit:  It seems I didn't explain very well. The catastrophe was prior to installing Syncthing and caused by an OS installation failure as explained below. Syncthing works perfectly so far.
registered Linux user number 324659  ||    The importance of Reading The *Fine* Manual! :D
Posted ImagePosted ImagePosted ImagePosted Image
For the things we have to learn before we can do them, we learn by doing them.

#7 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 06 April 2019 - 09:22 AM

View Postsunrat, on 06 April 2019 - 09:21 AM, said:

It took a little time to get around to it, but I finally set up Syncthing between the 2 computers. Did some reading of the docs, watched a couple of videos and then installed on both computers and had it up and running in about 10 minutes! So far only added the default test folders so I'll have to get up some more nerve to sync the actual hundreds of gigabytes that exist on both computers.

This was prompted by a minor installation catastrophe on my production computer where I somehow wiped out the whole partition table on the SSD. No data lost as that's on separate drives and backed up as well, but I wiped out about 7 installed operating systems most of which needed some spring cleaning due to 3 years of experimental program installs and such. The most painful bit was hours reinstalling and tweaking Win10 and still have my favourite games to go. I'd just installed Forza Horizon 3 which took ages. :(

Wow, I guess that didn't work as intended ;)
Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984

#8 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 06 April 2019 - 09:28 AM

So you synced your working drive with a blank one and it synced the wrong way?

It would be a lot easier to simply add this to a daily/hourly cron job:

 rsync -ar --delete /Source /Target

Quote

-a
indicates that files should be archived, meaning that most of their characteristics are preserved (but not ACLs, hard links or extended attributes such as capabilities)

--delete
means files deleted on the source are to be deleted on the backup as well

-r
recurse into directories


This works flawlessly and I have been using it for probably a decade now. https://wiki.archlin...index.php/Rsync

Quote

Cerberus :: ~ » crontab -l
00 15 * * * rsync -ar --delete /Various/comhack/Downloads /Various/comhack/Miscellaenous /Various
00 16 * * * rsync -ar --delete /MEDIA/Documents /Various/comhack/
00 17 * * * rsync -ar --delete /Backup2/Movies/ /Various/Moves/
00 17 * * * rsync -ar --delete /MEDIA/Other /Backup/

Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984

#9 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 06 April 2019 - 09:31 AM

Sorry, I didn't include a link to the important part for you...
Automated backup with SSH
Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984

#10 ONLINE   sunrat

sunrat

    Thread Kahuna

  • Forum Moderators
  • 5,946 posts

Posted 06 April 2019 - 09:31 AM

The catastrophe was before Syncthing in case I was unclear. Some glitch while installing Debian Buster I think but compounded by some misconfiguration of the partitions which I had already had minor issues with before. It destroyed the partition table rather than wiping the drive.

I installed Buster successfully now on the new setup.
It felt good to start fresh with a clean slate.
Syncthing so far appears to be working perfectly and browsing through the manual made me aware of just how powerful and configurable it is.
The idea is to have both computers with identical data so I can change something on either one and have it sync to the other. Not to just have one as backup for the other. That's harder to do with rsync.
registered Linux user number 324659  ||    The importance of Reading The *Fine* Manual! :D
Posted ImagePosted ImagePosted ImagePosted Image
For the things we have to learn before we can do them, we learn by doing them.

#11 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 06 April 2019 - 09:33 AM

View Postsunrat, on 06 April 2019 - 09:31 AM, said:

The catastrophe was before Syncthing in case I was unclear. Some glitch while installing Debian Buster I think but compounded by some misconfiguration of the partitions which I had already had minor issues with before. It destroyed the partition table rather than wiping the drive.

I installed Buster successfully now on the new setup.
It felt good to start fresh with a clean slate.
Syncthing so far appears to be working perfectly and browsing through the manual made me aware of just how powerful and configurable it is.

Ah ok, I may check it out.
Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984

#12 ONLINE   sunrat

sunrat

    Thread Kahuna

  • Forum Moderators
  • 5,946 posts

Posted 06 April 2019 - 09:39 AM

It's like having a Dropbox sync folder on 2 devices but without having to upload to a Dropbox server in between.
registered Linux user number 324659  ||    The importance of Reading The *Fine* Manual! :D
Posted ImagePosted ImagePosted ImagePosted Image
For the things we have to learn before we can do them, we learn by doing them.

#13 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 06 April 2019 - 10:22 AM

View Postsunrat, on 06 April 2019 - 09:39 AM, said:

It's like having a Dropbox sync folder on 2 devices but without having to upload to a Dropbox server in between.

Well I host my own cloud storage using Nextcloud.
Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984

#14 ONLINE   sunrat

sunrat

    Thread Kahuna

  • Forum Moderators
  • 5,946 posts

Posted 06 April 2019 - 06:11 PM

View Postsecuritybreach, on 06 April 2019 - 10:22 AM, said:

View Postsunrat, on 06 April 2019 - 09:39 AM, said:

It's like having a Dropbox sync folder on 2 devices but without having to upload to a Dropbox server in between.

Well I host my own cloud storage using Nextcloud.

You wouldn't be needing Syncthing then. :) Having cloud storage is a step too far for me as I never need access to my data when I'm not home.
It can be used securely over the internet as it uses strong encryption for blocks of data. It can also be used to sync any number of computers which has the added advantage of speeding up the process similar to bittorrent sharing.
Available for Linux, Windows, Mac, Android, BSD, Solaris etc.

Check it out: https://syncthing.net/
registered Linux user number 324659  ||    The importance of Reading The *Fine* Manual! :D
Posted ImagePosted ImagePosted ImagePosted Image
For the things we have to learn before we can do them, we learn by doing them.

#15 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 07 April 2019 - 12:21 AM

View Postsunrat, on 06 April 2019 - 06:11 PM, said:

View Postsecuritybreach, on 06 April 2019 - 10:22 AM, said:

View Postsunrat, on 06 April 2019 - 09:39 AM, said:

It's like having a Dropbox sync folder on 2 devices but without having to upload to a Dropbox server in between.

Well I host my own cloud storage using Nextcloud.

You wouldn't be needing Syncthing then. :) Having cloud storage is a step too far for me as I never need access to my data when I'm not home.

And that is why I use rsync for local stuff..
Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984

#16 ONLINE   sunrat

sunrat

    Thread Kahuna

  • Forum Moderators
  • 5,946 posts

Posted 07 April 2019 - 06:21 PM

View Postsecuritybreach, on 07 April 2019 - 12:21 AM, said:

And that is why I use rsync for local stuff..

I use rsync to back up my home folder and it's perfect for that. But I wouldn't know how to set it up to automatically do a 2-way sync between 2 computers, or even if that's possible.
I did the full sync of my 132GB data partitions yesterday and it went almost without a hitch. I had to set the .firefox and .thunderbird profile folders to be excluded as I used them while the sync was happening and they both create temp files which confuse the process. Otherwise all was perfect. It did take a few hours to do it as syncthing does a hash of every file on each computer. Subsequent syncs are almost instant though.
I'll score it 9/10. Unbelievably easy to use after you learn the basics. :thumbsup: B)
registered Linux user number 324659  ||    The importance of Reading The *Fine* Manual! :D
Posted ImagePosted ImagePosted ImagePosted Image
For the things we have to learn before we can do them, we learn by doing them.

#17 ONLINE   sunrat

sunrat

    Thread Kahuna

  • Forum Moderators
  • 5,946 posts

Posted 17 April 2019 - 09:13 PM

Another 10 days of using Syncthing and it's definitely a winner. I synced up my 680GB music collection which went well.

A couple of minor things to be aware of:
- It tends to copy things faster than deleting other things so be careful you have enough disk space. I was moving many GB of files from one directory to another on the same drive which were simultaneously syncing to the other system and its disk got to 97% full. Worked out ok though.
- It makes a database so it doesn't have to read all the synced files again for each scheduled scan. This got to about 1.5GB for 800GB of synced files.
- All the directories I synced had been copied from one computer to the other many months ago so there were quite a few changes. Syncthing creates copies of files with the same name but different content and renames the older ones with a "syncthing-conflict-" prefix. Usually it suffices to search for these and delete them but at least you get the chance to compare and ok it manually.
- I have just been starting it manually on each system to start sync but it can also be run as a daemon and autostarted at boot time with DE autostart or a systemd timer.

It has already saved me quite a few instances of sneakernet! :thumbsup: B)
registered Linux user number 324659  ||    The importance of Reading The *Fine* Manual! :D
Posted ImagePosted ImagePosted ImagePosted Image
For the things we have to learn before we can do them, we learn by doing them.

#18 OFFLINE   securitybreach

securitybreach

    CLI Phreak

  • Forum Admins
  • 24,658 posts

Posted 17 April 2019 - 09:15 PM

Cool :thumbsup:
Posted ImagePosted Image
Configs/PGP Key/comhack π

"Do you begin to see, then, what kind of world we are creating? It is the exact opposite of the stupid hedonistic Utopias that the old reformers imagined. A world of fear and treachery and torment, a world of trampling and being trampled upon, a world which will grow not less but more merciless as it refines itself. Progress in our world will be progress toward more pain." -George Orwell, 1984




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users