Trying to understand how to do a simple backup

[wa4chq]

OK, I just finished backing up what I wanted to save from my /home directory. I used LuckyBackup and took advantage of the user defined template feature. For me, it saved a lot of work. Since I'm still learning, there may be shortcuts for excluding things from the backup but for now, this worked. I'll continue to read more about rsync and it's options etc....

thanks for the help and support.

[securitybreach]

[wa4chq]

hmmm....the shift+insert worked if "copying from clipboard" to here but will not copy to xterm...... more research.....

[wa4chq]

Ahhhhh....just discovered xclipboard! Now if I can remember this, I think my problem is solved! To be honest, I don't think I've used xclipboard before. So I found that if xclipboard is active and I hit "save to clipboard" in luckbackup it automagically pastes the rsync command to xclipboard. I can save the rsync command paste to xterm for future use.......

[crp]

How is the rsync backup working out?

[wa4chq]

Hi crp....well, it's done well considering I'm only using it to back up certain things from my home directory. I haven't automated it since I don't have access to the internet all the time. I used LuckyBackup. One big question is where to you store the backup? I guess the best thing to do is have two copies stored in separate places. Luckily most of the stuff is not super important but it would be nice not to lose them....

[Hedon James]

a backup of anything is always good, but a backup to an external device/location is even better. If possible, you should ALWAYS backup to an external device; ideally, that device is at an external location (I haven't done this yet, as I backup my business machine and media server weekly, each needing approximately 1TB of backup. External location solutions are just too expensive, so I backup to external devices and periodically shuttle them back & forth between a 2nd location).

 

As to scheduling backups, this is done with 'cron', which is also in the LuckyBackup GUI. Go ahead and schedule it...if rsync/LuckyBackup cannot execute, it "fails quietly". In other words, it will execute when all parameters are met; but if something is amiss (in your case, perhaps lack of internet?), it will try the scheduled task, but will fail without warning. LuckyBackup offers an option to send e-mail notifications for successful task completions (if backup is successfully completed, send e-mail to myself, else do nothing), but I've never really messed with that feature. If I recall correctly, it requires a terminal-based e-mail program as a dependency, in order to use that feature.

 

Congrats on getting everything working. Now you can tweak it to your needs. You can keep doing it manually if that works for you, but the whole point of backups (for most folks) is that it's too easy to forget them. The fail-safe move, IMO, is to "set it, and forget it". Go ahead and automate with 'cron"....if backup doesn't happen, it won't be because you got busy and forgot. The only downside is that you may have a false sense of security....you may believe everything is fine when, in fact, it's failing quietly behind the scenes. So you might wanna manually double-check your backups to ensure everything is running smoothly behind the scenes. The only thing worse than "no backups" is confidently believing that you have backups that don't actually exist, IMO!

 

Point being....just sharing the knowledge that you have the option to schedule/automatic your backups, even if you don't have internet sometimes (or the backup drive isn't available, or whatever parameter may be absent). Options are ALWAYS a good thing. Choose what works for ya, buddy!

[wa4chq]

Hey hendon james....You know, after rereading my post and seeing you post, I got to wondering why I thought I needed internet to do cron. Except for having an email sent to me wouldn't a cron job just do it's thing if the laptop is on at the time of the cron job? I don't recall ever using cron so it's new to me. So far, googling hasn't been any help. Again, it's not really much stuff or very important things. thanks for the info!

[Hedon James]

I wondered the same thing Wa4chq. Internet doesn't really have anything to do with backups, unless you're backing up to Google Drive, Dropbox, OneCloud, etc... I was giving ya the benefit of the doubt! I believe I discuss the "cron" job in my tutorial, which you have obviously read. If it's not in there, let me know and I'll walk you through it. You've already worked out the "hard" part of backups...automating regular backups is the easy part!

[wa4chq]

Good morning hendon james. Yesterday, using LB I tried setting up a cron job to execute when I boot up. That worked nicely. Next I'm gonna try a cron job set to a specific time and see what it will do when I'm using the lappy at that time and when I'm not using the lappy at that time.

One thing that I'm not sure about when using rsync (LuckyBackup) is how not to duplicate what I'm backing up. For a test I created a folder called "test_backup" and put one test file in it. I ran LB and it made a backup of "test_backup" with the one file. I created "test_file_2" and put it in the "test_backup" directory and ran LB and it made a back up of "test_file_2". Reading the info window on LB it showed that there were two files in the source directory and it only transferred one. OK, sounds good. But when I tried a directory that I've already backed up called "pictures" running in test mode, it appeared to me like it was backing up everything again. In other words I thought since the source and destination contained the same thing, nothing would be backed up or transferred. Am I making sense? UPDATE: I'm gonna reread your tutorial because I know what I'm looking for is in there.

Thanks.

Edited November 30, 2016 by wa4chq

[Hedon James]

Not sure what you're asking, but rsync is an "incremental" backup utility. It only copies over "changed" files; if there's no change, there's no copy. You CAN however, backup a file to 2 different locations as part of 2 different backups. For instance, perhaps you backup ALL media automatically to an external drive, and also backup all music files (part of media) manually to an mp3 device. When you add a new music file to your collection, the automated backup will copy over as part of your "media backup" operation, but will not copy to your "music backup" until manually invoked. But once invoked manually, your new music file will also copy over (a 2nd time) to your music destination as part of your manual "music backup" operation. Is that what you're asking? Or something else?

[securitybreach]

Not sure what you're asking, but rsync is an "incremental" backup utility. It only copies over "changed" files; if there's no change, there's no copy.

 

Exactly

[wa4chq]

OK....Here is the print out of a dry run. The directory "pictures" is already on my destination.

 

 

=====================================

execution of task : pictures, starting

Source : /home/wa4chq/pictures/

Destination : /media/pictures/

sending incremental file list

blah blah blah etc....

 

Number of files: 548 (reg: 536, dir: 12)

Number of created files: 121 (reg: 118, dir: 3)

Number of deleted files: 0

Number of regular files transferred: 118

Total file size: 3.53G bytes

Total transferred file size: 2.32G bytes

Literal data: 0 bytes

Matched data: 0 bytes

File list size: 0

File list generation time: 0.001 seconds

File list transfer time: 0.000 seconds

Total bytes sent: 14.22K

Total bytes received: 389

 

sent 14.22K bytes received 389 bytes 29.22K bytes/sec

total size is 3.53G speedup is 241,665.41 (DRY RUN)

 

execution of task : pictures, finished =====================================

 

Here is the command LuckyBackup sent:

 

rsync -h --progress --stats -r -tgo -p -l -D --update --exclude=**/*cache*/ --exclude=**/*Cache*/ --exclude=**/lost+found*/ --exclude=**/*Trash*/ --exclude=**/*trash*/ --exclude=**/.gvfs/ --exclude=/pictures/hammocking/video/ --exclude=/pictures/eagle/*.AVI --exclude=/pictures/hammocking/trayfoot_paynesrun/*.AVI --exclude=/pictures/latest_hammock/*.AVI --exclude=/pictures/video/ /home/wa4chq/pictures/ /media/pictures/

 

 

Am I reading this wrong? It looks like that it is replacing what is already there.

Thanks....

Edited November 30, 2016 by wa4chq

[amenditman]

It looks to be creating 121 new files out of 548 total.

That seems like an incremental backup not a recopy of the entire thing.

[Hedon James]

looks like you have 118 NEW files to be backed up, in 3 different NEW directories/folders. so you have either:

 

1. removed an exclusion from your list of exclusions (allowing 3 new folders/directories to be backed up)

2. added 3 new folder/directories to your main hard drive (source) since last backup (this would include restructuring your folder/file hierarachy)

3. have changed metadata of the 118 files copied, indicating to rsync that the file has been "updated" and should be copied over.

 

As an aside, I'm wondering if you're "backing up source inside destination" or "synchronizing source & destination". each has a slightly different use case....

 

When "backing up source inside destination" all files on "source" will be copied to "destination". When "source" file changes, rsync will copy changed file to "destination". However, if "source" file is deleted, it will still remain on "destination". IMO, this is how a backup should work, so you'll never lose an important file.

 

Conversely, when "synchronizing source & destination", if the "source" file is deleted, it will also be deleted from "destination". This is handy for keeping my music & video libraries consistent across devices. I have my music automatically backed up to my "DataBAK" drive, manually to an external drive in my basement studio, and manually with several android phones, mp3 devices, and flash drives for car usb. If I rip an Esteban cd and a Disturbed cd, they both get backed up automatically. Later, I'll manually back then up (synchronize them) to my other sources. Even later, perhaps I'll decide that Esteban makes me sleepy and I'll remove it from my master drive. It's kinda nice to have my other devices manually synchronized, so Esteban gets removed from all my devices as I keep them updated.

 

See the difference? Different uses for different purposes. I'm wondering how your backup is set? Perhaps you have deleted 118 files and 3 folders from your "source" and you're "synchronizing" to your destination. That could be a scenario #4, FWIW...

[wa4chq]

thanks you guys..... I don't have time to look at all of this now but will ASAP. It is likely I set up the original run wrong (now how could I do that?? ) So to delete the folder and do it all again won't be a problem. Have a great evening!

[wa4chq]

Good morning all.....I think I'm getting the hang of this backing up thang. Using the tutorial and making several trial runs, I'm seeing now what I think it's supposed to do. It's amazing how reading directions or tutorials can make a difference.....and having all the help from you guys is a big plus.

So, this morning I deleted the directory "pictures" from my flash drive. My source is:

/home/wa4chq/pictures --- and destination is:

/home/storage (flash drive)

Running LuckyBackup, the info window said I transferred 479 files (2.62GB) and no errors found. If nothing changes in /home/wa4chq/pictures and I run LB again nothing should transfer but if I add something to it, only new things will be transferred. I have a few more tests to make.....

Edited December 2, 2016 by wa4chq

[securitybreach]

Great

[jolphil]

Hi,

I have been reading this post with great interest.Glad you got it working wa4ghq. I also have been trying, but still don't quite get it yet. Hedon James, Great tutorial for your first try. Hedon, I understand you

saving your music files.But for me,Lucky backup is a lot more complicated than just "copy" "paste" of my home folder. I understand it is not a cron schedule but that is not important for me.As to not using extra cpu cycles, I just select "do not overwrite" and it copies just the new stuff.What am I missing here? I can understand if I wanted an image or a synched directory but in my case I just want to save my home folder files. Again, thanks for the information,just goes to prove you can learn something every day.

jolphil

[abarbarian]

Hi,

I have been reading this post with great interest.Glad you got it working wa4ghq. I also have been trying, but still don't quite get it yet. Hedon James, Great tutorial for your first try. Hedon, I understand you

saving your music files.But for me,Lucky backup is a lot more complicated than just "copy" "paste" of my home folder. I understand it is not a cron schedule but that is not important for me.As to not using extra cpu cycles, I just select "do not overwrite" and it copies just the new stuff.What am I missing here? I can understand if I wanted an image or a synched directory but in my case I just want to save my home folder files. Again, thanks for the information,just goes to prove you can learn something every day.

jolphil

 

https://wiki.archlin...index.php/Rsync

 

You might find some useful info here.

 

or try this

 

Edited December 6, 2016 by abarbarian

[securitybreach]

https://wiki.archlin...index.php/Rsync

 

You might find some useful info here.

 

[Hedon James]

Hi,

I have been reading this post with great interest.Glad you got it working wa4ghq. I also have been trying, but still don't quite get it yet. Hedon James, Great tutorial for your first try. Hedon, I understand you

saving your music files.But for me,Lucky backup is a lot more complicated than just "copy" "paste" of my home folder. I understand it is not a cron schedule but that is not important for me.As to not using extra cpu cycles, I just select "do not overwrite" and it copies just the new stuff.What am I missing here? I can understand if I wanted an image or a synched directory but in my case I just want to save my home folder files. Again, thanks for the information,just goes to prove you can learn something every day.

jolphil

 

Jolphil, LuckyBackup is just a GUI front end for the command-line tool rsync. rsync is an awesome tool that makes recursive copies of any INCREMENTAL changes in file or directory structures. it is absolutely the tool you want, based on your description above. If you type 'man rsync' into a terminal, you will see the manual of the rsync tool, as well as all the available flags, switches, options, and use cases.

 

LuckyBackup is just a GUI for the majority of us, allowing us to select these options, flags & switches with GUI drop-down menus, checkboxes, etc... There are other graphical front-end GUIs for rsync, such as grsync (referenced above by abarbarian), but I have no experience with them. My experience is limited to LuckyBackup. Looks like abarbarians experience is with grsync, while Security Breach's experience is with the command-line tool. Don't be confused by this, as it is just 3 different approaches to using the SAME TOOL! And I'm not advocating that LuckyBackup is any better, or worse, than grsync or rsync...it's just 3 different ways to do the same thing!

 

With that said, based on your description of what you say you're looking for, rsync is the right tool. Of course you could copy/paste your entire home directory and tada....you have a backup. However, if you change files in your SOURCE home directory and attempt to refresh your backup with a follow-up copy/paste action of your home directory, your "new" files and directories will be copied over without issues, but you will also be presented with numerous requests for intervention in the form of "an existing copy of 'filename' has been found. do you want to overwrite, rename, or skip". Rsync addresses this issue for you by comparing source & destination files, and then only copying over the new and/or changed files. As an added bonus, for various reasons and due to the way it works, rsync is quicker than copy/paste. Is any of this helpful, or am I just restating what you already knew?

 

As for my example with the music backup, this is easily adaptable for your Home directory. Instead of backing up Music directory, go up one level and choose your entire Home directory as SOURCE, then proceed from there, as written in the tutorial, substituting 'HOME" for 'MUSIC" in each instance. And you don't have to CRON the job if you don't want to....it's just another option in the LuckyBackup GUI and once I verify things are working as expected, I prefer the automation. Options...

 

And if you decide later on that grsync seems more streamlined and easier, or if you attend SBs command-line seminar and agree that CLI is the way to go, by all means, do it the way you prefer to get things done. I'm just trying to help others get their foot in the door, so to speak. Backups are near & dear to me due to some unpleasant life experiences, so I'm a bit vigilant about it now. But it's not important HOW you do it, it's only important that you DO! JMO...

[abarbarian]

Hi,

I have been reading this post with great interest.Glad you got it working wa4ghq. I also have been trying, but still don't quite get it yet. Hedon James, Great tutorial for your first try. Hedon, I understand you

saving your music files.But for me,Lucky backup is a lot more complicated than just "copy" "paste" of my home folder. I understand it is not a cron schedule but that is not important for me.As to not using extra cpu cycles, I just select "do not overwrite" and it copies just the new stuff.What am I missing here? I can understand if I wanted an image or a synched directory but in my case I just want to save my home folder files. Again, thanks for the information,just goes to prove you can learn something every day.

jolphil

 

There are other graphical front-end GUIs for rsync, such as grsync (referenced above by abarbarian),

 

 

 

 

 

I have never used grsync but I have used SB's cli method with a few tweaks. I came across grsync in this thread where VT praises it's use. SSo as always VT is to blame

 

http://forums.scotsnewsletter.com/index.php?showtopic=62396&hl=

[Hedon James]

Hi,

I have been reading this post with great interest.Glad you got it working wa4ghq. I also have been trying, but still don't quite get it yet. Hedon James, Great tutorial for your first try. Hedon, I understand you

saving your music files.But for me,Lucky backup is a lot more complicated than just "copy" "paste" of my home folder. I understand it is not a cron schedule but that is not important for me.As to not using extra cpu cycles, I just select "do not overwrite" and it copies just the new stuff.What am I missing here? I can understand if I wanted an image or a synched directory but in my case I just want to save my home folder files. Again, thanks for the information,just goes to prove you can learn something every day.

jolphil

 

There are other graphical front-end GUIs for rsync, such as grsync (referenced above by abarbarian),

 

 

 

 

 

I have never used grsync but I have used SB's cli method with a few tweaks. I came across grsync in this thread where VT praises it's use. SSo as always VT is to blame

 

http://forums.scotsn...topic=62396&hl=

 

D*** Slacker! He's corrupted me once or twice too!

[V.T. Eric Layton]

Heh!

 

Actually, that's not a Slacker thing. If I was to do it the Slackware way, I'd do it the way I used to... on the command line w/ rsync. However, since I use Porteus (from a CD) to boot and do syncs, I set a small file on my main drive that stores Porteus' settings (including the settings for grsync). This way every time I boot from the Porteus CD, it looks for that file and then starts up with all my settings intact. I just have to click the "start" button in grsync and away it goes. I only have to change between the saved settings with a pulldown window in grsync to perform all three of the syncs that I do every Sunday.

 

Now, the REAL easy way is to set it up like Josh... he uses a cron job (command line scheduler app) to set a time every day (or numerous times a day, if he wants) to make an rsync run on his specified partitions. Set it and forget it. That's the REAL Slackware way. However, Josh never powers down his systems, so that works well. Me, I power down whenever I'm not sitting in front of the soul-sucking boxes. That makes auto-scheduling problematic for me. That's why I just manually do my sync each Sunday.

 

Ain't Linux FUN?!

[wa4chq]

I boot from the Porteus CD,

Hey Eric..... would booting from a thumb drive be faster? Just wondering.....

[V.T. Eric Layton]

I boot from the Porteus CD,

Hey Eric..... would booting from a thumb drive be faster? Just wondering.....

 

Possibly, but my older system (ericsbane06) couldn't boot from USB, so I just got used to doing the CD thing. It loads fast, though. I dump the Porteus OS onto RAM in about 30-45 seconds. Once it's running from RAM, it's greased lightning.

[wa4chq]

Hey Eric....sounds good! Porteus is a great mini OS.

[V.T. Eric Layton]

Yes, I actually preferred SLAX for many years, but started using Porteus because of the really neat disk manager app that it had in it that would pull up all the SMART hdd data to show the health of your drive. For some darn reason, though, Porteus left the disk manager out of the 3.x version. That's OK. I still have my 2.x version on CD and thumbdrive.

[sunrat]

Me, I power down whenever I'm not sitting in front of the soul-sucking boxes. That makes auto-scheduling problematic for me. That's why I just manually do my sync each Sunday.

You could still automate by installing anacron. It will check for cron jobs that were scheduled during power-off times and run them after you boot up next.