Jump to content

Full System Backup with rsync


Recommended Posts

abarbarian

https://wiki.archlin...ckup_with_rsync

 

Well after me near disaster I thought it best to do a backup. :cool:

 

So I followed the ARCH WIKI made a script called " phew.sh " placed it in a newly made folder called " ~/Scripts " (/home/me/Scripts) and followed the instructions,

 

While the system is running, open up a terminal and run (as root):

# ~/Scripts/phew.sh /some/destination

So I was quite surprised that me terminal tells me there is no " ~/Scripts/phew.sh ".

However " # /home/me/Scripts/phew.sh /some/destination " does exist and the backup is now hopefully running.

 

So am I doing something wrong with " # ~/Scripts/phew.sh /some/destination " or is the Arch Wiki wrong ?

:breakfast:

Edited by abarbarian
Link to post
Share on other sites
  • Replies 71
  • Created
  • Last Reply

Top Posters In This Topic

  • securitybreach

    21

  • V.T. Eric Layton

    16

  • abarbarian

    14

  • Hedon James

    6

Top Posters In This Topic

Popular Posts

I think that today's the first time I've looked in on that, "The Restaurant at the Edge of the Universe." I've got tunnel vision for BATL when I come here.

Backstory: http://forlackofabet...mic.com/?id=230

Why make things so difficult? Shut your computer off. Reboot with your Porteus flash drive stuck in that hole in the front. Once Porteus has booted, open a terminal, su to root, type "grsync" w/o the

amenditman

So I was quite surprised that me terminal tells me there is no " ~/Scripts/phew.sh ".

However " # /home/me/Scripts/phew.sh /some/destination " does exist and the backup is now hopefully running.

 

So am I doing something wrong with " # ~/Scripts/phew.sh /some/destination " or is the Arch Wiki wrong ?

:breakfast:

 

The ~/ is the same as your user /home directory. The only thing you did differently is run the command as root.

 

Did you change the permissions on the script to be able to run it as user.

Usually, <chmod a+x phew.sh> - executable by any.

If you don't want just anybody to be able to run the script, but you want to run it as user, make sure who is the owner. root or your user.

If root is the owner (you created the script as root) then as root, <chown user phew.sh> and put your username as user.

Then <chmod 750 /home/me/Scripts/phew.sh> will make it so that the owner (you) can read, write, execute; the group can read, execute; and everyone else can go slice bread.

Edited by amenditman
Link to post
Share on other sites
securitybreach

Well for one you made the directory in ~/Scripts as root so therefore the folder is in /root/Scripts not /home/user/Script.

 

So yes, the wiki entry is mistaken as the command should of been:

# /home/user/Scripts/phew.sh /some/destination

 

instead of:

# ~/Scripts/phew.sh /some/destination

 

since you made the directory in /home/user not /root/

Link to post
Share on other sites
securitybreach

Also, if you are just backing up certain folders; I would use this instead: https://wiki.archlin...utomated_backup

 

I have been using that for years and it works beautifully:

comhack@Cerberus ~ % su
Password:
root@Cerberus /home/comhack # crontab -l
00 15 * * * rsync -ar --delete /home/comhack/Videos /home/comhack/Music /home/comhack/Android /home/comhack/Documents /home/comhack/Downloads /home/comhack/Pictures /home/comhack/Shared /MEDIA &> /dev/null

Link to post
Share on other sites
abarbarian

Thanks Josh I looked at your script before in other threads.I'm after a full backup though at the moment.

 

Guys I followed the wiki I pretty sure I did. Anyway I had another run through and did,

 

$ cd ~/Scripts
$ nano phew.sh

 

An copy/pasted the script given with a few alterations,

 

All of the above is as USER right.

Then I followed the wiki,

 

$ chmod +x phew.sh

 

which gives me

 

 

$ chmod +x phew.sh
chmod: changing permissions of 'phew.sh': Operation not permitted

 

 

Which means that the wiki is wrong and this " $ chmod +x phew.sh " will not work.

 

Am I right on this or am I going crazy.

:breakfast:

Edited by abarbarian
Link to post
Share on other sites
securitybreach

The command chmod +x filename, basically just makes it executable and it will work if you have the correct permissions to do it. To check permissions, run:

ls -al ~/Scripts/phew.sh

  • Like 1
Link to post
Share on other sites
securitybreach

You have to be root to run chmod, even on a script that is yours.

Should be:

#chmod +x phew.sh

 

You do not have to be root to use chmod. I just tested it with one of my scripts. That said, it would bypass your issue if you did it as root.

Link to post
Share on other sites
abarbarian

Yup I understand about permissions somewhat. I have had the script running but it refuses to exclude any folders and files.

 

$ cd ~/Scripts
$ nano backup.sh
$ chmod +x backup.sh

 

 

I tried the three commands after deleting the script and starting again and it gives me ," -rwxr-xr-x 1 bloodaxe user " So owner group and others can all execute the script.

 

So it look like I need to apologise to the Arch Wiki as it looks like I made a boo boo. :worthy:

 

However the script is not excluding folders and files. I'll have to have a look at it tomorrow as all this stress has worn me out. Ta fer yer patience guys. :breakfast:

  • Like 1
Link to post
Share on other sites
V.T. Eric Layton

Why make things so difficult? Shut your computer off. Reboot with your Porteus flash drive stuck in that hole in the front. Once Porteus has booted, open a terminal, su to root, type "grsync" w/o the "".

 

When that fabulous little graphic front end for rsync opens, set the source and the destination properly .

 

For example:

 

source -- /dev/sda1/

 

destination -- /dev/sdb1

 

*Note: if you want a true mirrored backup, you must have that all-important following / you see above in red. If you forget it, you'll get a mirror within a folder called /dev/sda1 on the sdb1 partition. That won't be bootable. However, it would still be a valid mirror backup that you could sync back to the original partition. I prefer 100% bootable root partition mirrors myself.

 

Anywho, you'll need to make sure you have some of the options in grsync set correctly, too. You'll want to copy all hard links and sym-links as-is. There are some other critical options you'll need, but I'd have to boot into Porteus to check for you. Let me know if you want me to do that.

 

The first rsync will take a while, depending on your system's speed. The latter rsyncs will go much faster because of rsync's characteristic ability to copy over only changed files, rather than everything again. Rsync is a very cool little tool. I rsync my Slackware /(root) and /home every Sunday... manually, not with cron.

  • Like 2
Link to post
Share on other sites
securitybreach

Why make things so difficult. Shut your computer off. Reboot with your Porteus flash drive stuck in that hole in the front. Once Porteus has booted, open a terminal, su to root, type "grsync" w/o the "".

 

When that fabulous little graphic front end for rsync opens, set the source and the destination properly (your destination partition).

 

For example: source -- /dev/sda1/ | destination -- /dev/sdb1

 

*Note: if you want a true mirrored backup, you must have that all-important following / you see above in red. If you forget it, you'll get a mirror within a folder called /dev/sda1 on the sdb1 partition. That won't be bootable. However, it would still be a valid mirror backup that you could sync back to the original partition. I prefer 100% bootable root partition mirrors myself.

 

Anywho, you'll need to make sure you have some of the options in grsync set correctly, too. You'll want to copy all hard links and sym-links as-is. There are some other critical options you'll need, but I'd have to boot into Porteus to check for you. Let me know if you want me to do that.

 

The first rsync will take a while, depending on your system's speed. The latter rsyncs will go much faster because of rsync's characteristic ability to copy over only changed files, rather than everything again. Rsync is a very cool little tool. I rsync my Slackware /(root) and /home every Sunday... manually, not with cron.

 

Hmm, that sounds interesting.... So what requirements are needed to do that? A partition the size of your / directory? And does it just copy the / partition or will it try to copy the mounts as well: /home /MEDIA /backup etc.?

Link to post
Share on other sites
V.T. Eric Layton

I don't remember all the settings. I'd have to boot into Porteus and start up grsync to let you know. I'll do it tomorrow. I'm outta' here shortly. Anyway, it creates a 100% perfect mirror of my root and home partitions. It is 100% bootable image of my primary Slackware installation. The partitions that I'm rsync'ing to are the exact same size as the originals. They don't have to be, though. They just have to be big enough.

 

For my /(root):

 

source: /dev/sda1

 

destination: /dev/sda5

 

For my /home:

 

source: /dev/sda2/

 

destination: /dev/sda6

 

In my case, I'm sync-ing to the same drive. For a true backup, you'd want to sync to a separate drive, of course.

 

The only thing I have to do to make the mirror bootable is adjust the /(root) and /home entries in the fstab.

 

That's it. Easy-peasy! :)

 

Be patient. I'll post the complete grsync settings tomorrow sometime.

 

G'night.

 

AH! I just remembered. I already posted how I did this here --> http://forums.scotsnewsletter.com/index.php?showtopic=60642&hl=grsync

 

Have fun! :)

  • Like 1
Link to post
Share on other sites
securitybreach

Well the reason I ask is because I would only want to clone my / partition as the other partitions mounted under / are ~4tb in size:

 

P92hHOO.png

Link to post
Share on other sites
securitybreach

Never mind, I get it... it only clones the partition you selected so it wouldn't clone the other mount points? Right?

Link to post
Share on other sites
securitybreach

Ok it worked beautifully but grsync is not installed by default and you must install it on Porteus. It took me about 15 minutes to copy 16gb, not bad :thumbsup:

 

/dev/sda2		46G 16G 29G 35% /
/dev/sdb3		46G 16G 29G 35% /mnt/usb

 

I am not clear about your last part though:

- After mirroring completed, to make the mirror bootable as the original, fstab must be edited to correct for source/destination partition number discrepancies
Link to post
Share on other sites
V.T. Eric Layton

My primary Slackware fstab looks like this:

 

/dev/sda7	    swap			 swap	    defaults		 0   0
/dev/sda1	    /			    ext3	    defaults		 1   1
/dev/sda2	    /home		    ext3	    defaults		 1   2

/dev/sr0		 /mnt/dvd_sr0	 auto	    noauto,user,ro  0   0
/dev/sr1		 /mnt/dvd_sr1	 auto	    noauto,user,ro  0   0
/dev/fd0		 /mnt/floppy	  auto	    rw,noauto,user,sync	 0   0
/dev/sdd4	    /mnt/zip		 vfat	    rw,noauto,user,sync	 0   0
/dev/sde1	    /mnt/usb		 auto	    rw,noauto,user,sync	 0   0

/dev/sda8	    /home/vtel57/vtel57_archives   ext2	 defaults	 0   2
/dev/sdc1	    /home/vtel57/vtel57_backups    ext2	 defaults	 0   2
/dev/sdc7	    /home/vtel57/vtel57_common	 vfat	 rw,gid=users,uid=vtel57	 0   0

devpts		   /dev/pts		 devpts	  gid=5,mode=620   0   0
proc			 /proc		    proc	    defaults		 0   0
tmpfs		    /dev/shm		 tmpfs	   defaults		 0   0

 

On the newly mirrored partitions, / and /home are not sda1 and sda2. They are sda5 and sda6, so I have to make that change in the fstab in order to make the mirrored Slackware bootable.

 

And that's correct about the mounts. Rsync doesn't care about the active mounts. It only mirrors the source partition itself.

 

You had to install grsync in Porteus? Hmm... I don't remember doing that. How old is your Porteus version?

 

Oh, and when you mirror the next time, it'll be a lot quicker because rsync will only copy the changed items. :)

Link to post
Share on other sites
sunrat

The easiest way to exclude files from rsync is to list them in an exclude file and specify it in the command eg - my music backup:-

rsync -av --delete --exclude-from=/mnt/music/rsync-exclude /mnt/music/ /mnt/backup1/Music-sdc

And the rsync-exclude file:-

lost+found
.Trash-1000

  • Like 2
Link to post
Share on other sites
securitybreach

You had to install grsync in Porteus? Hmm... I don't remember doing that. How old is your Porteus version?

I used Porteus-v2.0-x86_64.iso.

Link to post
Share on other sites
securitybreach

The easiest way to exclude files from rsync is to list them in an exclude file and specify it in the command eg.

rsync -avz --exclude-from 'exclude-list.txt' source/ destination/

 

I am pretty familiar with rsync, I had just never used Grsync with Porteus ;)

Link to post
Share on other sites
V.T. Eric Layton

That may be the difference. I'm using Porteus-XFCE-v1.2-i486. Grsync is a Gnome app, so it comes with the Xfce version of Porteus, I guess. :)

Link to post
Share on other sites
securitybreach

That may be the difference. I'm using Porteus-XFCE-v1.2-i486. Grsync is a Gnome app, so it comes with the Xfce version of Porteus, I guess. :)

 

Ah, yeah that explains it. I used the Standard Edition(KDE).

Link to post
Share on other sites
abarbarian

The easiest way to exclude files from rsync is to list them in an exclude file and specify it in the command eg - my music backup:-

rsync -av --delete --exclude-from=/mnt/music/rsync-exclude /mnt/music/ /mnt/backup1/Music-sdc

And the rsync-exclude file:-

lost+found
.Trash-1000

Good tip sunrat. I was going to try that out at some stage as I found similar examples with other's here,

 

http://www.thegeekst...es-and-folders/

 

An good stuff Eric, will give that a try sometime.

 

Meanwhile back on topic.

 

As I said I tried to follow the Wiki. So I copied and pasted the script and made some adjustments. Like swapping the " $1 " for " /my/destination/ " and adding a few folders to the " --exclude " list,

 

original,

 

 

rsync -aAXv /* $1 --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,/var/lib/pacman/sync/*}

FINISH=$(date +%s)

echo "total time: $(( ($FINISH-$START) / 60 )) minutes, $(( ($FINISH-$START) % 60 )) seconds"

 

touch $1/"Backup from $(date '+%A, %d %B %Y, %T')"

 

 

 

altered,

 

 

rsync -aAXv /* /media/sdf2-ata-SAMSUNG_HD103SJ_/Archphew/

--exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,/var/lib/pacman/sync/*,/home/*/.thumbnails/*,/home/*/.cache/chromium/*,/home/*/.mozilla/firefox/*.default/Cache/*,/home/*/dwhelper/*,/home/*/Downloads/*}

FINISH=$(date +%s)

echo "total time: $(( ($FINISH-$START) / 60 )) minutes, $((

($FINISH-$START) % 60 )) seconds"

touch /media/sdf2-ata-SAMSUNG_HD103SJ_/Archphew/"Backup from $(date

'+%A, %d %B

%Y, %T')"

 

 

I also made an alias in /etc/bash.bashrc. So I run the script and the exclusions do not work. The ecluded folders still get copied.

 

So is the script wrong ? Or am I missing something ?

 

I tried sunrats suggestion of a exclude file, and so far that looks to be working just fine. The reason I am banging on about this is there is a guide in the Arch Wiki and it does seem to have some holes in it. If that is the case then I would like to be able to fix the guide so that some other poor muppet like me can follow the guide and get a result.

 

Thoughts welcome. :breakfast:

Edited by abarbarian
Link to post
Share on other sites
abarbarian

Well it looks like I have a backup and using sunrats suggestion it seems to be almost perfect. Apart from this,

 

rsync: set_acl: sys_acl_set_file(var/log/journal/b5b4a6c0277be371da753ef9000005ae/.user-1000.journal.kpbTKh, ACL_TYPE_ACCESS): Operation not supported (95)

 

sent 24089809025 bytes received 3252385 bytes 9215169.79 bytes/sec

total size is 24077561215 speedup is 1.00

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]

total time: 43 minutes, 35 seconds

/usr/local/sbin/phew.sh: line 22: $'+%A, %d %B \n%Y, %T': command not found

 

from the guide,

 

 

/home/*/.gvfs should be added too, so that it won't complain at the end that "some files/attrs were not transferred".

 

 

Trouble is I do not have a ".gvfs" so why not ? How do I get one ? Will me backup still boot without one ?

 

The error in green I think I can solve meself.

 

A pox on all net guides ! Everyone should run exactly the same hardware as me then I would not have any problems. Just wait till I rule the world. I'll soon have things tiketty boo. :pirate:

 

Still I am making progress. I have a backup of sorts done from the cli not with some gui :harhar:

 

Now where is me Porteus usb stick :whistling:

Link to post
Share on other sites
sunrat

I hate rsync questions. :rolleyes: ^_^ They always get me started on trying to learn more about it, but as the man is over 3200 lines it often becomes a marathon.

 

For your acl error, look into the behaviour of the -A option and possibly the --fake-super option. Just a guess as I haven't used these.

 

I think the syntax may be wrong for exclude in your script. It looks much easier to get an exclude file working. From man rsync:

Note also that the --filter, --include, and --exclude options take one

rule/pattern each. To add multiple ones, you can repeat the options on

the command-line, use the merge-file syntax of the --filter option, or

the --include-from/--exclude-from options.

Link to post
Share on other sites
abarbarian

I hate rsync questions. :rolleyes: ^_^ They always get me started on trying to learn more about it, but as the man is over 3200 lines it often becomes a marathon.

 

For your acl error, look into the behaviour of the -A option and possibly the --fake-super option. Just a guess as I haven't used these.

 

I think the syntax may be wrong for exclude in your script. It looks much easier to get an exclude file working. From man rsync:

 

Well your idea to make a exclusion file worked just fine. So there could be something wrong with the script. Maybe I'll see if I can contact the person who posted it. :breakfast:

Link to post
Share on other sites
  • 9 months later...
abarbarian

Well I took the plunge.

 

Did the Full Backup with rsync,as per the wiki, used an exclusion text,an alias, to a external hdd. Everything seemed to go ok. Had some error messages they seemed to relate to systemd logs I think.

 

Then I did a Full Backup to a new ssd I came across cheap. I bought the ssd to play with bcache but a wedding album has halted that dream for the moment.

As I used the command line for this I used the drives uid for identification. Pressed enter and of it shot faster than the graphics in The Matrix. Super I thought, until the flying text stopped at some ssh file, sat there for ages.

I panicked and opened up spacefm to see what was going on, could see no action and some odd files on the new drive,lost and lonely them files looked.So I killed em to put em out of their misery.Noticed that the ssd was showing as /media/sdg1-ata-KINGSTON_SV300S37A60G_50026B723A0BC474. This looked similar to what I have in my rsync script so I entered that as an identifier.

An of it shot again,until the flying text stopped at some ssh file, sat there for ages. This time I did not panic, instead I looked at my cpu monitor which showed activity.So I let it run.

When rsync had completed I followed the wiki for editing fstab,I use uid's as I find it easier to keep track of different drives.Did the grub updating as per the wiki. Crossed me fingers and shutdown.

Rebooted and instead of a nice large text grub I found a tiny text grub squashed in one corner of the screen. I only have a five second slot to choose alternate boot options and by the time I had registered that there seemed to be twice as many boot options as normal I booted into me old Arch.

Rebooted and this time saw the new boot option, scrolled down and hit enter, keeping me fingers crossed. Before I could blink I was at a login prompt.

Yippeee !!!

So I am posting this from me old tech slow ssd and am dead impressed.

 

Now you may wonder why I abandoned the bcache dream. I thought that as I had a load of pictures to fiddle about with in Gimp the speed bump would be good. Also I thought rsync'ing would be a breeze and only take a couple of hours.

Wrong ! wrong wrong. It has taken me all evening as I am a slow reader. I'd forgotten that I had forgotten so much about a subject I know so little about. Still the quick test I ran with Gimp shows some considerable time saving.

 

:breakfast:

  • Like 1
Link to post
Share on other sites
V.T. Eric Layton

Good for you. Welcome to the rsync club. I do this every Sunday with my Slackware primary installation on my main system. I mirror the entire /(root) and /home partitions onto a secondary internal hdd. I cheat a little, though, as I use grsync, a graphic front-end for rsync. :)

  • Like 1
Link to post
Share on other sites
abarbarian

Good for you. Welcome to the rsync club.

 

Is the beer any good at the club ?

 

Every Sunday hmmm. Who a good geek then !

 

There is me evangelising and laughing at folks not doing backups, smugly superior an all that. When I looked at the date on my one and only backup it shows it is over six months old :w00t: Not much use apart from being a museum exhibit :hysterical:

 

Ha. I dreamt about os-prober, well more like had a nightmare about it.

So I installed it ON THE SSD and ran it( ran it from the original Arch yesterday), which I am assuming is what the wiki means, though it is a tad unclear to a inexperienced chap.

Os-prober did not seem to complete properly in the terminal.However a new "grub.conf.new" has been generated and this time it has the right uuid for the ssd.

Will that have fixed me funny grub screen ?

Time will tell :breakfast:

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...