14 replies to this topic

[frapper]

I realize you can do this through the Format function in Explorer. But on this page http://www.ntfs.com/quest22.htm
why would you do steps 1-5 to "optimize for performance" using write caching? Is it necessary? Recommended?

I want to store a couple disk images on a 64GB drive and figured NTFS would be best to avoid file splitting. Thoughts?

[crp]

Which OS are you using? Is this USB 3 ? Just how big are the 2 files?

[frapper]

Win7, USB 2 (both ports and flash drive), image files are about 20GB each.

[crp]

frapper, on 05 July 2012 - 04:50 PM, said:

Win7, USB 2 (both ports and flash drive), image files are about 20GB each.

  that is going to take a LLLLLLLLLLLLLLOOOOOOOOOOOOOOOOOOONNNNNNNNNNNNNNNNNNNGGGGGGGGGGGGGGGG time to get done.
If it was me, I wouldn't risk it and would let the application do the file splitting.

[Tushman]

frapper, on 05 July 2012 - 02:18 PM, said:

I realize you can do this through the Format function in Explorer. But on this page http://www.ntfs.com/quest22.htm
why would you do steps 1-5 to "optimize for performance" using write caching? Is it necessary? Recommended?

I want to store a couple disk images on a 64GB drive and figured NTFS would be best to avoid file splitting. Thoughts?

If you go with FAT32, you're going to be limited to 2GB.  Splitting that image file with the FAT32 limitation would mean that you're going to have roughly 20 separate files - definitely not recommended under any circusmstance.  NTFS is the way to go if you are planning on storing large sized files.

I prefer using Microsoft's Robocopy whenever I need to copy large sized files like what you're doing.  It's faster than using Windows Explorer interface, but the real benefit is that it's capable of writing/copying very large sized files with a high level of data integrity.  If you are running Vista/Win 7, it's built into the OS.  If you're running XP, you'll need to download Server 2003 resource kit.

[frapper]

Interesting. I hadn't thought about using the command-line Robocopy. Thanks, Tushman.

I may also just go with a file sync utility like FreeFileSync or SyncBack (instead of imaging) and do current file backups of both machines onto the stick. Then I could just occasionally update it.

[zlim]

FAT32 has a file limit of 4GB not 2 GB. But it would still give you too many files to deal with, instead of 1 large file.

[crp]

I don't think 10 files instead of 2 gets into the 'too many files to deal with' level.
Especially if it a one (or two) shot deal.
If weekly or even monthly, I'd spend the $120 or so and get a USB3 card and thumbdrive

[burninbush]

Re your question about steps 1,,,5 -- I think that normally windows will not complete a long write to a device [internal or external] immediately under normal conditions.  It caches data-to-be-written until it's convenient to write at a later time.  Obviously you can't safely remove the stick until all writing is complete, so if you want to quicken the time it takes to release it then you must defeat cached writing for that device.

As for writing large files to usbstick ... I can only say that I am very pleased with my Seagate FreeAgent 320gb usb [external] hard disk.  Cost me $40 at newegg last year.   I bought it as a gadget, thinking I could pull the drive from it if it turned out to be useless, but in fact it has seen heavy use ever since I got it.  Hdparm says it does 36MB /second transfers, and it doesn't slow down on big files.

I know you didn't ask -- but if those are important backup files, I'd put them on dvds for reliability.   You can probably coerce your app to break them into chunks that will fit.

[burninbush]

Tushman, on 05 July 2012 - 08:25 PM, said:

If you go with FAT32, you're going to be limited to 2GB.

Pssst ... 4gb (minus 1 byte.)  Nevermind, already mentioned up the thread.

Edited by burninbush, 06 July 2012 - 01:59 PM.

[Temmu]

there is also microsoft rich copy - similar to robocopy, but with a gui interface.
microsoft says it is the fastest file copying tool in existence.

please note, depending how new the copy is that you find, the 3 threads recommendation has been superseded by 8 threads.

i've used this to move large piles of files, the performance is noteworthy.

http://blogs.technet...oad-center.aspx

ps
to format ntfs, simply go to disk manager, format as ntfs, and choose quick - for only 64gb, it won't take long at all...

Edited by Temmu, 06 July 2012 - 11:17 PM.

[frapper]

Thanks Temmu. Here are some screenshots in a review at Softpedia (slightly older version).  http://www.softpedia...ew-107778.shtml

I assume RichCopy Build 4.0.216 is the most recent since I can't find anything with a higher number.

edit- .217 is supposedly available.

Edited by frapper, 07 July 2012 - 10:54 AM.

[Tushman]

crp, on 06 July 2012 - 12:53 PM, said:

If weekly or even monthly, I'd spend the $120 or so and get a USB3 card and thumbdrive

What does a USB3 and thumbdrive have to do with anything?  It doesn't matter what the storage medium is - as long as you format the drive with NTFS the issue is moot.  Even an external drive with USB 2.0 connection can transfer a large file relatively quickly.  I should know I've been doing it for the past 8 years on my home rig.  And no -- it does not take "forever".  Using robocopy command, it takes roughly 2 mins (or less) to transfer an 8 GB image file.

crp, on 06 July 2012 - 12:53 PM, said:

I don't think 10 files instead of 2 gets into the 'too many files to deal with' level.
Especially if it a one (or two) shot deal.

This isn't about 'too many files to deal with'.  Todays large capacity drives make that a non-issue.

When you split image files, there is a higher chance that one of them can go corrupt.  The issue here is about data integrity and ensuring that the image file will work when you need to restore it.  To me it is just asinine to split an image file into 10 separate pieces especially if it means an operating system on the line.

Edited by Tushman, 07 July 2012 - 05:20 PM.

[ross549]

An additional point about USB3. It does not matter how fast the USB interface is, as long as it is fast enough to handle the maximum read/write speed of the flash memory. You could 30MB/sec flash memory module and put it on a USB 3 interface, and it will not be any fast reading and writing. You need to look at the maximum sustained read/write speeds to have a good idea of the performance of the drive. The same applies to regular hard drives. You can put them on a SATA6 interface, but unless they are reading and writing at 500MB/sec (near impossible unless it is a SSD), that faster interface will not do squat for you.

Adam

[goretsky]

Hello,

You might want to consider using Microsoft's exFAT filesystem, which is optimized for flash drives.  It is technically superior to FAT and does away with many of that file system's limitations, however, it's long-term usability/viability is unknown, and OS compatibility is limited to recent versions of Windows and OS X.

Regards,

Aryeh Goretsky