ichase Posted March 27, 2015 Share Posted March 27, 2015 Greetings all, Do hope everyone has been doing well this year. I have been doing a lot of reading trying to figure out what is going on in regards to transfer speeds starting at something like 45 - 50 MiB/s then dropping during the transfer process to speeds as low as 9 - 11 KiB/s. This normally always happens when transferring large amount of files. For example, transferring 2.1 GB of files to a 16 GB flash drive USB 2.0 plugged into a USB 2.0 10 port hub that is plugged into the wall. The files started transferring very quickly until it got to about 1.4 GB, then it drops down to around 10 KiB/s transfer speed. You go from having 80% copied in about 2 minutes to being told it will be 3 hours estimated time to transfer the rest. Oddly enough, the transfer speed ended up speeding up on it's own after about 20 minutes to 4.5 MiB/s. As mentioned, I have done some reading via good ol' Google, but not seeing anything that is helpful. Not sure if I have some cache issue going on or what? Any suggestions? Thanks and all the best, Ian Quote Link to comment Share on other sites More sharing options...
securitybreach Posted March 27, 2015 Share Posted March 27, 2015 I found a few links but it seems to be more a kernel bug that hasn't gotten ironed out yet: http://superuser.com/questions/424512/why-do-file-copy-operations-in-linux-get-slower-over-time https://bugs.launchpad.net/ubuntu/+source/linux/+bug/582390 Quote Link to comment Share on other sites More sharing options...
ichase Posted March 27, 2015 Author Share Posted March 27, 2015 Thanks Josh, you seem to be right based on what I was reading. I have tried multiple flash drives even external HDDs and the results are always the same. Starts off fast then slows to a snails pace. SOMETIME I will even get errors before it completes but not to often. Quote Link to comment Share on other sites More sharing options...
securitybreach Posted March 27, 2015 Share Posted March 27, 2015 Well I guess we can't have everything working perfectly in Linux.... BTW I am sure you know, but this also affects windows as well. Not in the same way but they both do this sort of thing. I always chalked it up to i/o speed issues until I searched for you. Quote Link to comment Share on other sites More sharing options...
ichase Posted March 27, 2015 Author Share Posted March 27, 2015 I considered I/O speeds as an issue as well but figured as fast as the transfer started, the I/O issue would have been slow from the get go. Was leaning more towards a cache issue but, oh well.....The files do transfer so good to just let it do it's thing in the background once you start. Quote Link to comment Share on other sites More sharing options...
abarbarian Posted March 27, 2015 Share Posted March 27, 2015 D.M. 2014-01-31 16:32:25 UTC Heya, after some years I have resolved MY problem with the respopnsivness of the computer in regards to HIGH I/O. For all others I can only say, try it. If it works for you be happy, if not... For starters I wouldn't call this a bug. It's a DEFECT. Because if I have 20 servers with different linux flavours and distributions, many of them were compiled from scratch, and if I have 200 ubuntu desktops that behave all the same if I use the command dd if=/dev/zero of=test.img bs=1M count=xxx (above 1GB file size), by same meaning this command grinding the system to a halt, and then keeping this problem around for so many years and so many kernels, for me it is a DEFECT. For the past week I've been trying the BFQ patch for kernel 3.9 on several machines. On one machine I have been heavy testing. I have this machine for some years now, a CORE i7 with 12 GB and 6 HDs in RAID 10. On this I also had the problem, and it was somewhat better with the BFS patch but it was still happening. With the BFQ patch it's working perfectly. At one moment I had two dd's (dd if=/dev/zero of=test.img bs=1m count=100k, creating a VirtualBox vdi of 60GB, openining 10 ods documents, watching youtube, watching a hd movie in vlc and some other stuff and the desktop / system was as responsive as if nothing was using it. Just like I remember linux being some years ago. And now I have a sustained throughput of 470MB/s HD without my computer going to /dev/null. So BFQ solved this problem for me. Maybe it's not stable yet, but for me it's more stable than using CFS !!! Just my two cents. And this bug is closed for me, but only NOW !!! For all others out there I wish you luck Horst Schirmeier 2014-11-06 15:23:46 UTC I'm still seeing this. Setup: Debian 7 Wheezy, amd64 backports kernel (3.11-0.bpo.2-amd64), ~45MB/s write of a low number of large files by rsync (fed through a GBit ethernet link) on an ext3 FS (rw,noatime,data=ordered) in a LVM2 partition on a hardware RAID5. Observation: The machine (32-core Xeon E5-4650, 192 GB RAM), primarily servicing multiple interactive users via SSH, x2go and SunRay sessions, gets completely unusable during and quite some time after the rsync transfer. TCP connections to SunRay clients time out, IRC connections are dropped, even simple tools like "htop" don't do anything but clear the screen after being started. "iotop" shows a [jbd2/dm-1-8] process on top, reportedly doing "99.99%" I/O (but not reading or writing a single byte, maybe because it's a kernel thread?). Once I switch from the default CFQ I/O scheduler to "deadline" (echo deadline > /sys/block/sdb/queue/scheduler), the symptoms disappear completely. https://bugzilla.kernel.org/show_bug.cgi?id=12309#c11 Bit above my grey cell grade but these two chaps seem to have found different solutions.If you find an answer let me know as I have similar problems to ichase. 1 Quote Link to comment Share on other sites More sharing options...
crp Posted March 27, 2015 Share Posted March 27, 2015 I wonder if it really is a kernel bug. The overhead with USB2 when transferring large data amount is not the greatest and a low quality USB drive will suffer doing these processes even more so. If it is really an issue that gets in the way, try splurging on a thumb drive. I found this post which is similar, http://forums.pcper.com/showthread.php?471312-Why-do-USB-thumb-drives-transfer-so-slowly 1 Quote Link to comment Share on other sites More sharing options...
lewmur Posted March 27, 2015 Share Posted March 27, 2015 Greetings all, Do hope everyone has been doing well this year. I have been doing a lot of reading trying to figure out what is going on in regards to transfer speeds starting at something like 45 - 50 MiB/s then dropping during the transfer process to speeds as low as 9 - 11 KiB/s. This normally always happens when transferring large amount of files. For example, transferring 2.1 GB of files to a 16 GB flash drive USB 2.0 plugged into a USB 2.0 10 port hub that is plugged into the wall. The files started transferring very quickly until it got to about 1.4 GB, then it drops down to around 10 KiB/s transfer speed. You go from having 80% copied in about 2 minutes to being told it will be 3 hours estimated time to transfer the rest. Oddly enough, the transfer speed ended up speeding up on it's own after about 20 minutes to 4.5 MiB/s. As mentioned, I have done some reading via good ol' Google, but not seeing anything that is helpful. Not sure if I have some cache issue going on or what? Any suggestions? Thanks and all the best, Ian I think you are facing a combo of issues. I have noticed many times when transferring large files to a flash drive that the "process bar" slows down drastically in the middle of the transfer. I have always assumed this was just poor reporting by the "bar". But from what I'm reading in this thread, there may also be a problem in the kernel. But to verify it isn't just poor tracking by the "bar", why don't you actually time whole file transfer and then divide by the file size. For cheap USB 2.0 flash drives, 5-6 mb/sec is about all you can expect. Quote Link to comment Share on other sites More sharing options...
V.T. Eric Layton Posted March 27, 2015 Share Posted March 27, 2015 Fast starts and then slow finishes are often signs of a bottleneck somewhere... SATA buss, CPU, etc. 1 Quote Link to comment Share on other sites More sharing options...
amenditman Posted March 27, 2015 Share Posted March 27, 2015 Fast starts and then slow finishes are often signs of a bottleneck somewhere... SATA buss, CPU, etc. USB i/o controller and cache, alsoDon't forget, that USB stick has it's own little computer built right in to handle the data you give and request from it. 1 Quote Link to comment Share on other sites More sharing options...
burninbush Posted March 28, 2015 Share Posted March 28, 2015 I have come to believe it's just slow usbsticks, and the system's use of cache. The xfer goes fast until some local cache is filled, after which it slows down to the innate speed of the stick. If you specify a smaller file transfer it may seem like it went fast, but then when you sync before removing the stick that's where you'll encounter the wait. It is interesting to me to try a different usb device, an external hard disk like the little Seagate FreeAgent drive I have used and loved for a couple years now. It does NOT slow down with large transfers even though plugged into the same port as a usbstick that shows horrible slowdowns. Hdparm -t shows it conistently running abound 34MB/second. So from this I conclude that it's not the usb interface or the filesystem at fault. If you don't mind the inconvenience, a cheap pci-express card will get you a couple usb-3 ports. Sticks made for that are capable of going much faster. 2 Quote Link to comment Share on other sites More sharing options...
ichase Posted March 28, 2015 Author Share Posted March 28, 2015 Great responses here, have learned a lot more than I was expecting to learn. :thumbsup" burninbush was thinking along the lines that I was thinking. Cache fills up then poof....speed quickly drops. Yes Eric, definitely a "bottleneck" somewhere, but as Josh pointed out, this has been a problem within the kernel and I guess I really have not paid so much attention to it until recently. After having to switch HDDs after a bad Seagate, I have been transferring a lot more files plus getting data off other people's dead computers. 2 Quote Link to comment Share on other sites More sharing options...
securitybreach Posted March 28, 2015 Share Posted March 28, 2015 Great responses guys, that is basically what I was thinknig too. Quote Link to comment Share on other sites More sharing options...
.thumb.jpg.7165b5e8fbb3ef44efbcc2ad57bf033d.jpg)
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.