Second, once you have that, you need to be able to drive a lot of small files. But if you use find and pipe it to cpio, there is no overhead of making a list because files are listed in inode number order (sequential, as per filesystem maps) without a call to anything "ls" like.įirst, as someone above said, NFS for your NAS is the only option that makes sense on a mac. When using rsync, it needs to generate this list. This will shorten the list of files in any particular directory, and getting that list is very expensive for many utilities. I think rsync might be your best bet if you don't have cpio (don't bother looking for it), but if you eliminate the SMB overhead you will gain speed on the target.Īnother option is for the user to group the files into sub-directories first. It's like tar, but not really, because tar works on files and cpio works on a stream. | (cd /NAS/folder cpio -pdlm -) to unwind the stream. If MacOS has cpio available, you might give that a try because it's a great streaming option so you could do find. If it is choking at the volume, you might consider chopping the list of files down by using glob patterns in the file selection. Others have mentioned rsync, and it's a very good option. You should use rsync - there are many options and a few macros that are used to mirror sites.
#EMCOPY MIRROR MAC#
Mount the NAS to the Mac via NFS - even if you have to set it up for this one special project. I havent used it as I never copy at file level anymore just move the VHD as a blockDoes ParSync gracefully handle large amounts of small files? If approaching this dilemma from the angle of creating a vhd or image, would that make sense to do vs parsync?
You can run rsync in parallel, there is a build here thats Mac friendly I was only talking to a colleague today about the time it took us a day to copy 500GB of data for a team that had millions of small files vs an hour for twice as much video footage Things go fast until we hit a pocket of thousands of small files.Putting aside my fear of fluffy terms like RAIDZ2 and FreeNAS I would agree - whilst not the fastest setup for write (in fact one of the slowest) lots of little files is your problem here. I don't think the destination is the bottleneck for this, I think it's the source. How have you gotten it done dealing with these types of small files? What tools are up to the challenge? I don't care about the hardlinks in the time machine backups nor permissions.ĭestination is running stripes of 10 disks in RaidZ2 I know you guys have been in my shoes, especially with a lot of pressure to get things done. Currently, we're zipping root folders using the host Mac, copying them across the network and then unzipping but that is taking a long time and a lot of resources and attention. What is the smart way to approach this? So far, we've tried transferring with cp, mv, Beyond Compare, Directory Opus, etc.
#EMCOPY MIRROR PRO#
Mac Pro Trashcan (2013) fully loaded with 128 GB RAM.
#EMCOPY MIRROR FULL#
Keep the data synced until full cutover.Copy all data from thunderbolt RAID enclosure to main NAS.Things have been fairly smooth until I hit a user that has an Areca external RAID enclosure with 20+ TB of data and 20+ million small files. We are in the process of consolidating all our digital assets into our new main storage array so we can better manage and organize the data. We have a small corporate network running 10 GbE to all servers and to some workstations.
0 kommentar(er)
