Calling all RAID array experts...

[jloeb]

A question for those with some technical knowlege/experience with RAID arrays:

So i'm upgrading the storage capacity on the ol' G5, and since the factory drive ports are full, i'm thinking that the best way to accommodate all those huge sample libraries i'll want to stream from disk in the near future is to put in the additional triple-bay internal rack and add 2 or 3 matched drives as a striped RAID array.

So here are the options i'm considering:
Either:
A) Two Western Digital Caviar 320GB 7200rpm 3GB/sec sustained transfer rate, for ~$300,

or

B) Three Western Digital Caviar 250GB 7200rpm 1.5GB/sec sustained transfer rate, for ~$210

My reasoning process so far has been something like:

- The two larger drives with a 3GB/sec transfer rate (option A) cost a bit more, but the advantages would be a higher theoretical data transfer rate (6GB/sec) and one fewer drive to make noise, suck power, or break down;

- On the other hand, it may be that three 1.5GB/sec drives (option B), would in reality provide similar performance, or at least performance in excess of what i would ever need (4.5GB/sec) given that i'm streaming audio notes and not HD video, the one additional drive won't make that big a difference in noise pollution or power consumption, the total storage capacity is larger, the 250GB format might be more durable than the higher capacity format, and it costs less.

So i humbly ask: which makes more sense to you?

[cyril]

Hello

Putting 2 disk in raid will not double the rate transfer ! sorry

It's better to go on a 1.5 factor, if you do not want to be disappointed !

Be careful on internal raid with the G5
1) you will lose Apple care support
2) Your G5 is going to be much more noisy

It's better to have an external SATA raid

Best

Cyril
-----
Audio and Video Configuration :

Quad G5 2.5 Ghz, 8GB + 250 GB + 74 WD Raptor Mac OS X 10.4 + Motu PCI 424 E + 2048 MKIII

G5 2x2 Ghz 3.5 GB MAC OS X 10.4
Raid 0 for the libraries made with Rocket Raid 1820A and 6 x 36 gb raptor 10 000 rmp
+ 160+250 GB + FW 800 Lacie + 2x24" DELL

Motu system including : PCI 424 + 24i + 2048 MKII + 1296 (54 ins 12 outs)
Logic Control + 4 Logic Control XT
Midi interfaces : Stealth + OPCODE STUDIO 4 + MOTU MTP I ; USB EMAGICAMT 8 (48 ins/out)

[jloeb]

Cyril-
Thanks for your response. To follow up:

1) If i'm out of warranty (1yr), have i already lost Apple care support anyway?
2) An external RAID box won't be equally noisy?

Also-
Do i need to worry about speed so much if i'm not working with any video, just a lot of streamed instruments?
Can you recommend a good external box?

[jloeb]

Thanks, Lee, you likely saved me one large headache there. Or many small headaches. :D

From a quick google search it sure seems like there are a lot of PC turnkey DAWs that proffer RAID as a feature. But Digi indeed does not support it, as you said.

[geronimo001]

Is it a good idea to do a raid on a PC for sample library? Or would balancing my lib between 2 drive be enough? What kind of improvement should i expect with a raid?

thanks!

[cyril]

> For audio, I would not do RAID on a Mac. The conventional wisdom is that audio uses many small streams of data, unlike video, which uses one large stream.

Sorry to go against your idea but :

If on one song you have a lot of strings and on another a lot of brass how are you going to organise your libs !

What is a Raid, you spread your files at the block level (64 or 128 kb)
More disk you have faster the RAID will be.

Splitting you libs on multiple drives is what the RAID software is doing automatically for you !

I use a RAID 0 to put my QLSO PLATINUM, before I had this Raid I could not play my test song on a G5 2x2 3.5 gb (this song was just playing with Logic/EXS/VSL on a G4 533 MP !!!)


Concerning Jloeb :


> 1) If i'm out of warranty (1yr), have i already lost Apple care support anyway?
- Apple care if you got it is 3 years, also if you put too many device in the G5 you may ruin the power supply

> 2) An external RAID box won't be equally noisy?
it depends of the box

> Do i need to worry about speed so much if i'm not working with any video, just a lot of streamed instruments?
- What lib are you going to use ?

The best organization are :
Your system must be on a separate disk. (not on the Raid)

1) The ultimate :
A Raid 5 to put your audio files
+ A Raid 0 to put your libs
+ A separate disk or volume to put your swap files

2) The best : A Raid 0 to put your libs
A separate disk to put your audio files

> Can you recommend a good external box?
- You have to check that the noise pressure is under 37 dBA
You may need an extra power supply
Also it depends of your budget

I have put my 6 disk on a metallic tray in my Omnidesk from Omnirax and I have add 2 small fans, this is the cheapest, I have 7°C more that the external temperature

What you should do is buy the box near where you live so if it is too noisy you can take it back, because a know at least one manufacturer that has announced me 35.55 dBA before I bought there box I got it and I have mesure 53 dBA.

I cannot give you the name right now because I am trying to get my money back and I just gave then a 2 weeks notice before putting the information on all the forums

Best

Cyril

[geronimo001]

Thank you Cyril.

[jloeb]

So it seems like the principle that's being argued over could be articulated as follows:

There is a maximum fragmentation level of the target read-file beyond which using a RAID will not gain you any advantage, and can in fact hurt performance due to the increased total number of seek events.


This sounds intuitive to me, but i don't know enough about RAID technology to know whether it's actually true or not. Two pieces of empirical evidence are:
- there certainly are people out there who claim to be using it for digital audio, but
- digidesign certainly does not support its use, and this is a technology that's been around for a long time now so there is almost certainly a reason for that.

So: ?????

[jloeb]

Hmm. So from my own Googling around, i have come up with a couple of points from storagereview.net:

"RAID 0 has no positive effect on seek times. It can induce additional seeking delays."

"In other words, sequential transfers already complete so quickly that they have in effect written themselves out of the performance equation. Doubling the performance of a factor that exerts such a small effect nets a small improvement... as one should expect.

This is something I've long considered to be one of the dominate reasons behind the poor performance improvement offered by striping. It's great to have exact numbers to quantify it."

http://forums.storagereview.net/index.php?showtopic=21104&st=0&p=217603&#entry217603


A Groove Machine presentation on designing DAWs says:

"What makes a great Digital Audio Workstation?
Speed/Throughput
Hard Drive Speed/Interface Type-
• So why can’t my current music PC handle this many tracks?
Many reasons:
1. Fragmented hard drives – defrag as often as possible
2. Speed ratings are interface maximums, not actual throughputs capability
of drives- RAID Arrays solve this."

http://www.digital-groove.com/gm/DAW_Design_Optimization_2_Web.pdf

In this presentation, the purpose of the RAID arrays seems to be for *recording* tracks, not playback of streamed instruments. This makes more sense, as you're reading/writing a relatively small number of large-block audio files.

Can anyone speak from expert experience on this issue?

[irvind]

I found this article http://en.wikipedia.org/wiki/Redundant_array_of_independent_disks (http://en.wikipedia.org/wiki/Redundant_array_of_independent_disks)

a while back.

Makes for interesting reading, and even I understood it!

[mbmb]

what cluster sizes are best for sample streaming ?

[james vogts]

I bought a Supermicro P4SCT+II motherboard for my GS3 Orchestra /Scope DSP live rig. It is overkill, though I used RAID 1 for my O.S.+Apps. HDD, and another RAID 1 on my samples HDD. Thought that in a live scenario it would be wise for redundancy. Trouble is, when there is a HDD failure w/ audio HDDs, there is still a noticable dropout. So I thought what is the point? Then in my RAID cage, I had copies of each HDD from the RAID 1 configurations. So I disconnected the RAID setups and now have 2 copies for hot swapping if nescassary. Since my Raptors are quite reliable I just purchased 2 Raptor X's for the samples. Now I have 2 36GB Raptors w/ identical O.S.+ Apps., 2 74GB Raptors w/ indentical GigaPulse Pro impulses, and 2 150GB Raptors w/ identical sample library. For live gigging, I feel quite confident, for my hot swap RAID cage works w/o trouble. However, if I was in a recording situation, where I needed good RAID performance w/o audio dropping during a HDD failure, the only real viable array would be RAID 3 on a NetCell RAID Controller. It has buffering that allows HDD failure w/o missing a lick, which is why A/V experts swear by it. It has RAID 0 performance with RAID 5 redundancy. Check out Storage Review, or Toms Hardware for good audio benchmarking, plus video dub benchmarks.

[karmacomposer]

I have been using external usb 2.0 hard drives on multiple pci cards (this is for a PC, not a Mac) - a total of 1.6 TB (4 drives of 400GB each - 7200rpm with 8MB buffer) - and have had ZERO problems thus far. I have over 300GB of samples, and growing, in a variety of sample libraries, not to mention tons of vst instruments and THEIR sample libraries.

All of this runs off the externals and not the tiny little 40GB hard drive I have internally for the OS (windows xp home) and programs ONLY (sonar 5 producer, FL studio 6, etc).

Just thought I'd mention this to give you an idea.

Oh, and I USED to be a network engineer and set up network servers and raid arrays for companies (about 6 years ago), so I do know something about raid arrays (back then - not today). Today I would much rather use external drives on dedicated pci usb 2.0 cards.

Mike

[cyril]

Except for the fact that all the heads on every disk of your array need to go and locate their piece of each tiny sample file.

A sample is more than 64 Kb so instead to read a block of 64 K in one go it read 6 blocks of 64K at the same time (if you have six disk)




That doesn't make sense. There must have been some other problem with the way things were initially set up or installed on your G5. Since you're using Logic, are you using the Redmatica application to re-link things?

Currently, on my G5 DP 2.5, I have my sample files split between my two internal SATA disks (VSL library, plus lots of other stuff), and I use external FW drives for all my audio files. Things have been working wonderfully.

Lee Blaske

Re-read carefully !

You dont know much about QLSO Platinum, it is not EXS format it is Native instrument format so you must use Kontakt to read the samples. !

The EXS is very powerfull because the engine is included in Logic.
Test done with the same samples not using VM show that Kontakt is 6 times less powerfull that the EXS, this is why you less need a Raid 0 for the EXS.
I can play 80 simultaneous notes with K2, 480 notes with the EXS !

If you use a Raid 0 with the EXS there is a gain, try it !
You will gain quite a few voices before having sample "not read in time"

[jloeb]

Thanks to all for your replies, and there were good arguments on both sides. I tried a RAID configuration out, and thought i'd post the results of my experiments:

I needed additional storage space for sample libraries on a Mac G5 2x2.0MHz 4.5GB RAM, and was considering the possibility of configuring two or three drives as a RAID 0 array to increase speed of loading and efficiency of sample streaming.

The configuration i settled on was two matched 400GB Western Digital Caviar drives 150M/sec 16M buffer. I installed and formatted as RAID 0 using Apple Disk Utility, then copied my existing 160GB sample library from a full Barracuda drive with similar performance specs to the new RAID volume.

I compared loading times and streaming performance of identical large, streaming instruments by identifying first the single drive and then the array as the source drive from within several VIs (ArtVista VGP, EWQL RA, Superior C&V, EWQLSO Gold) and a softsampler (SampleTekk White Grand and a privately assembled KH Solo Strings violin library multi in MachFive).

In all cases, the loading times were highly comparable to within 2-12 seconds, in some cases the single drive winning out and in others the RAID volume winning. I called it a draw.

As far as performance on a single demanding instrument such as the ArtVista piano on a Jazz setting, playing lots of thick chords with the pedal down yielded no reliably detectable difference in available polyphony.

As an acid test, i set up a session in DP with several streaming instruments playing at high polyphony: the White Grand in MachFive, RA and EWQLSO Gold. I started in the single-drive configuration and purposely played thickly enough to red out the processor without causing audible glitches a couple of times (Which is DP's way of indicating that streaming is being maxed). I saved the session, reset the instruments to load and stream from the RAID volume, and then reloaded the session. This time, audible glitches were unavoidable and reproducible after numerous playbacks.

Changing RA alone to stream from the single drive solved the problem - no audible glitches and more consistent CPU meter behavior overall. I decided right there to delete the RAID array, reinitialize as separate volumes and split my libraries among the drives. In cases where the library is small (5-14GB) and likely to be used often i put a copy on two drives for now so that i can choose to invoke from a second drive if the first is streaming other instruments in a given session.

So to summarize, on my setup, the answer seems to be that using a RAID array for audio sample streaming confers a moderate disadvantage, and the dream setup would actually consist of numerous drives, each streaming a single library or instrument.

Thanks to all for your input on the matter.

[karmacomposer]

Jloeb:

Well thought out and well written - thank you for a great little experiment.

My experience has taught me that using multiple USB 2.0 drives using separate PCI USB 2.0 cards per 2 drives gives me excellent speed and allows for a LOT of streaming using Kontakt 2 and Sonar 5 Producer (on a Windows XP box) with no glitches or problems. Right now I have 4 identical 400GB drives on two separate dedicated USB 2.0 cards (the ones with 5 actual inputs) and they work better than expected and honestly, faster IMO than internal drives. Internal drives attached directly to the MB, whether primary or secondary, could not handle the data nearly as well as the USB 2.0 drives. Don't ask me why because I cannot give you the reason - but that has been my experience and I can run relatively large sample collections off of them, alongside vsti instruments with equally large samples with zero problems.

Mike

[cyril]

Thanks to all for your replies, and there were good arguments on both sides. I tried a RAID configuration out, and thought i'd post the results of my experiments:

If you want good results you need a raid card !

Also you should have read what i wrote :
1) more you have drive faster it will be. I have chose the solution to have 6 small disk 36 GB WD raptor instead of 2 bigger ones ! see the result of xbench after !

2) The system and if possible the swap files has to be on a different drive.
Also you have to set the DFD depending of your configuration

Can you run Xbench www.xbench.com on your 400GB Western Digital Caviar drives 150M/sec 16M buffer and post the results.

I have run Xbench 1.1.3* on a G5 2x2 3 GB here are the marks given by Xbench on different configuration (higher is better)

You can seen that there is not much difference between a 2 drive raid and a fast disk ! Speed is also depending in which slot you put your PCI X raid card !

86 with Lacie 160 GB FW800
97 with Lacie 200 GB FW800
103 with Lacie 250 GB FW800
109 with my internal Sata
122 with a Maxtor 250 GB
156 with Western Digital Raptor 36 GB
158 with a raid 0 of 2 Sata using Apple raid software
160 with a raid 0 of 2 Sata using Softraid
184 with the Rocket raid card in slot 3 and 4 Raptor 10 000 (with 1024 KB segments)
198 with the Rocket raid card in slot 3 and 4 Raptor 10 000 (with 64 KB segments)
205 with the Rocket raid card in slot 3 and 4 Raptor 10 000 with Softraid driver optimize for digital audio
227 with the Rocket raid card in slot 4 and 2 Raptor 10 000 with Softraid driver optimize for digital audio
237 with the Rocket raid card in slot 4 and 4 Raptor 10 000 with Softraid driver optimize for digital audio
268 with the Rocket raid card in slot 4 and 6 Raptor 10 000 with Softraid driver optimize for digital audio

* be carefull : Xbench 1.2 gives different results as they have changed the base reference

Best

Cyril

[cyril]

Jloeb:

Well thought out and well written - thank you for a great little experiment.

My experience has taught me that using multiple USB 2.0 drives using separate PCI USB 2.0 cards per 2 drives gives me excellent speed and allows for a LOT of streaming using Kontakt 2 and Sonar 5 Producer (on a Windows XP box) with no glitches or problems.

strange idea to by an USB 2 disk ! why not a SATA ! or SCSI !

[karmacomposer]

Cyril:

Not so strange at all. They are external, so no overheating problems. I can take them with me. Backups are lightning fast and when I built this computer SATA drives were ungodly expensive, unlike now. I do have SATA on my motherboard, but I am simply totally satisfied with what I have now.

Nope - not so strange.

Mike

[jloeb]

If you want good results you need a raid card !

Also you should have read what i wrote :

I read it, Cyril. There were other considerations at work: aesthetic, monetary and acoustic.

I was already decided that i wanted an internal solution, so for the G5 that meant at most three additional drives.


1) more you have drive faster it will be. I have chose the solution to have 6 small disk 36 GB WD raptor instead of 2 bigger ones ! see the result of xbench after !

I considered 3x 250GB drives or even 3 additional 160GB Barracudas matched to the one i have, but the price/GB, plus the additional probability of failure with each added drive, plus most importantly, acoustics, led me to choose two 400GB drives highly rated for acoustics. That's what led me to the WD4000KD - it is among the best rated high-capacity drives acoustically, and they are indeed extremely quiet during idle and seek, even right up next to the front grill of the G5 (i'm using the G5 Jive internal mount). Two is quieter than three, and i don't own a separate control room to put the computer in (at the moment :cool: ). And if i need more than 800GB of samples, i can always pop in one more.


2) The system and if possible the swap files has to be on a different drive.
Also you have to set the DFD depending of your configuration

That's an additional reason i didn't see a point in matching my 160GB seagate with 3 more; I wanted to use the one that was holding the samples as my new recording drive, instead of the system drive which is what i had been using. DFD has been played with extensively for each VI.


Can you run Xbench www.xbench.com on your 400GB Western Digital Caviar drives 150M/sec 16M buffer and post the results.

I have run Xbench 1.1.3* on a G5 2x2 3 GB here are the marks given by Xbench on different configuration (higher is better)

I would run Xbench but everyone's opinion on VersionTracker appears to be that the current version is total crap.

From the results you post, though, it looks like i made the right choice considering my goals. I would have had to add a lot of small external drives, plus a RAID card with fewer config options than the Tempo-X eSATA 4x4 i have for the same money in order to get any significant performance increases. Not really a preferred option for me.

Thanks for your input!