3Ware Sucks!
FYI Im not going to provide benchmarks or anything remotely useful here.
3Ware raid controllers suck when it comes to random IO r/w. I recently installed a new mail delivery server with 15 1TB Hitachi hard drives and a 3ware 9650SE-24M8. Raid 6 for the mail spool raid 10 for the mail indexes and raid 1 for / and everything else.
I always heard that Random IO sucked on these cards from other colleagues but brushed it off thinking I could live with less then optimal disk r/w throughput. Little did I know it was SO bad that the machine wasn’t even usable. I/O wait was through the roof! Like over 90%! Waiting to write mail to mail spools was timing out from locks that were still being written. It was REALLY BAD!
I quickly tore down the Hardware Raid and ran the raid 6 and raid 10 JBOD on the 3ware controller with software raid with the same I/O mail load. I/O wait is never over 10% now.
Note: 3Ware cards are pretty solid cards with nice management. They just have some random i/o issues to work out. Their support is not that helpful either on the issue. I turned all the nobs they suggested with little or no results. The cards seem to be tuned and geared towards high sequential I/O. And they do work great for that. 3ware will probably never fix the problem as long as people keep buying them though.
This was done on Centos 5 with dovecot and postfix. Was tested w/ same results on FreeBSD.
BTW: Munin is pretty nice! Thanks to dforbes for turning me on to it.