I'm running ProTools on OSX 10.2.6 with a 550 mhz CPU and 1 gig RAM. And I can usually get 7 or 8 instruments running at one time. I'd expect that you should get many more instances at one time with your specs. For comparison, have you tried running GPO Studio with a notation program and not used Pro Tools at all? See if that gives you more voices. Try to narrow down where the bottleneck is occuring.