Has having try servers run on demand, say in Amazon EC2, been discussed in the past? Seems like it might make sense. We'd be able to have as many try servers as we need (especially important when there is a lot of load, like before a code freeze - but also in general, to reduce wait times). And we'd be able to not run any try servers when we don't need them.
I guess it can't handle all cases - EC2 can't run all OSes, and the machines are servers, so no normal screen rendering, etc. Also their hardware and performance might not be standardized enough for performance results to be informative. But certainly for building and for a lot of the tests (xpcshell, etc.), it seems like it could speed development up. Maybe building could be sent to EC2 servers on demand, and the results sent back to normal servers for the tests?