[BAILOPAN]

jsterling: Thanks for your replies. The server is a single quad-core Xeon (E5430) with 4GB of RAM. The problem with 500 errors isn't a spec problem, I think. We really have no idea how they happen. We tried debugging them last year and got nowhere.

Part of the problem is we've been relying on mod_fcgid which is really, really buggy. Some versions are totally unusuable, and we've had to hand-patch it in the past, or use specific revisions in between "stable" releases. The error messages it spews might as well be in Egyptian hieroglyphs.

Another part of the problem is that no one seems to really know how any of this works. There are tons of tutorials that have cargo-culted mod_fcgid/PHP settings with no explanation or real understanding of what they do. You can scour the Internet and find tons of people getting the same mod_fcgid problems as us, but every lead is a dead end - these threads always go unanswered, or end up being unrelated.

We're at the point where we're starting to get some useful data. Fyren is adding debug instrumentation to mod_fcgid so we can see why PHP processes are failing, and I've started to notice some new patterns in load when errors start occurring. Part of our infrastructure changes could involve putting MySQL on a separate server, in case the problem is actually load.