What's happening with the builds failing / taking super long in experimental the last week or so?

I’m not a programmer, but I’m curious about the current apparent woes with the experimental builds for the last week or so. I can’t imagine that it’s normal for it to take over three hours to compile a build. Any idea what broke or what the problem is?

whenever you try and fix something programming wise, you often end up breaking several other things… devs are people too, so they may save a build simply because they’re done working on it for the day, or to give to other devs to fix, but that build is unsuitable for actual release for people to play (because the broken part would be something everyone immediately points out as being broken… when they already know full well its broken.)

… that said i’m not on the team for this project, thats just from what i know of the development process.

Builds are not too slow - there are 6 desktop targets built in 3 hours - that is 30 minutes per build target in average. Depending on what was changed, builds can be much faster (as low as 5 minutes), while some targets could take an hour to compile. We optimize compile times occasionally by reducing includes, but there is constantly adding overhead of commit history which we cannot avoid for some reasons yet (e.g. build server needs full commit history to build a changelog, so we cannot fetch shallow copy).

We also had to reduce number of threads to 1 recently because of hardware limitations (limited memory and hard drive) of build server.