Diagnosing Thermal Throttling in Sustained Cloud Mac Builds

When the same commit takes 11 minutes on the first build but 15 minutes on the fourth during repeated builds on a cloud Mac, do not immediately conclude that the machine’s performance is unstable. Cache hits, indexing processes, test concurrency, and thermal pressure can all produce similar patterns. The correct approach is to control the variables, collect consecutive samples, and then determine whether the increase in build time correlates with power consumption and thermal pressure.

Define a Repeatable Build Sample First

The diagnostic target should be a command that can be reproduced consistently, not a Scheme triggered manually by a developer. Pin the code commit, Xcode path, SDK, target, build configuration, and DerivedData directory, and record how long the machine has been idle since startup. If every run switches branches or updates dependencies, the results only reflect changes in workload.

On VMOrbit dedicated physical nodes, host resources are not affected by virtual machine scheduling from other tenants. However, processes on the node can still compete for resources, including indexing, dependency resolution, simulators, and leftover build processes. Save the basic environment information before starting:

mkdir -p "$HOME/build-thermal-check"
system_profiler SPHardwareDataType > "$HOME/build-thermal-check/hardware.txt"
xcodebuild -version > "$HOME/build-thermal-check/xcode.txt"
pmset -g > "$HOME/build-thermal-check/power.txt"
ps -axo pid,%cpu,%mem,etime,command > "$HOME/build-thermal-check/processes-before.txt"

Measure Cold and Warm Builds Separately

Cold builds reveal the sustained load of a full compilation, while warm builds show the incremental build path. Do not mix them in the same statistical sample. Cold builds should use a new dedicated DerivedData directory for each group, while warm builds should reuse the directory from the previous run. Avoid clearing global caches, as doing so could affect other tasks on the node.

Collect Build Times and Thermal Pressure Together

Total build time alone cannot prove thermal throttling. Run at least three consecutive builds and collect powermetrics data throughout each build. This tool usually requires administrator privileges, so authorization should be limited to the specific command and parameters required.

RUN_ROOT="$HOME/build-thermal-check/run-01"
mkdir -p "$RUN_ROOT/DerivedData"

sudo powermetrics --samplers cpu_power -i 1000 -n 900 \
  -o "$RUN_ROOT/powermetrics.txt" &
METRIC_PID=$!

/usr/bin/time -lp xcodebuild \
  -workspace App.xcworkspace \
  -scheme App \
  -configuration Release \
  -destination 'generic/platform=iOS' \
  -derivedDataPath "$RUN_ROOT/DerivedData" \
  build > "$RUN_ROOT/build.log" 2> "$RUN_ROOT/time.txt"

wait "$METRIC_PID"
pmset -g therm > "$RUN_ROOT/thermal.txt"

-n 900 collects at most 900 samples. If the build finishes sooner, background sampling will continue. A project script can send TERM to the sampling process when the build ends, but it must preserve exit handling so that CI does not leave the sampler running. If powermetrics is unavailable, save at least the output of pmset -g therm, time -lp, and a process snapshot sorted by CPU usage.

Thermal throttling is a conclusion supported jointly by build-time trends and system signals. A single observation of high CPU usage is not enough to confirm it.

Draw Conclusions from the Series, Not a Single Data Point

Record each run’s elapsed time, maximum resident memory, thermal pressure, and concurrent high-load processes in the same table. If the first run is slow and subsequent runs become significantly faster, cache warm-up is usually responsible. If one run suddenly slows down while dependency downloads or indexing processes appear, resource contention is the more likely cause.

The pattern that deserves closer attention is this: with the code and cache state unchanged, build times increase gradually over several consecutive runs; the CPU remains fully utilized for an extended period; thermal pressure rises at the same time; and after the workload stops and the machine is left idle, the next run recovers. Thermal throttling should be treated as the primary cause only when all of this evidence appears together.

Exclude Three Common False Positives

First, check whether the build is silently running tests, creating an archive, or downloading files through a script. Next, compare the number of compiled files in the logs to confirm that every run performs the same amount of work. Finally, verify that multiple xcodebuild, simulator, or package-management processes are not running in parallel. During an anomalous run, save the following:

date -u
pgrep -alf 'xcodebuild|swift-frontend|clang|Simulator'
top -l 3 -o cpu -stats pid,command,cpu,mem,time
du -sh "$RUN_ROOT/DerivedData"

The instantaneous ranking from top only shows which processes were consuming resources at the time of sampling. It cannot establish the root cause on its own. The task counts and durations in the build log provide the evidence needed to connect system state with the project workload.

Use Controlled Experiments to Identify the Trigger

After confirming the trend, change only one variable at a time. First, reduce the number of concurrent CI jobs to 1 while keeping all other conditions unchanged. Then restore concurrency but stagger the job start times. Finally, compare cold and warm builds. If the build-time curve stabilizes after concurrency is reduced, the issue is more likely caused by competition between tasks on the same node or excessive sustained power consumption than by degradation in the compiler itself.

Do not treat more aggressive cache clearing as an immediate fix. Clearing caches increases the workload of the next run and hides the original trend. Do not use a restart as the sole piece of evidence either: restarting terminates background processes, clears some state, and gives the machine time to cool down, so it cannot reveal which change actually resolved the issue.

Adjust Build Scheduling Instead of Chasing Instantaneous Clock Speeds

A stable solution usually involves limiting the number of heavy tasks on the same node, staggering archives, full test suites, and static analysis, and assigning a separate DerivedData directory to each pipeline. For workloads that must run in parallel, first measure the median build time at concurrency levels of 1, 2, and 3. Then choose the level that produces the highest total throughput rather than the best peak performance for a single task.

Turn Diagnostic Results into a CI Baseline

The baseline should record the command, commit, Xcode version, build type, number of consecutive runs, and median build time. Do not use the fastest run as the benchmark, and do not block merges because of a single timeout. A more reliable rule is to compare rolling medians from the most recent runs and generate a diagnostic record only after the threshold has been exceeded repeatedly.

The final artifacts should include at least the build log, time output, thermal-pressure records, process snapshots, and sampling parameters. If the issue needs to be escalated, include the node region, occurrence time, reproduction steps, and the minimum necessary logs, but never submit passwords, private keys, or complete access credentials. The next time similar fluctuations appear, the team can rerun the same sample directly instead of starting the investigation from guesswork.

Frequently asked questions