There is no evidence that the OP's scenario is caused by background rendering or by corrupted cache files. The situation is more complex.
This is an old architectural design issue that has existed for years. I have previously discussed it with Apple escalation support. The FCP caching system is complex and creates MD5 hashes of rendered frames/segments based on several things, including effect parameters and content. The internal name for render cache is "segment store."
There are two separate caching systems in FCP, one for render cache, handled mostly in the "Flexo" framework, and the other for effects cache, handled mostly in the "Ozone" framework. Any caching system must be careful about validation; otherwise, a "stale" or inconsistent cache will be used, producing incorrect results.
This system is complex and must handle many different states. From a design standpoint, the simplest approach is to be conservative with render cache state and, if in doubt, re-render it. However, that hurts performance.
There appear to be cache dependencies between parent and child clips, such that the parent clip can invalidate the cache on all child clips. There is some evidence that the render state trackers don't fully cover all scenarios involving proxy vs. original clips. This is likely why switching between proxy and original views sometimes invalidates cache segments (shown as render dots).
FCP's render state tracking is complex. While FCP is running, an in-memory cache state tracker is used. When FCP quits and relaunches, it has to reconstruct what's valid, potentially losing nuanced information about cache validity.
Upon relaunch, it appears to validate segment stores based on multiple factors that might vary between launches, e.g., file system timing, render property matching, path validation, etc. The MD5 validation should be deterministic, but there could be issues with launch-time discovery of segment stores and the conditions under which they're validated. The file system state, timing, and initialization order could all play roles in whether the existing cache is recognized as valid.
There's a separate caching system for the Motion runtime engine that interacts with the segment-based render cache. However, there is some evidence that the two caching systems are not fully coordinated, which could explain some of the observed behaviors. In some scenarios, a temporal effect (i.e., one that uses a sliding window of frames) may not be handled properly, resulting in a non-cached state. Example: Neat Video.
From a cache designer's standpoint, there is always a bias toward taking the safest approach. For example, if you find a cache coherence bug, the easiest and safest step is to tweak the code so that 're-render is required' in that case. That avoids the risk of incorrect results and damaged data, which would be much worse than slower performance. Fixing the scenario to maintain performance without integrity issues is difficult and costly from a testing standpoint.
