TeamCity investigation
Prepared 2026-07-13 10:17 HST
QUEST UNIT TESTS · JOHN DEERE

Build #718 has two separate regressions

The failed Maintainer test is old and deterministic. The four-hour timeout is newer, repeatable, and more damaging because it prevents roughly 90% of the suite from reporting results.

Fix the play-mode hang firstBuilds #716 and #718 both stopped at the 240-minute ceiling after the same lesson-test transition.
Then reconnect 4 referencesTwo array entries are missing in each of two 210P prefabs.
4h 0m 43sBuild #718 runtime
13 / 125Tests reported vs prior full run
2 in a rowTimeout builds: #716, #718
4 uniqueMissing references

Finding 1 — play-mode test runner hangs

HIGH CONFIDENCE: REGRESSION WINDOW

The timeout begins with build #716

Build #715 completed in 1h 2m 9s and reported all 125 tests. Build #716 moved to revision fb3bd5d, ran 4h 0m 48s, and reported only 13 tests. Build #718 repeats the same shape at 4h 0m 43s.

BuildRevisionResultReported testsRuntime
#71580f29554 failed1251h 2m 9s
#716fb3bd5dtimeout134h 0m 48s
#7185e62ce9timeout134h 0m 43s
MEDIUM-HIGH CONFIDENCE: PRIMARY SUSPECT

Smart Grade Demo changes are the likely trigger

The first timeout revision brought the Excavator Development branch into develop. The strongest suspect is f1dac6c — “210P Smart Grade Demo Updates”. It modified the Smart Grade Demo lesson, Lesson 08 controller, simulator UI, and lesson-completion behavior.

  • The last completed activity in #718 is a 210P Hotspots lesson test and teardown at 06:01:26 UTC.
  • The runner immediately starts the next Lesson_Scenes.TestLessonSteps_EventState case but never logs a completion.
  • Only licensing heartbeats and allocator warnings appear for the next three hours before TeamCity terminates Unity at 240 minutes.
  • The same incomplete 13-test result occurred in #716 on the same agent.
Most efficient isolation: run the play-mode lesson suite with the new 210P Smart Grade Demo disabled. If the suite completes, re-enable that lesson alone and add a per-test timeout around TestLessonSteps_EventState so one lesson cannot consume the entire build.

Finding 2 — four persistent missing references

HIGH CONFIDENCE: EXACT ONSET

The references first fail in build #701

Build #700 scanned zero Maintainer issues. Build #701 was the first build to report exactly four issues. Every build from #701 through #718 reports the same four. The notification prints each issue twice, but these are four unique references, not eight.

  1. States/Design Setup → Preset Options → ExcavatorP_SmartGradeOffsetPresetEditModal → Value Field Guidance[0]
  2. States/Design Setup → Preset Options → ExcavatorP_SmartGradeOffsetPresetEditModal → Value Field Guidance[1]
  3. Excavator PDU Presenter with Haptics → …/Preset Options → ExcavatorP_SmartGradeOffsetPresetEditModal → Value Field Guidance[0]
  4. Excavator PDU Presenter with Haptics → …/Preset Options → ExcavatorP_SmartGradeOffsetPresetEditModal → Value Field Guidance[1]

The regression entered between revisions 35236dd (#700) and 0c974c0 (#701). That window contains two directly relevant commits:

Fix: open both prefabs in Unity and reconnect Value Field Guidance[0] and [1] to the intended L07 guidance assets. Validate both the source prefab and the nested presenter instance, then run Maintainer alone before the full suite.

What is not causal

  • PR #860 / Tips UI overhaul: it is included in #718 but cannot explain either onset. The missing references date to #701; the timeout first appears in #716.
  • Bucket_ID7 convex-mesh warnings: these appear in successful Maintainer runs before the regression and are not counted among the four issues.
  • Agent VM-05_BA-03 alone: the missing-reference failure reproduces on both BA-02 and BA-03. The timeout has only been observed on BA-03, but both timeout runs share the same source regression and execution pattern.
  • Increasing the 240-minute limit: this would conceal the hang. The previous full run completed in roughly one hour.

Recommended repair order

Disable the new Smart Grade Demo in lesson unit-test settings and rerun play-mode tests.

If the run completes, isolate Lesson 08 and fix its completion/teardown path.

Reconnect the four Value Field Guidance slots in the two 210P prefabs.

Run the Maintainer test alone; expected issue count is zero.

Add a per-lesson timeout and print the parameterized lesson name before setup.

The current log cannot name the unfinished case because Unity reports the parameterized result only after completion.

Run the full configuration and require roughly 125 reported tests with no timeout.

Do not accept a green Maintainer-only run as proof that the play-mode hang is fixed.

Evidence

TeamCity REST APIBuild metadata, test occurrences, 40-build Maintainer history, revisions, change lists, build configuration, and full #718 log.
TeamCity build log240-minute termination, Unity process command, last substantive test activity, teardown/setup boundary, and post-hang heartbeat pattern.
GitHub commit APIPrivate forgefx/Johndeere revision comparison and per-commit file patches for the two regression windows.
Confidence boundaryThe missing-reference attribution is exact. The timeout’s regression window is exact; f1dac6c is the strongest suspect, but a targeted rerun is required to prove the exact hanging lesson.