Photo3D · Alignment Questions · 2026-07-08
10 questions.
So we're building the same thing.
The contradictions are settled; these are the open intent questions that will shape the next weeks of autonomous work. Same format: options per question, my recommendation in green, reply with number–letter pairs.
Reply like before, e.g. 1C 2A 3B 4A 5B 6A 7A 8B 9A 10A. Add notes anywhere the flavor matters.
What does "the skill is learned" actually mean?
The vision says self-play should make the system better at the skill, not just better at six specific photos. Those are different finish lines: a system can grind one photo to 85 by overfitting and still fail a brand-new photo.
Per-scene mastery: every manifest photo reaches 85. Overfitting per photo is fine — each scene is its own product.
Generalization: the real test is a photo the system has never seen, hitting 85 with zero human help.
Staged: grind the manifest set to 85 first (proves the techniques exist), then a held-out never-seen photo is the graduation exam.recommended
Why C: the manifest grind discovers the playbook; the held-out exam proves it's a skill and not memorization. Same reason ML splits train from test.
When does Babylon.js enter the picture?
You've said Babylon is the potentially preferred engine for the final game-ready deliverable, while the whole judged pipeline (renderer, screenshots, genomes) runs on Three.js today.
Stay Three.js for the judged loop until a scene sustains ~70; then build a genome→Babylon exporter so the same scene runs in both. Judge keeps scoring Three.js renders.recommended
Port the renderer to Babylon now so we're never building on the wrong engine.
Three.js is fine as the permanent engine; drop the Babylon preference.
Why A: the score climb is engine-agnostic (it's about scene content), so switching now is churn with zero score payoff. An exporter later gets Babylon's game-engine strengths exactly when a scene is worth playing in.
What's the API spend ceiling for autonomous runs?
Each optimizer iteration costs roughly 2 vision calls (judge + editor, ~$0.10–0.25). "Run for a day or a week" at full tilt across 6 photos is hundreds to thousands of iterations — real money, unattended.
Frugal: ~$10/day cap. Slower climb, negligible burn.
Working budget: ~$50/day cap with a running spend line on the Live Board. Roughly 200–400 iterations/day.recommended
Uncapped until the first photo hits 85, then reassess.
Why B: enough throughput to learn something every single day, bounded enough that a stuck loop can't surprise you. The spend line keeps it honest.
Who runs the loop when nobody's watching?
Recursive self-improvement with no human in the loop needs an actual always-on driver. Today the optimizer runs when I kick it.
The existing photo3d cron routine becomes the Orchestrator's heartbeat: each fire runs optimizer passes, updates the hypotheses ledger, reports progress and spend.recommended
Continuous background process on AREA51 with a kill switch — maximum iterations, needs supervision plumbing built.
On-demand only: runs happen when you or I start them.
Why A: the cron infrastructure (reporting, dashboards, failure surfacing) already exists — the loop inherits supervision for free. B is where it can evolve once A proves stable.
Meshy models: per-photo conversions or a shared library?
Meshy conversions cost credits and time. The same object classes recur in every construction photo: boulders, stockpiles, trees, machines, mounds.
Build a shared reusable library — one great boulder, gravel pile, tree, excavator, etc. — and let the editor place/scale/tint them per photo. Convert something new only when a photo shows an object class we lack.recommended
Convert fresh models from each photo's own crops every time — maximum fidelity to that photo, maximum credit burn.
Why A: at judged render distances a good generic boulder is indistinguishable from a photo-specific one, and the library compounds — every conversion makes all future photos cheaper. B stays available for hero objects the judge specifically calls out.
What proves "navigable" at the milestone?
Ruling 1B says multi-angle becomes the metric after ~70 sustained. But which off-angle test defines "the scene holds up"?
Orbit test: the judge also scores a render from a camera orbited ~15–30° off the source pose; the scene's score is the worse of the two.recommended
Walkthrough test: score several ground-level viewpoints along a path through the scene.
Free-fly: sample random cameras anywhere in the scene volume.
Why A: it's the cheapest test that kills billboards dead (a projection that only works from one pose collapses at 20° off), and the orbit machinery already exists in the renderer. B and C are later graduations.
When do new photos join the set?
Ruling 8B fixed the official set at the manifest (6 photos). The open question is growth policy — more scenes means more diverse training signal but divides compute.
Freeze at the current 6 until one photo sustains 70+; then add photos deliberately, one at a time, picking scene types that stress untested skills.recommended
Grow now: add several diverse photos immediately so techniques never overfit to earthwork valleys.
Why A: right now every point of judge feedback should compound on the same scenes so hypotheses resolve fast. Diversity matters exactly when the playbook starts working — that's the moment to stress it.
When does the 5-page refactor happen?
The assessment stands: 18 pages, ~9k LOC of scripts, three duplicated servers, and the nav bug class only dies with a shared-component rebuild. But refactoring now competes with score-climbing.
Refactor now — clean foundation before more experiments pile up.
Refactor after the first 70+ scene: until then the pages are lab scaffolding, and we'll know which surfaces actually earned a place in the 5-page story.recommended
Don't refactor — keep patching the static pages.
Why B: the score is the product; the site is instrumentation. Refactoring instrumentation mid-experiment costs the thing you're measuring. The one exception already handled: nav/footer are now generated from shared patterns.
Does the app get its real name now?
50 names were generated; forgescene was the front-runner, forgetwin the dark horse. It's still photo3d everywhere.
Rename to forgescene at the refactor moment (Q8) — one rename, one migration, zero churn before then.recommended
Rename now — the name shapes how we talk about it.
Different name (say which — forgetwin, forgeworld, …).
Why A: renames touch every path, port registry, and cron reference; batching it with the refactor pays that tax once.
Is the judge pinned, and to what?
The judge IS the reward function. Today it's pinned to one model (claude-sonnet-4-6). Change the judge and every historical score stops being comparable; keep it forever and the reward ossifies below frontier quality.
Pin the judge for comparability; rebaseline deliberately at named milestones (e.g. when multi-angle turns on), re-scoring the current bests to bridge old and new trajectories.recommended
Always use the strongest available model as judge; accept trajectory breaks.
Ensemble: average 3 judge models to smooth single-model quirks — 3x judging cost.
Why A: a stable reward is what makes the trajectory chart mean something. Scheduled rebaselines get frontier quality without silently moving the goalposts mid-climb.
My full recommendation in one line
1C 2A 3B 4A 5A 6A 7A 8B 9A 10A