Leonid Pershin
1785ab369c
- Updated the idle-killer logic to treat SwarmUI `empty` and `disabled` states as busy, preventing unnecessary idle time during provisioning. - Enhanced the `wait_backend_idle` function to recognize suspended backends as ready, improving resource utilization and user feedback. - Refined the `install_swarm_comfy` script to skip installation when backends are already present, streamlining the setup process. - Improved the `resolve_llm_runtime` function to prioritize live configuration over stale state notes, ensuring accurate runtime detection. - Added tests to validate the new backend status handling and idle management logic, ensuring robustness and reliability.
99 lines
5.4 KiB
Markdown
99 lines
5.4 KiB
Markdown
# Project Review — 2026-08-21 (4)
|
||
|
||
Фокус: скрытые логические инверсии / wait-forever / billing (как idle≠running).
|
||
|
||
## Prior Reviews Summary
|
||
|
||
> Based on `2026-08-21-review-3`, `review-2`, `review-1`.
|
||
|
||
### Still Open (carried forward)
|
||
None.
|
||
|
||
### Resolved Since Last Review
|
||
None newly verified in this pass (prior reviews already all `[x]`).
|
||
|
||
---
|
||
|
||
## Phase 1: Code Quality
|
||
|
||
### SOLID
|
||
No issues found.
|
||
|
||
### Performance
|
||
No issues found beyond existing network-disk `du` / walk costs (acceptable).
|
||
|
||
### Correctness & Bugs
|
||
|
||
1. **Bug (critical)** — idle-killer treats SwarmUI `backend_status=empty` as not busy — `src/gpu_rent/remote/idle_killer.py` `swarm_busy()`. During first Comfy install (15–40+ min) status can stay `empty`/`unknown` with empty queue; after grace the idle clock can delete the VM mid-install. Same class as treating wrong Swarm status as “done”.
|
||
|
||
2. **Bug (critical)** — Ollama pull marker expires at 45m while pull timeout is 7200s — `src/gpu_rent/remote/idle_killer.py` `llm_busy()` + `src/gpu_rent/remote/ollama_pull.py`. Marker timestamp is written once and never refreshed; after 45m killer clears it and may stop GPU during a long pull.
|
||
|
||
3. **Bug (critical)** — `seed_autocomplete` can overwrite entire `Settings.fds` — `src/gpu_rent/provision.py` `seed_autocomplete()`. If meta exists without `settings_applied=true`, `put_text` replaces Settings with only AutoComplete stub — wipes `IsInstalled` / other settings.
|
||
|
||
4. **Bug (high)** — `wait_backend_idle` treats SwarmUI `idle` as forever BUSY with no wake — `src/gpu_rent/ready.py`. After auto-suspend, backends report `idle` (cannot generate) and never become `running` without a wake; `up` can burn the full timeout.
|
||
|
||
5. **Bug (high)** — `install_swarm_comfy` skip path still labels Swarm `idle` as “already Idle” — `src/gpu_rent/remote/install_swarm_comfy.py` `main()`. Misleading; should skip because backends exist, then align with wake/`running` wait.
|
||
|
||
6. **Bug (high)** — local tunnel verify forces OK on TCP-only — `src/gpu_rent/ready.py` `verify_stack_local()`. If `GET /` and API fail, still `ok=True` with `TCP :port open` — marks tunnel ready when SwarmUI is dead.
|
||
|
||
7. **Bug (high)** — torch without CUDA polls 600s — `src/gpu_rent/remote/stack_env_probe.py` + `verify_gpu_env`. Instant-fail covers driver/libcuda/empty dlbackend, but not “torch imported, cuda=False” (CPU wheel won’t heal).
|
||
|
||
8. **Bug (high)** — perf tune execs `venv/bin/pip` → `OSError: Exec format error` — `src/gpu_rent/remote/tune_swarm_perf.py` `pip_install_sage()`. Observed on live up; OSError not caught → script exit 1; sageattention missing. **Fixing in this session via `python -m pip`.**
|
||
|
||
9. **Bug (medium)** — access card / open use `resolve_llm_runtime` (notes override) but tunnel uses cfg only — `access_card.py` vs `tunnel.py`. Can advertise Ollama URLs with no forward.
|
||
|
||
10. **Bug (medium)** — `status` tunnel check only probes SwarmUI local port — `cli.py`. llm-only looks like “туннель нет”.
|
||
|
||
11. **Bug (medium)** — tunnel Nova poll soft-fail returns fake `ACTIVE` — `tunnel.py` `_poll_nova()`. Masks DELETED/ERROR.
|
||
|
||
### Code Quality
|
||
No separate issues beyond the above.
|
||
|
||
---
|
||
|
||
## Phase 2: Logical Consistency
|
||
|
||
### Domain & Application Layer
|
||
No issues found.
|
||
|
||
### Data Flow
|
||
See #9 (cfg vs notes for LLM runtime).
|
||
|
||
### State Management
|
||
See #1–#2 (busy markers / status semantics).
|
||
|
||
### Consistency
|
||
SwarmUI status vocabulary (`running` / `idle` / `empty` / `loading`) was inconsistent across wait vs killer; `running`=ready is fixed, remaining gaps in #1/#4/#5.
|
||
|
||
---
|
||
|
||
## Phase 3: UI/UX
|
||
|
||
### Usability
|
||
Access card correctly deferred until tunnel (recent change). Mid-up wait messages improved.
|
||
|
||
### Visual & Consistency
|
||
No issues found.
|
||
|
||
### Interaction & Feedback
|
||
Perf tune failure was noisy traceback then continued — should be WARN without stack (addressed with OSError catch + `-m pip`).
|
||
|
||
### Accessibility
|
||
N/A (CLI).
|
||
|
||
---
|
||
|
||
## Tasks
|
||
|
||
- [x] 1. [Bug] Treat SwarmUI `empty` (and likely `disabled` during provision) as busy in idle-killer — `src/gpu_rent/remote/idle_killer.py` `swarm_busy`
|
||
- [x] 2. [Bug] Refresh `.gpu-rent-ollama-pulling` during pull and/or raise stale max-age ≥ pull timeout (7200s) — `src/gpu_rent/remote/idle_killer.py` + `ollama_pull.py`
|
||
- [x] 3. [Bug] Never overwrite full `Settings.fds` in autocomplete seed — merge AutoComplete only — `src/gpu_rent/provision.py` `seed_autocomplete`
|
||
- [x] 4. [Bug] On wait, wake suspended backends or accept healthy path when status=`idle` after install; don’t burn 2400s — `src/gpu_rent/ready.py` `wait_backend_idle`
|
||
- [x] 5. [Bug] Fix `install_swarm_comfy` skip wording/logic for `idle` vs backends-present — `src/gpu_rent/remote/install_swarm_comfy.py`
|
||
- [x] 6. [Bug] `verify_stack_local`: do not mark SwarmUI ok on TCP-only when HTTP/API fail — `src/gpu_rent/ready.py`
|
||
- [x] 7. [Bug] Fail-fast when Comfy torch imports but `cuda=False` — `stack_env_probe.py` / `verify_gpu_env`
|
||
- [x] 8. [Bug] Perf tune: use `python -m pip` (not `venv/bin/pip`) and catch OSError — `src/gpu_rent/remote/tune_swarm_perf.py`
|
||
- [x] 9. [Logic] Single source of truth for LLM runtime in access card vs tunnel — `access_card.py` / `tunnel.py`
|
||
- [x] 10. [UX] `status` should probe ports from `tunnel_forwards(cfg)` — `cli.py`
|
||
- [x] 11. [Bug] Tunnel Nova soft-fail must not fake ACTIVE forever — `tunnel.py` `_poll_nova`
|