diff --git a/docs/cli.md b/docs/cli.md index b2873fe..27662c3 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -232,8 +232,9 @@ Application credential для idle-killer CLI создаёт на `up` (узко 1. Nova `ACTIVE` 2. TCP 22 / SSH 3. На VM: HTTP сервисов стека (SwarmUI `:7801` / Ollama `:11434` / llama.cpp `:8080`) — `verify_stack_on_vm` -4. Backend Idle (если SwarmUI) → toast (если `NOTIFY_READY`) -5. Туннель + проверка **localhost** тех же сервисов → access-card +4. На VM: **nvidia-smi / CUDA** (+ **torch+cuda** в Comfy venv, если SwarmUI) — `verify_gpu_env` +5. Backend Idle (если SwarmUI) → toast (если `NOTIFY_READY`) +6. Туннель + проверка **localhost** тех же сервисов → access-card Локальный порт UI: **17801** (на VM по-прежнему 7801 на loopback). diff --git a/docs/reviews/2026-08-21-review-3.md b/docs/reviews/2026-08-21-review-3.md new file mode 100644 index 0000000..b14e54e --- /dev/null +++ b/docs/reviews/2026-08-21-review-3.md @@ -0,0 +1,90 @@ +# Project Review — 2026-08-21 (3) + +Critical-only pass (мелочи / style / UX polish ignored). Focus: billing safety, silent GPU left on, security of secrets, broken stop/auto-stop paths. + +## Prior Reviews Summary + +> Based on `2026-08-21-review-2.md` and `2026-08-21-review-1.md`. + +### Still Open (carried forward) + +None. All tasks in review-1 and review-2 are `[x]`. + +### Resolved Since Last Review + +- [x] Prior review-2 closed the fail-closed idle-killer access_rules + revoke on stop, Ollama pull exact-tag, provision_llm not swallowed, etc. Those remain in place; this review finds **new** regressions/gaps around arm false-positive and unreachable Swarm. + +--- + +## Phase 1: Code Quality + +### SOLID + +No critical issues (god-flow in `session`/`provision` is known debt, not a break). + +### Performance + +No critical issues. + +### Correctness & Bugs + +1. **`arm_idle_killer` soft-fails on missing app cred, but provision still marks `armed`.** + `create_application_credential` failure is caught, logged (`idle-killer слеп`), and `return`s without raising. Caller always sets `notes["idle_killer"] = "armed"`. Access card only warns on `"failed"`. → User thinks auto-stop works; GPU bills forever. + +2. **Remote idle-killer treats any Swarm HTTP error as busy forever.** + `swarm_busy`: unreachable → `(True, "swarm unreachable…")`, which resets idle timer every tick. No grace/timeout to delete when Swarm is down for hours. → Crashed Swarm / bad Comfy ExtraArgs → never delete. + +3. **`provision_llm` / seed failures can leave ACTIVE compute before killer arm.** + Killer is armed last in `provision_vm`. Earlier `CloudError` aborts `up` with server already created and no timer. Known risk amplified by LLM/extensions paths. + +4. **Perf tune sets `pip_ok=True` even when `pip install triton/sageattention` fails**, then may still patch `--use-sage-attention` and restart Swarm. Marker prevents retry. → Can leave Swarm broken → compounds (2). + +### Code Quality + +No critical issues. + +--- + +## Phase 2: Logical Consistency + +### Domain & Application Layer + +No critical issues. + +### Data Flow + +Requires-filter for extensions / Ollama install paths / balance notify (notify-only) — no critical billing harm found. + +### State Management + +`notes.idle_killer = "armed"` does not match actual arm success (see bug 1). + +### Consistency + +No critical issues. + +--- + +## Phase 3: UI/UX + +### Usability + +Misleading “armed” status when killer is blind — treated as Bug/Logic above, not UX polish. + +### Visual / Interaction / Accessibility + +N/A for CLI critical pass (or no critical issues). + +--- + +## Tasks + +Critical only: + +- [ ] 1. [Bug] `arm_idle_killer`: on cred create failure **raise** or return False; never set `notes.idle_killer=armed`; surface same ⚠ as `"failed"` — `src/gpu_rent/idle_killer.py` line 151, `src/gpu_rent/provision.py` line 516 +- [ ] 2. [Bug] Idle-killer: if Swarm unreachable longer than N minutes (e.g. 2× idle or fixed 60m), treat as idle/allow delete (llm-only already bypasses) — `src/gpu_rent/remote/idle_killer.py` line 75 +- [ ] 3. [Logic] On mid-`up` failure after server create, arm killer anyway or fail loudly and refuse to leave session without killer / document mandatory `stop` — `src/gpu_rent/provision.py` / `session.py` +- [ ] 4. [Bug] Perf tune: set `pip_ok` only if pip succeeded; do **not** write `--use-sage-attention` ExtraArgs unless install OK (or allow retry when `pip_ok` false) — `src/gpu_rent/remote/tune_swarm_perf.py` line 109 +- [ ] 5. [Security] Write idle-killer creds JSON with `mode=0o600` before `mv` (same as GIT_TOKEN) — `src/gpu_rent/idle_killer.py` line 156 + +**Verified OK (critical):** `cmd_stop` delete path, tunnel Ctrl+C detach, local-watchdog stop on stale lease, `requires: ollama` filter, balance notify (toast only), Ollama unit bind to data dir, 127 tests green at review time. diff --git a/src/gpu_rent/ready.py b/src/gpu_rent/ready.py index 5ec22fb..bab51be 100644 --- a/src/gpu_rent/ready.py +++ b/src/gpu_rent/ready.py @@ -311,6 +311,107 @@ def _http_local(url: str, timeout: float = 3.0) -> tuple[bool, str]: return False, str(exc)[:160] +def verify_gpu_env( + cfg: Config, + host: str, + log: Log, + *, + timeout: float = 600.0, + poll_every: float = 15.0, + raise_on_fail: bool = True, +) -> list[ServiceCheck]: + """nvidia-smi / CUDA / torch(+cuda) in Comfy venv when SwarmUI is on.""" + from importlib.resources import files + + from gpu_rent.ssh_ops import run_python + + want_swarm = bool(getattr(cfg, "enable_swarmui", True)) + script = files("gpu_rent.remote").joinpath("stack_env_probe.py").read_text( + encoding="utf-8" + ) + swarm_flag = "1" if want_swarm else "0" + prefix = f"import os\nos.environ['GPU_RENT_CHECK_SWARM']={swarm_flag!r}\n" + log( + "проверка GPU-стека: nvidia-smi, CUDA" + + (", torch в Comfy venv" if want_swarm else " (llm-only — без torch)") + ) + + deadline = time.time() + timeout + last: list[ServiceCheck] = [] + while time.time() < deadline: + try: + out = run_python( + cfg, + host, + prefix + script, + remote_path="/tmp/gpu-rent-stack_env_probe.py", + timeout=180, + log=None, + ) + except Exception as exc: + last = [ServiceCheck("gpu-env", False, str(exc)[:200], "vm")] + log(f" … gpu-env: {exc}") + time.sleep(poll_every) + continue + + data: dict = {} + for line in reversed(out.splitlines()): + line = line.strip() + if line.startswith("{"): + try: + data = json.loads(line) + break + except json.JSONDecodeError: + continue + checks_raw = data.get("checks") if isinstance(data, dict) else None + if not isinstance(checks_raw, list): + last = [ServiceCheck("gpu-env", False, f"нет JSON: {out[-180:]}", "vm")] + time.sleep(poll_every) + continue + + last = [] + for item in checks_raw: + if not isinstance(item, dict): + continue + name = str(item.get("name") or "?") + ok = bool(item.get("ok")) + detail = str(item.get("detail") or "") + required = bool(item.get("required", True)) + if not required: + if not ok: + log(f" [warn] {name}: {detail}") + last.append( + ServiceCheck( + name, True, detail if ok else f"(optional) {detail}", "vm" + ) + ) + else: + last.append(ServiceCheck(name, ok, detail, "vm")) + + hard = [c for c in last if not c.ok] + if not hard: + for c in last: + log(f" [ok] {c.name}: {c.detail}") + log("проверка GPU-стека: ок") + return last + + bad = ", ".join(f"{c.name}={c.detail}" for c in hard) + log(f" … ещё нет: {bad}") + time.sleep(poll_every) + + for c in last: + mark = "ok" if c.ok else "FAIL" + log(f" [{mark}] {c.name}: {c.detail}") + if raise_on_fail and any(not c.ok for c in last): + failed = [c.name for c in last if not c.ok] + raise CloudError( + f"GPU-стек не готов за {int(timeout)} с: {', '.join(failed)}. " + "Нужны nvidia-smi, CUDA; для SwarmUI — torch с cuda в Comfy venv " + "(journalctl -u swarmui / первый старт backend)." + ) + return last + + def verify_stack_local( cfg: Config, log: Log, @@ -326,7 +427,11 @@ def verify_stack_local( targets.append(("swarmui", int(cfg.swarmui_local_port), None)) if want_ollama: targets.append( - ("ollama", int(cfg.ollama_local_port), f"http://127.0.0.1:{cfg.ollama_local_port}/api/tags") + ( + "ollama", + int(cfg.ollama_local_port), + f"http://127.0.0.1:{cfg.ollama_local_port}/api/tags", + ) ) if want_llama: p = int(cfg.llamacpp_local_port) @@ -335,7 +440,10 @@ def verify_stack_local( if not targets: return [] - log("проверка туннеля (localhost): " + ", ".join(f"{n}:{port}" for n, port, _ in targets)) + log( + "проверка туннеля (localhost): " + + ", ".join(f"{n}:{port}" for n, port, _ in targets) + ) deadline = time.time() + timeout last: list[ServiceCheck] = [] while time.time() < deadline: @@ -347,12 +455,9 @@ def verify_stack_local( if url: ok, detail = _http_local(url) if not ok and name == "llamacpp": - ok, detail = _http_local( - f"http://127.0.0.1:{port}/v1/models" - ) + ok, detail = _http_local(f"http://127.0.0.1:{port}/v1/models") last.append(ServiceCheck(name, ok, detail, "local")) else: - # SwarmUI: TCP enough (API may need POST); try API too ok, detail = _http_local(f"http://127.0.0.1:{port}/") if not ok: try: @@ -365,9 +470,7 @@ def verify_stack_local( with urllib.request.urlopen(req, timeout=4) as resp: ok = True detail = f"API HTTP {getattr(resp, 'status', 200)}" - except Exception as exc: - detail = f"TCP ok; HTTP {exc}"[:160] - # TCP open is enough for swarm local check + except Exception: ok = True detail = f"TCP :{port} open" last.append(ServiceCheck(name, ok, detail, "local")) diff --git a/src/gpu_rent/remote/stack_env_probe.py b/src/gpu_rent/remote/stack_env_probe.py new file mode 100644 index 0000000..cf10e26 --- /dev/null +++ b/src/gpu_rent/remote/stack_env_probe.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""Check NVIDIA driver / CUDA / PyTorch on the VM. Prints one JSON object.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +COMFY_PIPS = [ + Path("/opt/swarmui/dlbackend/comfy/venv/bin/python"), + Path("/opt/swarmui/dlbackend/comfy/ComfyUI/venv/bin/python"), +] + + +def run(argv: list[str], timeout: float = 30.0) -> tuple[int, str]: + try: + out = subprocess.check_output( + argv, text=True, stderr=subprocess.STDOUT, timeout=timeout + ) + return 0, out.strip() + except subprocess.CalledProcessError as exc: + return exc.returncode, (exc.output or "").strip() + except (OSError, subprocess.TimeoutExpired) as exc: + return 1, str(exc) + + +def check_nvidia() -> dict: + code, out = run(["nvidia-smi", "-L"]) + if code != 0 or not out: + return {"ok": False, "detail": out or "nvidia-smi missing/failed"} + # Driver / CUDA from nvidia-smi header + code2, q = run( + [ + "nvidia-smi", + "--query-gpu=driver_version,name,memory.total", + "--format=csv,noheader", + ] + ) + detail = out.splitlines()[0] if out else "ok" + if code2 == 0 and q: + detail = q.splitlines()[0].strip() + return {"ok": True, "detail": detail} + + +def check_cuda_runtime() -> dict: + # libcuda present? + code, out = run(["bash", "-lc", "ldconfig -p 2>/dev/null | grep -E 'libcuda\\.so' | head -n1"]) + if code == 0 and out: + return {"ok": True, "detail": out.strip()} + # Fallback: nvidia-smi reports CUDA Version + code2, smi = run(["nvidia-smi"]) + if code2 == 0 and "CUDA Version" in smi: + for line in smi.splitlines(): + if "CUDA Version" in line: + return {"ok": True, "detail": line.strip()[:120]} + return {"ok": False, "detail": "libcuda.so не найден (драйвер/CUDA runtime)"} + + +def find_comfy_python() -> Path | None: + for p in COMFY_PIPS: + if p.is_file() and os.access(p, os.X_OK): + return p + return None + + +def check_torch(py: Path) -> dict: + script = ( + "import json,sys\n" + "try:\n" + " import torch\n" + "except Exception as e:\n" + " print(json.dumps({'ok':False,'detail':f'import torch: {e}'})); sys.exit(0)\n" + "cuda=bool(torch.cuda.is_available())\n" + "dev=torch.cuda.get_device_name(0) if cuda else ''\n" + "print(json.dumps({\n" + " 'ok': cuda,\n" + " 'detail': (\n" + " f'torch={torch.__version__} cuda={torch.version.cuda} '\n" + " f'available={cuda} device={dev}'\n" + " ),\n" + "}))\n" + ) + code, out = run([str(py), "-c", script], timeout=120.0) + line = "" + for row in reversed((out or "").splitlines()): + row = row.strip() + if row.startswith("{"): + line = row + break + if not line: + return {"ok": False, "detail": f"torch probe fail: {(out or '')[:200]}"} + try: + return json.loads(line) + except json.JSONDecodeError: + return {"ok": False, "detail": f"bad torch JSON: {line[:160]}"} + + +def check_optional_pip(py: Path, names: list[str]) -> dict: + missing = [] + present = [] + for name in names: + code, _ = run([str(py), "-c", f"import {name}"], timeout=30.0) + if code == 0: + present.append(name) + else: + missing.append(name) + return { + "ok": True, # optional — never hard-fail + "detail": ( + f"есть: {', '.join(present) or '—'}; нет: {', '.join(missing) or '—'}" + ), + "missing": missing, + "present": present, + } + + +def main() -> int: + want_swarm = (os.environ.get("GPU_RENT_CHECK_SWARM") or "1").strip() != "0" + checks: list[dict] = [] + + nv = check_nvidia() + checks.append({"name": "nvidia-smi", "required": True, **nv}) + + cuda = check_cuda_runtime() + # CUDA runtime required whenever we have a GPU session + checks.append({"name": "cuda", "required": True, **cuda}) + + if want_swarm: + py = find_comfy_python() + if py is None: + checks.append( + { + "name": "torch", + "required": True, + "ok": False, + "detail": "ComfyUI venv python не найден (ещё не поставился?)", + } + ) + else: + torch_c = check_torch(py) + checks.append({"name": "torch", "required": True, "venv": str(py), **torch_c}) + opt = check_optional_pip(py, ["triton", "sageattention"]) + checks.append({"name": "triton/sage", "required": False, **opt}) + else: + checks.append( + { + "name": "torch", + "required": False, + "ok": True, + "detail": "skip (llm-only, без Comfy venv)", + } + ) + + required_ok = all(c.get("ok") for c in checks if c.get("required")) + print( + json.dumps( + {"ok": required_ok, "checks": checks}, + ensure_ascii=False, + ) + ) + return 0 + + +if __name__ == "__main__": + # Always 0 — caller polls JSON `ok` (torch may appear after first Comfy start). + main() + sys.exit(0) diff --git a/src/gpu_rent/session.py b/src/gpu_rent/session.py index 55faa95..baa144d 100644 --- a/src/gpu_rent/session.py +++ b/src/gpu_rent/session.py @@ -22,7 +22,7 @@ from gpu_rent.cloud import ( ) from gpu_rent.bootstrap import run_bootstrap from gpu_rent.provision import provision_vm, tune_swarm_perf -from gpu_rent.ready import verify_stack_on_vm, wait_backend_idle +from gpu_rent.ready import verify_gpu_env, verify_stack_on_vm, wait_backend_idle from gpu_rent.snapshot import ensure_boot_snapshot from gpu_rent.notify import notify_ready from gpu_rent.config import Config @@ -140,6 +140,18 @@ def _bind_access( save_state(state) raise + try: + gpu_checks = verify_gpu_env(cfg, ip, log, timeout=600.0) + state.notes = dict(state.notes or {}) + state.notes["gpu_env"] = [ + {"name": c.name, "ok": c.ok, "detail": c.detail} for c in gpu_checks + ] + except CloudError as exc: + state.notes = dict(state.notes or {}) + state.notes["gpu_env_error"] = str(exc)[:500] + save_state(state) + raise + try: ensure_boot_snapshot( conn, diff --git a/tests/test_session.py b/tests/test_session.py index 10525e2..b9195cc 100644 --- a/tests/test_session.py +++ b/tests/test_session.py @@ -48,6 +48,18 @@ def _mock_bind(monkeypatch): ) monkeypatch.setattr("gpu_rent.session.provision_vm", lambda cfg, host, log, **kw: None) monkeypatch.setattr("gpu_rent.session.wait_backend_idle", lambda cfg, host, log, **kw: None) + monkeypatch.setattr( + "gpu_rent.session.verify_stack_on_vm", + lambda cfg, host, log, **kw: [], + ) + monkeypatch.setattr( + "gpu_rent.session.verify_gpu_env", + lambda cfg, host, log, **kw: [], + ) + monkeypatch.setattr( + "gpu_rent.session.tune_swarm_perf", + lambda cfg, host, log: False, + ) monkeypatch.setattr( "gpu_rent.session.ensure_boot_snapshot", lambda conn, boot_volume_id, cfg, log: None, diff --git a/tests/test_verify_stack.py b/tests/test_verify_stack.py index 90bd3ad..1c5b458 100644 --- a/tests/test_verify_stack.py +++ b/tests/test_verify_stack.py @@ -1,4 +1,5 @@ -from gpu_rent.ready import ServiceCheck, _expected_services, verify_stack_local +from gpu_rent.errors import CloudError +from gpu_rent.ready import ServiceCheck, _expected_services, verify_gpu_env, verify_stack_local class _Cfg: @@ -41,7 +42,9 @@ def test_verify_stack_local_fails_closed_port(monkeypatch): ollama_local_port = 17999 llamacpp_local_port = 17812 - monkeypatch.setattr("gpu_rent.ready._tcp_ok", lambda port, host="127.0.0.1", timeout=0.8: False) + monkeypatch.setattr( + "gpu_rent.ready._tcp_ok", lambda port, host="127.0.0.1", timeout=0.8: False + ) logs: list[str] = [] try: verify_stack_local(C(), logs.append, timeout=0.3, poll_every=0.1) @@ -53,3 +56,86 @@ def test_verify_stack_local_fails_closed_port(monkeypatch): def test_service_check_dataclass(): c = ServiceCheck("x", True, "ok", "vm") assert c.ok and c.where == "vm" + + +def test_verify_gpu_env_ok(monkeypatch): + payload = { + "ok": True, + "checks": [ + {"name": "nvidia-smi", "required": True, "ok": True, "detail": "A100"}, + {"name": "cuda", "required": True, "ok": True, "detail": "libcuda"}, + { + "name": "torch", + "required": True, + "ok": True, + "detail": "torch=2.0 cuda=12 available=True", + }, + { + "name": "triton/sage", + "required": False, + "ok": True, + "detail": "есть: triton; нет: —", + }, + ], + } + + def fake_run_python(cfg, host, script, **kw): + assert "GPU_RENT_CHECK_SWARM" in script + return json.dumps(payload) + "\n" + + import json + + monkeypatch.setattr("gpu_rent.ssh_ops.run_python", fake_run_python) + # patch where used + monkeypatch.setattr( + "gpu_rent.ssh_ops.run_python", + fake_run_python, + raising=False, + ) + + import gpu_rent.ready as ready_mod + + monkeypatch.setattr( + ready_mod, + "run_python", + fake_run_python, + raising=False, + ) + + # verify_gpu_env imports run_python inside the function + import gpu_rent.ssh_ops as ssh_ops + + monkeypatch.setattr(ssh_ops, "run_python", fake_run_python) + + logs: list[str] = [] + out = verify_gpu_env(_Cfg(), "1.2.3.4", logs.append, timeout=5.0, poll_every=0.1) + assert all(c.ok for c in out) + assert any(c.name == "torch" for c in out) + + +def test_verify_gpu_env_fails_without_cuda(monkeypatch): + import json + + import gpu_rent.ssh_ops as ssh_ops + + payload = { + "ok": False, + "checks": [ + {"name": "nvidia-smi", "required": True, "ok": True, "detail": "ok"}, + {"name": "cuda", "required": True, "ok": False, "detail": "нет libcuda"}, + {"name": "torch", "required": True, "ok": False, "detail": "no venv"}, + ], + } + monkeypatch.setattr( + ssh_ops, + "run_python", + lambda *a, **k: json.dumps(payload), + ) + logs: list[str] = [] + try: + verify_gpu_env( + _Cfg(), "1.2.3.4", logs.append, timeout=0.4, poll_every=0.1 + ) + assert False, "expected CloudError" + except CloudError as exc: + assert "cuda" in str(exc).lower() or "GPU-стек" in str(exc)