Implement GPU environment verification in session management

- Added a new function `verify_gpu_env` to check GPU stack readiness, including nvidia-smi, CUDA, and torch in the Comfy virtual environment when SwarmUI is enabled.
- Updated the session management to call `verify_gpu_env`, capturing GPU environment status and errors in the state notes.
- Enhanced documentation in `cli.md` to reflect the new GPU environment verification process.
- Added tests for `verify_gpu_env` to ensure proper functionality and error handling during GPU checks.

docs/cli.md

@@ -232,8 +232,9 @@ Application credential для idle-killer CLI создаёт на `up` (узко
 1. Nova `ACTIVE`
 2. TCP 22 / SSH
 3. На VM: HTTP сервисов стека (SwarmUI `:7801` / Ollama `:11434` / llama.cpp `:8080`) — `verify_stack_on_vm`
-4. Backend Idle (если SwarmUI) → toast (если `NOTIFY_READY`)
+4. На VM: **nvidia-smi / CUDA** (+ **torch+cuda** в Comfy venv, если SwarmUI) — `verify_gpu_env`
-5. Туннель + проверка **localhost** тех же сервисов → access-card
+5. Backend Idle (если SwarmUI) → toast (если `NOTIFY_READY`)
+6. Туннель + проверка **localhost** тех же сервисов → access-card
 
 Локальный порт UI: **17801** (на VM по-прежнему 7801 на loopback).
 

docs/reviews/2026-08-21-review-3.md

@@ -0,0 +1,90 @@
+# Project Review — 2026-08-21 (3)
+
+Critical-only pass (мелочи / style / UX polish ignored). Focus: billing safety, silent GPU left on, security of secrets, broken stop/auto-stop paths.
+
+## Prior Reviews Summary
+
+> Based on `2026-08-21-review-2.md` and `2026-08-21-review-1.md`.
+
+### Still Open (carried forward)
+
+None. All tasks in review-1 and review-2 are `[x]`.
+
+### Resolved Since Last Review
+
+- [x] Prior review-2 closed the fail-closed idle-killer access_rules + revoke on stop, Ollama pull exact-tag, provision_llm not swallowed, etc. Those remain in place; this review finds **new** regressions/gaps around arm false-positive and unreachable Swarm.
+
+---
+
+## Phase 1: Code Quality
+
+### SOLID
+
+No critical issues (god-flow in `session`/`provision` is known debt, not a break).
+
+### Performance
+
+No critical issues.
+
+### Correctness & Bugs
+
+1. **`arm_idle_killer` soft-fails on missing app cred, but provision still marks `armed`.**  
+   `create_application_credential` failure is caught, logged (`idle-killer слеп`), and `return`s without raising. Caller always sets `notes["idle_killer"] = "armed"`. Access card only warns on `"failed"`. → User thinks auto-stop works; GPU bills forever.
+
+2. **Remote idle-killer treats any Swarm HTTP error as busy forever.**  
+   `swarm_busy`: unreachable → `(True, "swarm unreachable…")`, which resets idle timer every tick. No grace/timeout to delete when Swarm is down for hours. → Crashed Swarm / bad Comfy ExtraArgs → never delete.
+
+3. **`provision_llm` / seed failures can leave ACTIVE compute before killer arm.**  
+   Killer is armed last in `provision_vm`. Earlier `CloudError` aborts `up` with server already created and no timer. Known risk amplified by LLM/extensions paths.
+
+4. **Perf tune sets `pip_ok=True` even when `pip install triton/sageattention` fails**, then may still patch `--use-sage-attention` and restart Swarm. Marker prevents retry. → Can leave Swarm broken → compounds (2).
+
+### Code Quality
+
+No critical issues.
+
+---
+
+## Phase 2: Logical Consistency
+
+### Domain & Application Layer
+
+No critical issues.
+
+### Data Flow
+
+Requires-filter for extensions / Ollama install paths / balance notify (notify-only) — no critical billing harm found.
+
+### State Management
+
+`notes.idle_killer = "armed"` does not match actual arm success (see bug 1).
+
+### Consistency
+
+No critical issues.
+
+---
+
+## Phase 3: UI/UX
+
+### Usability
+
+Misleading “armed” status when killer is blind — treated as Bug/Logic above, not UX polish.
+
+### Visual / Interaction / Accessibility
+
+N/A for CLI critical pass (or no critical issues).
+
+---
+
+## Tasks
+
+Critical only:
+
+- [ ] 1. [Bug] `arm_idle_killer`: on cred create failure **raise** or return False; never set `notes.idle_killer=armed`; surface same ⚠ as `"failed"` — `src/gpu_rent/idle_killer.py` line 151, `src/gpu_rent/provision.py` line 516
+- [ ] 2. [Bug] Idle-killer: if Swarm unreachable longer than N minutes (e.g. 2× idle or fixed 60m), treat as idle/allow delete (llm-only already bypasses) — `src/gpu_rent/remote/idle_killer.py` line 75
+- [ ] 3. [Logic] On mid-`up` failure after server create, arm killer anyway or fail loudly and refuse to leave session without killer / document mandatory `stop` — `src/gpu_rent/provision.py` / `session.py`
+- [ ] 4. [Bug] Perf tune: set `pip_ok` only if pip succeeded; do **not** write `--use-sage-attention` ExtraArgs unless install OK (or allow retry when `pip_ok` false) — `src/gpu_rent/remote/tune_swarm_perf.py` line 109
+- [ ] 5. [Security] Write idle-killer creds JSON with `mode=0o600` before `mv` (same as GIT_TOKEN) — `src/gpu_rent/idle_killer.py` line 156
+
+**Verified OK (critical):** `cmd_stop` delete path, tunnel Ctrl+C detach, local-watchdog stop on stale lease, `requires: ollama` filter, balance notify (toast only), Ollama unit bind to data dir, 127 tests green at review time.

src/gpu_rent/ready.py

@@ -311,6 +311,107 @@ def _http_local(url: str, timeout: float = 3.0) -> tuple[bool, str]:
         return False, str(exc)[:160]
 
 
+def verify_gpu_env(
+    cfg: Config,
+    host: str,
+    log: Log,
+    *,
+    timeout: float = 600.0,
+    poll_every: float = 15.0,
+    raise_on_fail: bool = True,
+) -> list[ServiceCheck]:
+    """nvidia-smi / CUDA / torch(+cuda) in Comfy venv when SwarmUI is on."""
+    from importlib.resources import files
+
+    from gpu_rent.ssh_ops import run_python
+
+    want_swarm = bool(getattr(cfg, "enable_swarmui", True))
+    script = files("gpu_rent.remote").joinpath("stack_env_probe.py").read_text(
+        encoding="utf-8"
+    )
+    swarm_flag = "1" if want_swarm else "0"
+    prefix = f"import os\nos.environ['GPU_RENT_CHECK_SWARM']={swarm_flag!r}\n"
+    log(
+        "проверка GPU-стека: nvidia-smi, CUDA"
+        + (", torch в Comfy venv" if want_swarm else " (llm-only — без torch)")
+    )
+
+    deadline = time.time() + timeout
+    last: list[ServiceCheck] = []
+    while time.time() < deadline:
+        try:
+            out = run_python(
+                cfg,
+                host,
+                prefix + script,
+                remote_path="/tmp/gpu-rent-stack_env_probe.py",
+                timeout=180,
+                log=None,
+            )
+        except Exception as exc:
+            last = [ServiceCheck("gpu-env", False, str(exc)[:200], "vm")]
+            log(f"  … gpu-env: {exc}")
+            time.sleep(poll_every)
+            continue
+
+        data: dict = {}
+        for line in reversed(out.splitlines()):
+            line = line.strip()
+            if line.startswith("{"):
+                try:
+                    data = json.loads(line)
+                    break
+                except json.JSONDecodeError:
+                    continue
+        checks_raw = data.get("checks") if isinstance(data, dict) else None
+        if not isinstance(checks_raw, list):
+            last = [ServiceCheck("gpu-env", False, f"нет JSON: {out[-180:]}", "vm")]
+            time.sleep(poll_every)
+            continue
+
+        last = []
+        for item in checks_raw:
+            if not isinstance(item, dict):
+                continue
+            name = str(item.get("name") or "?")
+            ok = bool(item.get("ok"))
+            detail = str(item.get("detail") or "")
+            required = bool(item.get("required", True))
+            if not required:
+                if not ok:
+                    log(f"  [warn] {name}: {detail}")
+                last.append(
+                    ServiceCheck(
+                        name, True, detail if ok else f"(optional) {detail}", "vm"
+                    )
+                )
+            else:
+                last.append(ServiceCheck(name, ok, detail, "vm"))
+
+        hard = [c for c in last if not c.ok]
+        if not hard:
+            for c in last:
+                log(f"  [ok] {c.name}: {c.detail}")
+            log("проверка GPU-стека: ок")
+            return last
+
+        bad = ", ".join(f"{c.name}={c.detail}" for c in hard)
+        log(f"  … ещё нет: {bad}")
+        time.sleep(poll_every)
+
+    for c in last:
+        mark = "ok" if c.ok else "FAIL"
+        log(f"  [{mark}] {c.name}: {c.detail}")
+    if raise_on_fail and any(not c.ok for c in last):
+        failed = [c.name for c in last if not c.ok]
+        raise CloudError(
+            f"GPU-стек не готов за {int(timeout)} с: {', '.join(failed)}. "
+            "Нужны nvidia-smi, CUDA; для SwarmUI — torch с cuda в Comfy venv "
+            "(journalctl -u swarmui / первый старт backend)."
+        )
+    return last
+
+
 def verify_stack_local(
     cfg: Config,
     log: Log,
@@ -326,7 +427,11 @@ def verify_stack_local(
         targets.append(("swarmui", int(cfg.swarmui_local_port), None))
     if want_ollama:
         targets.append(
-            ("ollama", int(cfg.ollama_local_port), f"http://127.0.0.1:{cfg.ollama_local_port}/api/tags")
+            (
+                "ollama",
+                int(cfg.ollama_local_port),
+                f"http://127.0.0.1:{cfg.ollama_local_port}/api/tags",
+            )
         )
     if want_llama:
         p = int(cfg.llamacpp_local_port)
@@ -335,7 +440,10 @@ def verify_stack_local(
     if not targets:
         return []
 
-    log("проверка туннеля (localhost): " + ", ".join(f"{n}:{port}" for n, port, _ in targets))
+    log(
+        "проверка туннеля (localhost): "
+        + ", ".join(f"{n}:{port}" for n, port, _ in targets)
+    )
     deadline = time.time() + timeout
     last: list[ServiceCheck] = []
     while time.time() < deadline:
@@ -347,12 +455,9 @@ def verify_stack_local(
             if url:
                 ok, detail = _http_local(url)
                 if not ok and name == "llamacpp":
-                    ok, detail = _http_local(
+                    ok, detail = _http_local(f"http://127.0.0.1:{port}/v1/models")
-                        f"http://127.0.0.1:{port}/v1/models"
-                    )
                 last.append(ServiceCheck(name, ok, detail, "local"))
             else:
-                # SwarmUI: TCP enough (API may need POST); try API too
                 ok, detail = _http_local(f"http://127.0.0.1:{port}/")
                 if not ok:
                     try:
@@ -365,9 +470,7 @@ def verify_stack_local(
                         with urllib.request.urlopen(req, timeout=4) as resp:
                             ok = True
                             detail = f"API HTTP {getattr(resp, 'status', 200)}"
-                    except Exception as exc:
+                    except Exception:
-                        detail = f"TCP ok; HTTP {exc}"[:160]
-                        # TCP open is enough for swarm local check
                         ok = True
                         detail = f"TCP :{port} open"
                 last.append(ServiceCheck(name, ok, detail, "local"))

src/gpu_rent/remote/stack_env_probe.py

@@ -0,0 +1,170 @@
+#!/usr/bin/env python3
+"""Check NVIDIA driver / CUDA / PyTorch on the VM. Prints one JSON object."""
+
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+COMFY_PIPS = [
+    Path("/opt/swarmui/dlbackend/comfy/venv/bin/python"),
+    Path("/opt/swarmui/dlbackend/comfy/ComfyUI/venv/bin/python"),
+]
+
+
+def run(argv: list[str], timeout: float = 30.0) -> tuple[int, str]:
+    try:
+        out = subprocess.check_output(
+            argv, text=True, stderr=subprocess.STDOUT, timeout=timeout
+        )
+        return 0, out.strip()
+    except subprocess.CalledProcessError as exc:
+        return exc.returncode, (exc.output or "").strip()
+    except (OSError, subprocess.TimeoutExpired) as exc:
+        return 1, str(exc)
+
+
+def check_nvidia() -> dict:
+    code, out = run(["nvidia-smi", "-L"])
+    if code != 0 or not out:
+        return {"ok": False, "detail": out or "nvidia-smi missing/failed"}
+    # Driver / CUDA from nvidia-smi header
+    code2, q = run(
+        [
+            "nvidia-smi",
+            "--query-gpu=driver_version,name,memory.total",
+            "--format=csv,noheader",
+        ]
+    )
+    detail = out.splitlines()[0] if out else "ok"
+    if code2 == 0 and q:
+        detail = q.splitlines()[0].strip()
+    return {"ok": True, "detail": detail}
+
+
+def check_cuda_runtime() -> dict:
+    # libcuda present?
+    code, out = run(["bash", "-lc", "ldconfig -p 2>/dev/null | grep -E 'libcuda\\.so' | head -n1"])
+    if code == 0 and out:
+        return {"ok": True, "detail": out.strip()}
+    # Fallback: nvidia-smi reports CUDA Version
+    code2, smi = run(["nvidia-smi"])
+    if code2 == 0 and "CUDA Version" in smi:
+        for line in smi.splitlines():
+            if "CUDA Version" in line:
+                return {"ok": True, "detail": line.strip()[:120]}
+    return {"ok": False, "detail": "libcuda.so не найден (драйвер/CUDA runtime)"}
+
+
+def find_comfy_python() -> Path | None:
+    for p in COMFY_PIPS:
+        if p.is_file() and os.access(p, os.X_OK):
+            return p
+    return None
+
+
+def check_torch(py: Path) -> dict:
+    script = (
+        "import json,sys\n"
+        "try:\n"
+        " import torch\n"
+        "except Exception as e:\n"
+        " print(json.dumps({'ok':False,'detail':f'import torch: {e}'})); sys.exit(0)\n"
+        "cuda=bool(torch.cuda.is_available())\n"
+        "dev=torch.cuda.get_device_name(0) if cuda else ''\n"
+        "print(json.dumps({\n"
+        " 'ok': cuda,\n"
+        " 'detail': (\n"
+        "  f'torch={torch.__version__} cuda={torch.version.cuda} '\n"
+        "  f'available={cuda} device={dev}'\n"
+        " ),\n"
+        "}))\n"
+    )
+    code, out = run([str(py), "-c", script], timeout=120.0)
+    line = ""
+    for row in reversed((out or "").splitlines()):
+        row = row.strip()
+        if row.startswith("{"):
+            line = row
+            break
+    if not line:
+        return {"ok": False, "detail": f"torch probe fail: {(out or '')[:200]}"}
+    try:
+        return json.loads(line)
+    except json.JSONDecodeError:
+        return {"ok": False, "detail": f"bad torch JSON: {line[:160]}"}
+
+
+def check_optional_pip(py: Path, names: list[str]) -> dict:
+    missing = []
+    present = []
+    for name in names:
+        code, _ = run([str(py), "-c", f"import {name}"], timeout=30.0)
+        if code == 0:
+            present.append(name)
+        else:
+            missing.append(name)
+    return {
+        "ok": True,  # optional — never hard-fail
+        "detail": (
+            f"есть: {', '.join(present) or '—'}; нет: {', '.join(missing) or '—'}"
+        ),
+        "missing": missing,
+        "present": present,
+    }
+
+
+def main() -> int:
+    want_swarm = (os.environ.get("GPU_RENT_CHECK_SWARM") or "1").strip() != "0"
+    checks: list[dict] = []
+
+    nv = check_nvidia()
+    checks.append({"name": "nvidia-smi", "required": True, **nv})
+
+    cuda = check_cuda_runtime()
+    # CUDA runtime required whenever we have a GPU session
+    checks.append({"name": "cuda", "required": True, **cuda})
+
+    if want_swarm:
+        py = find_comfy_python()
+        if py is None:
+            checks.append(
+                {
+                    "name": "torch",
+                    "required": True,
+                    "ok": False,
+                    "detail": "ComfyUI venv python не найден (ещё не поставился?)",
+                }
+            )
+        else:
+            torch_c = check_torch(py)
+            checks.append({"name": "torch", "required": True, "venv": str(py), **torch_c})
+            opt = check_optional_pip(py, ["triton", "sageattention"])
+            checks.append({"name": "triton/sage", "required": False, **opt})
+    else:
+        checks.append(
+            {
+                "name": "torch",
+                "required": False,
+                "ok": True,
+                "detail": "skip (llm-only, без Comfy venv)",
+            }
+        )
+
+    required_ok = all(c.get("ok") for c in checks if c.get("required"))
+    print(
+        json.dumps(
+            {"ok": required_ok, "checks": checks},
+            ensure_ascii=False,
+        )
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    # Always 0 — caller polls JSON `ok` (torch may appear after first Comfy start).
+    main()
+    sys.exit(0)

src/gpu_rent/session.py

@@ -22,7 +22,7 @@ from gpu_rent.cloud import (
 )
 from gpu_rent.bootstrap import run_bootstrap
 from gpu_rent.provision import provision_vm, tune_swarm_perf
-from gpu_rent.ready import verify_stack_on_vm, wait_backend_idle
+from gpu_rent.ready import verify_gpu_env, verify_stack_on_vm, wait_backend_idle
 from gpu_rent.snapshot import ensure_boot_snapshot
 from gpu_rent.notify import notify_ready
 from gpu_rent.config import Config
@@ -140,6 +140,18 @@ def _bind_access(
         save_state(state)
         raise
 
+    try:
+        gpu_checks = verify_gpu_env(cfg, ip, log, timeout=600.0)
+        state.notes = dict(state.notes or {})
+        state.notes["gpu_env"] = [
+            {"name": c.name, "ok": c.ok, "detail": c.detail} for c in gpu_checks
+        ]
+    except CloudError as exc:
+        state.notes = dict(state.notes or {})
+        state.notes["gpu_env_error"] = str(exc)[:500]
+        save_state(state)
+        raise
+
     try:
         ensure_boot_snapshot(
             conn,

tests/test_session.py

@@ -48,6 +48,18 @@ def _mock_bind(monkeypatch):
     )
     monkeypatch.setattr("gpu_rent.session.provision_vm", lambda cfg, host, log, **kw: None)
     monkeypatch.setattr("gpu_rent.session.wait_backend_idle", lambda cfg, host, log, **kw: None)
+    monkeypatch.setattr(
+        "gpu_rent.session.verify_stack_on_vm",
+        lambda cfg, host, log, **kw: [],
+    )
+    monkeypatch.setattr(
+        "gpu_rent.session.verify_gpu_env",
+        lambda cfg, host, log, **kw: [],
+    )
+    monkeypatch.setattr(
+        "gpu_rent.session.tune_swarm_perf",
+        lambda cfg, host, log: False,
+    )
     monkeypatch.setattr(
         "gpu_rent.session.ensure_boot_snapshot",
         lambda conn, boot_volume_id, cfg, log: None,

tests/test_verify_stack.py

@@ -1,4 +1,5 @@
-from gpu_rent.ready import ServiceCheck, _expected_services, verify_stack_local
+from gpu_rent.errors import CloudError
+from gpu_rent.ready import ServiceCheck, _expected_services, verify_gpu_env, verify_stack_local
 
 
 class _Cfg:
@@ -41,7 +42,9 @@ def test_verify_stack_local_fails_closed_port(monkeypatch):
         ollama_local_port = 17999
         llamacpp_local_port = 17812
 
-    monkeypatch.setattr("gpu_rent.ready._tcp_ok", lambda port, host="127.0.0.1", timeout=0.8: False)
+    monkeypatch.setattr(
+        "gpu_rent.ready._tcp_ok", lambda port, host="127.0.0.1", timeout=0.8: False
+    )
     logs: list[str] = []
     try:
         verify_stack_local(C(), logs.append, timeout=0.3, poll_every=0.1)
@@ -53,3 +56,86 @@ def test_verify_stack_local_fails_closed_port(monkeypatch):
 def test_service_check_dataclass():
     c = ServiceCheck("x", True, "ok", "vm")
     assert c.ok and c.where == "vm"
+
+
+def test_verify_gpu_env_ok(monkeypatch):
+    payload = {
+        "ok": True,
+        "checks": [
+            {"name": "nvidia-smi", "required": True, "ok": True, "detail": "A100"},
+            {"name": "cuda", "required": True, "ok": True, "detail": "libcuda"},
+            {
+                "name": "torch",
+                "required": True,
+                "ok": True,
+                "detail": "torch=2.0 cuda=12 available=True",
+            },
+            {
+                "name": "triton/sage",
+                "required": False,
+                "ok": True,
+                "detail": "есть: triton; нет: —",
+            },
+        ],
+    }
+
+    def fake_run_python(cfg, host, script, **kw):
+        assert "GPU_RENT_CHECK_SWARM" in script
+        return json.dumps(payload) + "\n"
+
+    import json
+
+    monkeypatch.setattr("gpu_rent.ssh_ops.run_python", fake_run_python)
+    # patch where used
+    monkeypatch.setattr(
+        "gpu_rent.ssh_ops.run_python",
+        fake_run_python,
+        raising=False,
+    )
+
+    import gpu_rent.ready as ready_mod
+
+    monkeypatch.setattr(
+        ready_mod,
+        "run_python",
+        fake_run_python,
+        raising=False,
+    )
+
+    # verify_gpu_env imports run_python inside the function
+    import gpu_rent.ssh_ops as ssh_ops
+
+    monkeypatch.setattr(ssh_ops, "run_python", fake_run_python)
+
+    logs: list[str] = []
+    out = verify_gpu_env(_Cfg(), "1.2.3.4", logs.append, timeout=5.0, poll_every=0.1)
+    assert all(c.ok for c in out)
+    assert any(c.name == "torch" for c in out)
+
+
+def test_verify_gpu_env_fails_without_cuda(monkeypatch):
+    import json
+
+    import gpu_rent.ssh_ops as ssh_ops
+
+    payload = {
+        "ok": False,
+        "checks": [
+            {"name": "nvidia-smi", "required": True, "ok": True, "detail": "ok"},
+            {"name": "cuda", "required": True, "ok": False, "detail": "нет libcuda"},
+            {"name": "torch", "required": True, "ok": False, "detail": "no venv"},
+        ],
+    }
+    monkeypatch.setattr(
+        ssh_ops,
+        "run_python",
+        lambda *a, **k: json.dumps(payload),
+    )
+    logs: list[str] = []
+    try:
+        verify_gpu_env(
+            _Cfg(), "1.2.3.4", logs.append, timeout=0.4, poll_every=0.1
+        )
+        assert False, "expected CloudError"
+    except CloudError as exc:
+        assert "cuda" in str(exc).lower() or "GPU-стек" in str(exc)