Add read-only Debug API support and update documentation

- Introduced a local read-only Debug API accessible at `http://127.0.0.1:17821` for diagnostics and agent interactions.
- Updated CLI commands to include `gpu-rent debug` for launching the Debug API.
- Enhanced documentation to reflect the new Debug API features and usage.
- Modified configuration to include `DEBUG_LOCAL_PORT` for easier customization.
- Added tests to ensure Debug API links are correctly generated in access card outputs.
This commit is contained in:
Leonid Pershin
2026-08-23 06:11:20 +03:00
parent ac0797530d
commit 5847eaab3f
10 changed files with 1558 additions and 11 deletions
+4 -1
View File
@@ -5,11 +5,13 @@
``` ```
┌─ локальная машина (Windows / Linux) ─────────────────────────┐ ┌─ локальная машина (Windows / Linux) ─────────────────────────┐
│ gpu-rent CLI │ │ gpu-rent CLI │
│ up / stop / status / doctor / hold │ up / stop / status / doctor / hold / debug
│ tunnel / open — SSH localhost:17801 → VM :7801 │ │ tunnel / open — SSH localhost:17801 → VM :7801 │
│ Debug API (read-only) — http://127.0.0.1:17821 │
│ state <repo>/.gpu-rent/state.json │ │ state <repo>/.gpu-rent/state.json │
│ │ │ │
│ браузер / MCP / curl API → http://127.0.0.1:17801 │ │ браузер / MCP / curl API → http://127.0.0.1:17801 │
│ агент / curl debug → http://127.0.0.1:17821/snapshot │
│ локальный SwarmUI → http://127.0.0.1:7801 (не трогаем) │ локальный SwarmUI → http://127.0.0.1:7801 (не трогаем)
└───────────────────────────────┬──────────────────────────────┘ └───────────────────────────────┬──────────────────────────────┘
│ SSH :22 и OpenStack API │ SSH :22 и OpenStack API
@@ -48,6 +50,7 @@
| `sync_files` | SFTP `Models` / Wildcards / workflows / Output | | `sync_files` | SFTP `Models` / Wildcards / workflows / Output |
| `notify` | Toast/звук при backend Idle | | `notify` | Toast/звук при backend Idle |
| `tunnel` | sshtunnel + Nova EXPIRED watchdog | | `tunnel` | sshtunnel + Nova EXPIRED watchdog |
| `debug_api` / `debug_checks` | localhost read-only HTTP sidecar (`:17821`) для агента |
| `local_watchdog` | Опциональный локальный тик → stop при unclean exit | | `local_watchdog` | Опциональный локальный тик → stop при unclean exit |
| `llm_runtime` / `setup_wizard` | Opt-in Ollama + `ollama-models.yaml` | | `llm_runtime` / `setup_wizard` | Opt-in Ollama + `ollama-models.yaml` |
| `idle_killer` / `hold` | systemd на VM + hold-файл | | `idle_killer` / `hold` | systemd на VM + hold-файл |
+19
View File
@@ -68,6 +68,8 @@ gpu-rent up --yes --ollama
| `gpu-rent tunnel` / `tunnel --open` | Повторный проброс; `--open` сразу браузер. **Ctrl+C / Ctrl+D** вызывают `stop` | | `gpu-rent tunnel` / `tunnel --open` | Повторный проброс; `--open` сразу браузер. **Ctrl+C / Ctrl+D** вызывают `stop` |
| `gpu-rent open` / `open --llm` | Браузер на SwarmUI / LLM-порт (туннель уже должен слушать) | | `gpu-rent open` / `open --llm` | Браузер на SwarmUI / LLM-порт (туннель уже должен слушать) |
| `gpu-rent status` | State, Nova, диск, killer/hold, LLM, local-watchdog | | `gpu-rent status` | State, Nova, диск, killer/hold, LLM, local-watchdog |
| `gpu-rent diag` | SwarmUI/Comfy diagnostics с VM (API + journal + paths) |
| `gpu-rent debug` | Локальный read-only Debug API (`:17821`) — для агента после `--no-tunnel` / без активного `up` |
| `gpu-rent hold` / `--minutes N` / `--until ISO` / `--clear` | Пауза idle-killer (нужны живая VM + SSH) | | `gpu-rent hold` / `--minutes N` / `--until ISO` / `--clear` | Пауза idle-killer (нужны живая VM + SSH) |
| `gpu-rent stop` / `stop --no-pull` | Удалить compute (+ FIP), диски оставить; optional pull Output | | `gpu-rent stop` / `stop --no-pull` | Удалить compute (+ FIP), диски оставить; optional pull Output |
| `gpu-rent destroy --i-understand-data-loss` | `stop` + диски | | `gpu-rent destroy --i-understand-data-loss` | `stop` + диски |
@@ -120,6 +122,22 @@ Exit 0 → можно `up`. Exit 1 → причина в таблице / кра
--- ---
## Debug API (localhost)
На `gpu-rent up` и `gpu-rent tunnel` CLI поднимает **read-only** HTTP sidecar на `127.0.0.1:17821` (`DEBUG_LOCAL_PORT`). Ссылка печатается сразу и на access card.
Агент: `GET /openapi.json``GET /snapshot` → точечные пути (`/progress`, `/events?since=`, `/checks`, `/logs`, `/diag`, `/gpu`, `/swarm`, `/ollama`, `/assistent`). Пока SSH нет — VM-эндпоинты отвечают `ssh_unavailable`; прогресс установщика всё равно виден в `/progress` и `/events`.
После `up --no-tunnel` процесс завершается и sidecar гаснет — держи отдельно:
```text
gpu-rent debug
```
Мутаций нет (restart/hold/stop — как раньше через CLI).
---
## Local watchdog (опционально) ## Local watchdog (опционально)
Основной авто-stop — **idle-killer на VM** (ноут можно закрыть). Local watchdog — если хочешь гасить GPU при «убили окно / ребут» без `stop`. Основной авто-stop — **idle-killer на VM** (ноут можно закрыть). Local watchdog — если хочешь гасить GPU при «убили окно / ребут» без `stop`.
@@ -194,6 +212,7 @@ AUTOCOMPLETE_ENABLED=true
SWARMUI_LOCAL_PORT=17801 SWARMUI_LOCAL_PORT=17801
LLM_RUNTIME=none LLM_RUNTIME=none
OLLAMA_LOCAL_PORT=17811 OLLAMA_LOCAL_PORT=17811
DEBUG_LOCAL_PORT=17821
UPDATE_GIT=true UPDATE_GIT=true
DEFAULT_FLAVOR_ID= DEFAULT_FLAVOR_ID=
+2
View File
@@ -43,6 +43,8 @@ SWARMUI_LOCAL_PORT=17801
# WORKLOAD=llm # llm-only (no SwarmUI); requires LLM_RUNTIME=ollama # WORKLOAD=llm # llm-only (no SwarmUI); requires LLM_RUNTIME=ollama
LLM_RUNTIME=none LLM_RUNTIME=none
OLLAMA_LOCAL_PORT=17811 OLLAMA_LOCAL_PORT=17811
# Localhost read-only Debug API (up/tunnel/debug) — agent: /openapi.json /snapshot
DEBUG_LOCAL_PORT=17821
# OLLAMA_MODELS_MANIFEST= # OLLAMA_MODELS_MANIFEST=
# Pin Ollama (несecреты; удобнее в gpu-rent.vars): # Pin Ollama (несecреты; удобнее в gpu-rent.vars):
# OLLAMA_VERSION=0.6.5 # OLLAMA_VERSION=0.6.5
+14 -1
View File
@@ -29,10 +29,23 @@ def resolve_llm_runtime(cfg: Config) -> str:
return normalize_runtime(getattr(cfg, "llm_runtime", "none")) return normalize_runtime(getattr(cfg, "llm_runtime", "none"))
def debug_api_base(cfg: Config) -> str:
port = int(getattr(cfg, "debug_local_port", 17821) or 17821)
return f"http://127.0.0.1:{port}"
def collect_access_links(cfg: Config, *, tunneled: bool) -> list[AccessLink]: def collect_access_links(cfg: Config, *, tunneled: bool) -> list[AccessLink]:
"""Build the list of user-facing endpoints (unit-tested).""" """Build the list of user-facing endpoints (unit-tested)."""
links: list[AccessLink] = [] links: list[AccessLink] = []
swarm = bool(getattr(cfg, "enable_swarmui", True)) swarm = bool(getattr(cfg, "enable_swarmui", True))
# Debug sidecar lives on the laptop process — show even without Swarm tunnel.
links.append(
AccessLink(
"Debug API",
f"{debug_api_base(cfg)}/",
"диагностика · /snapshot · /openapi.json",
)
)
if tunneled: if tunneled:
if swarm: if swarm:
port = cfg.swarmui_local_port port = cfg.swarmui_local_port
@@ -71,7 +84,7 @@ def collect_access_links(cfg: Config, *, tunneled: bool) -> list[AccessLink]:
), ),
] ]
) )
if not links: if len(links) == 1:
links.append( links.append(
AccessLink( AccessLink(
"Туннель", "Туннель",
+69 -8
View File
@@ -529,9 +529,12 @@ def up(
"""Create/unshelve GPU; SwarmUI и/или LLM; по умолчанию туннель.""" """Create/unshelve GPU; SwarmUI и/или LLM; по умолчанию туннель."""
cfg = None cfg = None
up_ok = False up_ok = False
debug_srv = None
active_log = log
try: try:
from dataclasses import replace from dataclasses import replace
from gpu_rent.debug_api import start_debug_server, stop_debug_server
from gpu_rent.llm_runtime import ( from gpu_rent.llm_runtime import (
decide_runtime, decide_runtime,
ensure_ollama_manifest_from_example, ensure_ollama_manifest_from_example,
@@ -543,13 +546,23 @@ def up(
from gpu_rent.varsfile import upsert_vars from gpu_rent.varsfile import upsert_vars
clock_reset() clock_reset()
log("запускаю проверку…") # Sidecar before doctor so an agent can watch installer hang from second 0.
early_cfg = load_config(require_auth=False)
debug_srv = start_debug_server(early_cfg, log=log, print_urls=True)
if debug_srv is not None:
active_log = debug_srv.wrap_log(log)
debug_srv.set_step("doctor")
active_log("запускаю проверку…")
checks = run_doctor() checks = run_doctor()
log(f"проверка заняла {format_duration(clock_elapsed())}") active_log(f"проверка заняла {format_duration(clock_elapsed())}")
code = _print_checks(checks, quiet=not verbose) code = _print_checks(checks, quiet=not verbose)
if code != 0: if code != 0:
raise typer.Exit(1) raise typer.Exit(1)
cfg = load_config(require_auth=True) cfg = load_config(require_auth=True)
if debug_srv is not None:
debug_srv.hub.cfg = cfg
debug_srv.set_step("up")
try: try:
runtime = decide_runtime( runtime = decide_runtime(
@@ -618,7 +631,7 @@ def up(
ollama_preset_menu(include_keep=False), ollama_preset_menu(include_keep=False),
default="recommended", default="recommended",
ask=_ask, ask=_ask,
show=log, show=active_log,
) )
except ValueError as exc: except ValueError as exc:
raise GpuRentError(str(exc)) from exc raise GpuRentError(str(exc)) from exc
@@ -640,7 +653,7 @@ def up(
ollama_preset_menu(include_keep=True), ollama_preset_menu(include_keep=True),
default="keep", default="keep",
ask=_ask2, ask=_ask2,
show=log, show=active_log,
) )
if key not in {"keep", "example", ""}: if key not in {"keep", "example", ""}:
write_ollama_models_preset(cfg.ollama_models_manifest, key) write_ollama_models_preset(cfg.ollama_models_manifest, key)
@@ -654,6 +667,8 @@ def up(
) )
cfg = replace(cfg, llm_runtime=runtime, enable_swarmui=enable_swarm) cfg = replace(cfg, llm_runtime=runtime, enable_swarmui=enable_swarm)
if debug_srv is not None:
debug_srv.hub.cfg = cfg
if enable_swarm: if enable_swarm:
ok("стек: SwarmUI" + (f" + {runtime}" if runtime != "none" else "")) ok("стек: SwarmUI" + (f" + {runtime}" if runtime != "none" else ""))
else: else:
@@ -665,6 +680,8 @@ def up(
def ask(msg: str, default: str = "") -> str: def ask(msg: str, default: str = "") -> str:
return typer.prompt(msg, default=default) return typer.prompt(msg, default=default)
if debug_srv is not None:
debug_srv.set_step("provisioning")
state = cmd_up( state = cmd_up(
cfg, cfg,
no_spot=no_spot, no_spot=no_spot,
@@ -674,13 +691,17 @@ def up(
update=(False if no_update else True if force_update else None), update=(False if no_update else True if force_update else None),
confirm=confirm, confirm=confirm,
ask=None if yes else ask, ask=None if yes else ask,
log=log, log=active_log,
) )
up_ok = True up_ok = True
if no_tunnel: if no_tunnel:
from gpu_rent.access_card import print_access_card
print_access_card(cfg, tunneled=False, host=state.floating_ip)
console.print( console.print(
f"[bold]готово[/bold] (без туннеля). " f"[bold]готово[/bold] (без туннеля). "
f"Доступы: gpu-rent tunnel --open | stop: gpu-rent stop" f"Доступы: gpu-rent tunnel --open | stop: gpu-rent stop | "
f"debug: gpu-rent debug"
) )
if state.floating_ip: if state.floating_ip:
console.print(f"FIP {state.floating_ip}") console.print(f"FIP {state.floating_ip}")
@@ -691,11 +712,13 @@ def up(
if not state.floating_ip: if not state.floating_ip:
raise GpuRentError("нет floating IP после up — туннель не открыть") raise GpuRentError("нет floating IP после up — туннель не открыть")
if debug_srv is not None:
debug_srv.set_step("tunnel")
run_tunnel( run_tunnel(
cfg, cfg,
state.floating_ip, state.floating_ip,
open_browser=open_browser, open_browser=open_browser,
log=log, log=active_log,
) )
except KeyboardInterrupt as exc: except KeyboardInterrupt as exc:
if ( if (
@@ -716,6 +739,11 @@ def up(
): ):
_stop_after_failed_up(cfg, exc) _stop_after_failed_up(cfg, exc)
_die(exc) _die(exc)
finally:
if debug_srv is not None:
from gpu_rent.debug_api import stop_debug_server
stop_debug_server(debug_srv)
@app.command() @app.command()
@@ -816,19 +844,52 @@ def tunnel(
open_browser: bool = typer.Option(False, "--open", help="Открыть браузер на 17801"), open_browser: bool = typer.Option(False, "--open", help="Открыть браузер на 17801"),
) -> None: ) -> None:
"""SSH localhost:17801 -> VM :7801. Ctrl+C / Ctrl+D — stop GPU (диски остаются).""" """SSH localhost:17801 -> VM :7801. Ctrl+C / Ctrl+D — stop GPU (диски остаются)."""
debug_srv = None
try: try:
from gpu_rent.debug_api import start_debug_server, stop_debug_server
cfg = load_config(require_auth=True) cfg = load_config(require_auth=True)
state = load_state() state = load_state()
if not state.floating_ip: if not state.floating_ip:
raise GpuRentError("нет floating IP — сначала gpu-rent up") raise GpuRentError("нет floating IP — сначала gpu-rent up")
debug_srv = start_debug_server(cfg, log=log, print_urls=True)
active_log = debug_srv.wrap_log(log) if debug_srv is not None else log
if debug_srv is not None:
debug_srv.set_step("tunnel")
run_tunnel( run_tunnel(
cfg, cfg,
state.floating_ip, state.floating_ip,
open_browser=open_browser, open_browser=open_browser,
log=log, log=active_log,
) )
except GpuRentError as exc: except GpuRentError as exc:
_die(exc) _die(exc)
finally:
if debug_srv is not None:
from gpu_rent.debug_api import stop_debug_server
stop_debug_server(debug_srv)
@app.command("debug")
def debug_cmd(
port: Optional[int] = typer.Option(
None,
"--port",
"-p",
help="Локальный порт (по умолчанию DEBUG_LOCAL_PORT / 17821)",
),
) -> None:
"""Локальный read-only Debug API (для агента), пока VM уже есть / после --no-tunnel."""
try:
from gpu_rent.debug_api import run_debug_blocking
cfg = load_config(require_auth=False)
run_debug_blocking(cfg, port=port, log=log)
except RuntimeError as exc:
_die(GpuRentError(str(exc)))
except GpuRentError as exc:
_die(exc)
@app.command() @app.command()
+2
View File
@@ -108,6 +108,7 @@ class Config:
enable_swarmui: bool enable_swarmui: bool
ollama_models_manifest: Path ollama_models_manifest: Path
ollama_local_port: int ollama_local_port: int
debug_local_port: int
default_flavor_id: str default_flavor_id: str
flavor_preference: tuple[str, ...] flavor_preference: tuple[str, ...]
@@ -240,6 +241,7 @@ def load_config(*, require_auth: bool = True) -> Config:
enable_swarmui=enable_swarmui, enable_swarmui=enable_swarmui,
ollama_models_manifest=ollama_manifest, ollama_models_manifest=ollama_manifest,
ollama_local_port=_as_int(os.environ.get("OLLAMA_LOCAL_PORT"), 17811), ollama_local_port=_as_int(os.environ.get("OLLAMA_LOCAL_PORT"), 17811),
debug_local_port=_as_int(os.environ.get("DEBUG_LOCAL_PORT"), 17821),
default_flavor_id=(os.environ.get("DEFAULT_FLAVOR_ID") or "").strip(), default_flavor_id=(os.environ.get("DEFAULT_FLAVOR_ID") or "").strip(),
flavor_preference=_csv( flavor_preference=_csv(
os.environ.get("FLAVOR_PREFERENCE"), os.environ.get("FLAVOR_PREFERENCE"),
+528
View File
@@ -0,0 +1,528 @@
"""Localhost read-only debug HTTP sidecar for gpu-rent.
Bind only 127.0.0.1. Started from `up` / `tunnel` / `debug`. No mutations.
"""
from __future__ import annotations
import json
import threading
import time
import traceback
from collections import deque
from collections.abc import Callable
from dataclasses import dataclass, field
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Any
from urllib.parse import parse_qs, urlparse
from gpu_rent.config import Config
from gpu_rent import debug_checks
Log = Callable[[str], None]
DEFAULT_PORT = 17821
EVENT_CAPACITY = 500
CACHE_TTL_HEALTH = 5.0
CACHE_TTL_DIAG = 30.0
CACHE_TTL_LOGS = 10.0
CACHE_TTL_DEFAULT = 8.0
_OPENAPI: dict[str, Any] = {
"openapi": "3.0.3",
"info": {
"title": "gpu-rent Debug API",
"version": "1.0.0",
"description": (
"Read-only localhost diagnostics for installer progress and "
"SwarmUI / Comfy / Ollama / Assistent. No mutations."
),
},
"servers": [{"url": "http://127.0.0.1:17821"}],
"paths": {
"/": {"get": {"summary": "HTML index of endpoints"}},
"/openapi.json": {"get": {"summary": "This OpenAPI document"}},
"/health": {"get": {"summary": "ok, phase, failed checks summary"}},
"/progress": {"get": {"summary": "Current up/tunnel step + last log"}},
"/events": {
"get": {
"summary": "Installer log ring buffer",
"parameters": [
{
"name": "since",
"in": "query",
"schema": {"type": "integer"},
"description": "Return events with seq > since",
}
],
}
},
"/snapshot": {"get": {"summary": "Cheap aggregate (no full diag)"}},
"/status": {"get": {"summary": "JSON mirror of gpu-rent status"}},
"/state": {"get": {"summary": "state.json without secrets"}},
"/config": {"get": {"summary": "Config with tokens/passwords redacted"}},
"/checks": {"get": {"summary": "Local + VM service checks"}},
"/logs": {
"get": {
"summary": "journalctl via SSH",
"parameters": [
{
"name": "unit",
"in": "query",
"schema": {"type": "string"},
"description": "swarm|ollama|killer|cloud-init|all",
},
{
"name": "lines",
"in": "query",
"schema": {"type": "integer", "default": 80},
},
],
}
},
"/diag": {"get": {"summary": "Full swarm_diag (expensive, cached ~30s)"}},
"/gpu": {"get": {"summary": "nvidia-smi / CUDA / torch probe"}},
"/swarm": {"get": {"summary": "SwarmUI backend_status + ListBackends"}},
"/ollama": {"get": {"summary": "Ollama tags + ps"}},
"/assistent": {"get": {"summary": "Extension + personas + LLM readiness"}},
},
}
@dataclass
class DebugHub:
"""Shared state for the sidecar process."""
cfg: Config
port: int
started_at: float = field(default_factory=time.time)
step: str = "starting"
step_started_at: float = field(default_factory=time.time)
_events: deque[dict[str, Any]] = field(default_factory=lambda: deque(maxlen=EVENT_CAPACITY))
_seq: int = 0
_lock: threading.Lock = field(default_factory=threading.Lock)
_cache: dict[str, tuple[float, Any]] = field(default_factory=dict)
@property
def base_url(self) -> str:
return f"http://127.0.0.1:{self.port}"
def set_step(self, step: str) -> None:
with self._lock:
if step != self.step:
self.step = step
self.step_started_at = time.time()
def push_event(self, message: str) -> None:
text = message
if text.startswith("\r"):
text = text[1:].rstrip()
if not text:
return
# Progress lines: update step hint, keep one event per distinct text.
text = text.rstrip()
if not text:
return
with self._lock:
self._seq += 1
self._events.append(
{
"seq": self._seq,
"ts": time.time(),
"msg": text[:2000],
}
)
# Infer step from log line (best-effort).
lower = text.lower()
for needle, label in (
("жду ready backend", "wait_backend_idle"),
("backend ready", "backend_ready"),
("проверка gpu", "verify_gpu"),
("проверка на vm", "verify_stack"),
("проверка туннеля", "verify_tunnel"),
("bootstrap", "bootstrap"),
("civitai", "seed_models"),
("ollama", "ollama"),
("extensions", "extensions"),
("туннель", "tunnel"),
("создаю", "provisioning"),
("doctor", "doctor"),
):
if needle in lower:
if self.step != label:
self.step = label
self.step_started_at = time.time()
break
def events_since(self, since: int = 0) -> list[dict[str, Any]]:
with self._lock:
return [e for e in self._events if int(e["seq"]) > since]
def events_tail(self, n: int = 30) -> list[dict[str, Any]]:
with self._lock:
items = list(self._events)
return items[-n:]
def progress(self) -> dict[str, Any]:
with self._lock:
last = self._events[-1] if self._events else None
step = self.step
step_age = int(time.time() - self.step_started_at)
uptime = int(time.time() - self.started_at)
return {
"ok": True,
"step": step,
"step_age_sec": step_age,
"uptime_sec": uptime,
"last_log": last,
}
def cached(self, key: str, ttl: float, factory: Callable[[], Any]) -> Any:
now = time.time()
with self._lock:
hit = self._cache.get(key)
if hit and now - hit[0] < ttl:
return hit[1]
value = factory()
with self._lock:
self._cache[key] = (now, value)
return value
@dataclass
class DebugServer:
hub: DebugHub
httpd: ThreadingHTTPServer
thread: threading.Thread
@property
def base_url(self) -> str:
return self.hub.base_url
@property
def port(self) -> int:
return self.hub.port
def set_step(self, step: str) -> None:
self.hub.set_step(step)
def push_event(self, message: str) -> None:
self.hub.push_event(message)
def wrap_log(self, log: Log) -> Log:
def _tee(msg: str) -> None:
try:
self.hub.push_event(msg)
except Exception:
pass
log(msg)
return _tee
def stop(self) -> None:
try:
self.httpd.shutdown()
except Exception:
pass
try:
self.httpd.server_close()
except Exception:
pass
self.thread.join(timeout=5.0)
_active: DebugServer | None = None
_active_lock = threading.Lock()
def get_active() -> DebugServer | None:
with _active_lock:
return _active
def _make_handler(hub: DebugHub):
class Handler(BaseHTTPRequestHandler):
def log_message(self, fmt: str, *args: Any) -> None:
return # quiet — do not spam CLI
def _send(self, code: int, body: bytes, content_type: str) -> None:
self.send_response(code)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(body)))
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(body)
def _json(self, code: int, payload: Any) -> None:
body = json.dumps(payload, ensure_ascii=False, indent=2, default=str).encode(
"utf-8"
)
self._send(code, body, "application/json; charset=utf-8")
def _html_index(self) -> None:
links = sorted(p for p in _OPENAPI["paths"] if p != "/")
rows = "\n".join(
f'<li><a href="{p}">{p}</a> — '
f'{_OPENAPI["paths"][p]["get"]["summary"]}</li>'
for p in links
)
html = (
"<!DOCTYPE html><html><head><meta charset=utf-8>"
"<title>gpu-rent debug</title></head><body>"
f"<h1>gpu-rent Debug API</h1>"
f"<p>base <code>{hub.base_url}</code> · read-only · 127.0.0.1</p>"
f"<p>Agent: start at <a href='/openapi.json'>/openapi.json</a> "
f"then <a href='/snapshot'>/snapshot</a>.</p>"
f"<ul>{rows}</ul></body></html>"
)
self._send(200, html.encode("utf-8"), "text/html; charset=utf-8")
def do_GET(self) -> None: # noqa: N802
try:
self._dispatch()
except Exception as exc:
self._json(
500,
{
"ok": False,
"error": str(exc)[:300],
"trace": traceback.format_exc()[-800:],
},
)
def _dispatch(self) -> None:
parsed = urlparse(self.path)
path = parsed.path.rstrip("/") or "/"
qs = parse_qs(parsed.query)
if path == "/":
self._html_index()
return
if path == "/openapi.json":
doc = dict(_OPENAPI)
doc["servers"] = [{"url": hub.base_url}]
self._json(200, doc)
return
if path == "/progress":
self._json(200, hub.progress())
return
if path == "/events":
since = 0
if qs.get("since"):
try:
since = int(qs["since"][0])
except ValueError:
self._json(400, {"ok": False, "error": "since must be int"})
return
events = hub.events_since(since)
self._json(
200,
{
"ok": True,
"since": since,
"next": events[-1]["seq"] if events else since,
"events": events,
},
)
return
if path == "/state":
self._json(200, {"ok": True, "state": debug_checks.redact_state()})
return
if path == "/config":
self._json(200, {"ok": True, "config": debug_checks.redact_config(hub.cfg)})
return
if path == "/health":
payload = hub.cached(
"health",
CACHE_TTL_HEALTH,
lambda: debug_checks.collect_health(
hub.cfg,
started_at=hub.started_at,
progress=hub.progress(),
),
)
self._json(200, payload)
return
if path == "/snapshot":
payload = hub.cached(
"snapshot",
CACHE_TTL_HEALTH,
lambda: debug_checks.collect_snapshot(
hub.cfg,
progress=hub.progress(),
events_tail=hub.events_tail(20),
),
)
self._json(200, payload)
return
if path == "/status":
payload = hub.cached(
"status",
CACHE_TTL_DEFAULT,
lambda: debug_checks.collect_status(hub.cfg),
)
self._json(200, payload)
return
if path == "/checks":
payload = hub.cached(
"checks",
CACHE_TTL_DEFAULT,
lambda: debug_checks.collect_checks(hub.cfg),
)
self._json(200, payload)
return
if path == "/logs":
unit = qs.get("unit", [None])[0]
lines = 80
if qs.get("lines"):
try:
lines = max(10, min(int(qs["lines"][0]), 500))
except ValueError:
self._json(400, {"ok": False, "error": "lines must be int"})
return
cache_key = f"logs:{unit}:{lines}"
payload = hub.cached(
cache_key,
CACHE_TTL_LOGS,
lambda: debug_checks.collect_logs(hub.cfg, unit=unit, lines=lines),
)
self._json(200, payload)
return
if path == "/diag":
payload = hub.cached(
"diag",
CACHE_TTL_DIAG,
lambda: debug_checks.collect_diag(hub.cfg),
)
self._json(200, payload)
return
if path == "/gpu":
payload = hub.cached(
"gpu",
CACHE_TTL_DEFAULT,
lambda: debug_checks.collect_gpu(hub.cfg),
)
self._json(200, payload)
return
if path == "/swarm":
payload = hub.cached(
"swarm",
CACHE_TTL_DEFAULT,
lambda: debug_checks.collect_swarm(hub.cfg),
)
self._json(200, payload)
return
if path == "/ollama":
payload = hub.cached(
"ollama",
CACHE_TTL_DEFAULT,
lambda: debug_checks.collect_ollama(hub.cfg),
)
self._json(200, payload)
return
if path == "/assistent":
payload = hub.cached(
"assistent",
CACHE_TTL_DEFAULT,
lambda: debug_checks.collect_assistent(hub.cfg),
)
self._json(200, payload)
return
self._json(404, {"ok": False, "error": f"unknown path {path}"})
return Handler
def start_debug_server(
cfg: Config,
*,
port: int | None = None,
log: Log | None = None,
print_urls: bool = True,
) -> DebugServer | None:
"""Bind 127.0.0.1 and serve in a daemon thread. Returns None on bind failure."""
global _active
wanted = int(port if port is not None else getattr(cfg, "debug_local_port", DEFAULT_PORT))
hub = DebugHub(cfg=cfg, port=wanted)
handler = _make_handler(hub)
try:
httpd = ThreadingHTTPServer(("127.0.0.1", wanted), handler)
except OSError as exc:
msg = (
f"⚠ Debug API не стартовал на 127.0.0.1:{wanted} ({exc}). "
f"up продолжается без sidecar — смени DEBUG_LOCAL_PORT или "
f"`gpu-rent debug` в другом терминале."
)
if log:
log(msg)
else:
print(msg)
return None
httpd.daemon_threads = True
thread = threading.Thread(
target=httpd.serve_forever,
name="gpu-rent-debug-api",
daemon=True,
)
thread.start()
server = DebugServer(hub=hub, httpd=httpd, thread=thread)
with _active_lock:
old = _active
_active = server
if old is not None and old is not server:
try:
old.stop()
except Exception:
pass
if print_urls:
base = server.base_url
lines = [
f"Debug API {base}/",
f" {base}/openapi.json",
f" {base}/snapshot",
]
if log:
for line in lines:
log(line)
else:
for line in lines:
print(line)
return server
def stop_debug_server(server: DebugServer | None = None) -> None:
global _active
target = server
with _active_lock:
if target is None:
target = _active
if _active is target:
_active = None
if target is not None:
target.stop()
def run_debug_blocking(
cfg: Config,
*,
port: int | None = None,
log: Log | None = None,
) -> None:
"""Foreground sidecar for `gpu-rent debug` (Ctrl+C to stop)."""
server = start_debug_server(cfg, port=port, log=log, print_urls=True)
if server is None:
raise RuntimeError("Debug API bind failed")
server.set_step("idle_debug")
try:
if log:
log("Debug API слушает (Ctrl+C — выход). Мутаций нет.")
while True:
time.sleep(3600)
except KeyboardInterrupt:
if log:
log("Debug API остановлен.")
finally:
stop_debug_server(server)
+744
View File
@@ -0,0 +1,744 @@
"""Read-only probes for the local debug HTTP sidecar.
Wraps existing status / diag / logs / verify helpers. Never mutates the VM.
"""
from __future__ import annotations
import json
import socket
import time
import urllib.error
import urllib.request
from dataclasses import asdict
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from gpu_rent.config import Config
from gpu_rent.llm_runtime import normalize_runtime
from gpu_rent.state import SessionState, load_state, preempt_window_end
SSH_UNAVAILABLE = "ssh_unavailable"
_SECRET_KEYS = frozenset(
{
"os_password",
"civitai_api_token",
"hf_token",
"git_token",
"selectel_api_token",
}
)
def _now_iso() -> str:
return datetime.now(timezone.utc).replace(microsecond=0).isoformat()
def ssh_host(state: SessionState | None = None) -> str | None:
st = state if state is not None else load_state()
ip = (st.floating_ip or "").strip()
return ip or None
def ssh_ready(cfg: Config, state: SessionState | None = None) -> bool:
host = ssh_host(state)
if not host:
return False
key = getattr(cfg, "ssh_private_key_path", None)
if key is None or not Path(key).is_file():
return False
return True
def _ssh_fail(phase: str | None = None) -> dict[str, Any]:
out: dict[str, Any] = {"ok": False, "error": SSH_UNAVAILABLE}
if phase:
out["phase"] = phase
return out
def redact_config(cfg: Config) -> dict[str, Any]:
raw = asdict(cfg)
out: dict[str, Any] = {}
for key, value in raw.items():
if key in _SECRET_KEYS:
out[key] = "***" if value else ""
elif isinstance(value, Path):
out[key] = str(value)
else:
out[key] = value
return out
def redact_state(state: SessionState | None = None) -> dict[str, Any]:
st = state if state is not None else load_state()
data = st.to_dict()
notes = dict(data.get("notes") or {})
# Drop anything that looks like a credential if nested later.
for k in list(notes.keys()):
kl = k.lower()
if any(s in kl for s in ("password", "token", "secret", "credential")):
notes[k] = "***"
data["notes"] = notes
return data
def access_card_warnings(state: SessionState | None = None) -> list[str]:
notes = dict((state or load_state()).notes or {})
warn: list[str] = []
if notes.get("idle_killer") == "failed":
warn.append(
"idle-killer НЕ вооружён — GPU может крутиться без авто-stop → gpu-rent stop"
)
if notes.get("stack_vm_error"):
warn.append(f"стек VM: {str(notes['stack_vm_error'])[:140]}")
if notes.get("gpu_env_error"):
warn.append(f"GPU-стек: {str(notes['gpu_env_error'])[:140]}")
if notes.get("llm_error"):
ollama_ok = any(
isinstance(x, dict)
and x.get("name") == "ollama"
and x.get("ok")
and not str(x.get("detail") or "").startswith("WARN")
for x in (notes.get("stack_vm") or [])
)
if not ollama_ok:
warn.append(f"LLM ошибка: {str(notes['llm_error'])[:120]}")
return warn
def _port_open(port: int, host: str = "127.0.0.1", timeout: float = 0.4) -> bool:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(timeout)
try:
return sock.connect_ex((host, port)) == 0
finally:
sock.close()
def collect_status(cfg: Config) -> dict[str, Any]:
"""JSON mirror of the main fields from `gpu-rent status`."""
state = load_state()
notes = dict(state.notes or {})
forwards: list[dict[str, Any]] = []
try:
from gpu_rent.tunnel import tunnel_forwards
for loc, rem in tunnel_forwards(cfg):
forwards.append(
{
"local": loc,
"remote": rem,
"open": _port_open(loc),
}
)
except Exception as exc:
forwards = [{"error": str(exc)[:120]}]
preempt: dict[str, Any] | None = None
end = preempt_window_end(state)
if end:
left = end - datetime.now(timezone.utc)
preempt = {
"until": end.isoformat(),
"hours_left": max(int(left.total_seconds() // 3600), 0),
"minutes_left": max(int((left.total_seconds() % 3600) // 60), 0),
}
out: dict[str, Any] = {
"ok": True,
"phase": state.phase,
"server_id": state.server_id,
"flavor": state.flavor_name or state.flavor_id,
"boot_volume_id": state.boot_volume_id,
"data_volume_id": state.data_volume_id,
"floating_ip": state.floating_ip,
"spot": state.spot,
"bootstrapped": state.bootstrapped,
"preempt": preempt,
"tunnel": forwards,
"data_volume_size_gb": cfg.data_volume_size_gb,
"idle_minutes": cfg.idle_minutes,
"idle_grace_minutes": cfg.idle_grace_minutes,
"warnings": access_card_warnings(state),
"notes_summary": {
"idle_killer": notes.get("idle_killer"),
"llm_error": notes.get("llm_error"),
"stack_vm_error": notes.get("stack_vm_error"),
"gpu_env_error": notes.get("gpu_env_error"),
},
}
if not ssh_ready(cfg, state):
out["disk"] = None
out["idle_killer"] = SSH_UNAVAILABLE
out["assistent_wanted"] = None
return out
host = ssh_host(state)
assert host is not None
try:
from gpu_rent.ssh_ops import run_ssh
df = run_ssh(
cfg,
host,
"df -h /mnt/swarm_data 2>/dev/null | tail -1",
check=False,
timeout=15,
).strip()
out["disk"] = df or None
except Exception as exc:
out["disk"] = f"SSH: {exc}"
try:
from gpu_rent.idle_killer import killer_status_lines
out["idle_killer"] = "; ".join(killer_status_lines(cfg, host))
except Exception as exc:
out["idle_killer"] = str(exc)[:160]
try:
from gpu_rent.provision import count_wanted_models_on_vm
out["assistent_wanted"] = count_wanted_models_on_vm(cfg, host)
except Exception:
out["assistent_wanted"] = None
return out
def collect_checks(cfg: Config) -> dict[str, Any]:
"""Cheap local + optional one-shot VM stack probe."""
state = load_state()
checks: list[dict[str, Any]] = []
# Local tunnel ports
swarm_on = bool(getattr(cfg, "enable_swarmui", True))
runtime = normalize_runtime(getattr(cfg, "llm_runtime", "none"))
if swarm_on:
port = int(cfg.swarmui_local_port)
open_ = _port_open(port)
checks.append(
{
"id": "local_swarmui_port",
"ok": open_,
"detail": f"127.0.0.1:{port} {'open' if open_ else 'closed'}",
}
)
if runtime == "ollama":
port = int(cfg.ollama_local_port)
open_ = _port_open(port)
checks.append(
{
"id": "local_ollama_port",
"ok": open_,
"detail": f"127.0.0.1:{port} {'open' if open_ else 'closed'}",
}
)
# Last verify snapshots from state (no SSH)
notes = dict(state.notes or {})
for key in ("stack_vm", "stack_local", "gpu_env"):
snap = notes.get(key)
if isinstance(snap, list):
for item in snap:
if not isinstance(item, dict):
continue
checks.append(
{
"id": f"note:{key}:{item.get('name')}",
"ok": bool(item.get("ok")),
"detail": str(item.get("detail") or "")[:200],
"source": "state.notes",
}
)
for w in access_card_warnings(state):
checks.append({"id": "warning", "ok": False, "detail": w})
if not ssh_ready(cfg, state):
return {
"ok": all(c.get("ok") for c in checks) if checks else True,
"phase": state.phase,
"ssh": False,
"checks": checks,
"error": SSH_UNAVAILABLE if not ssh_host(state) else None,
}
host = ssh_host(state)
assert host is not None
try:
from gpu_rent.ready import _probe_vm_once
live = _probe_vm_once(cfg, host)
for c in live:
checks.append(
{
"id": f"vm:{c.name}",
"ok": c.ok,
"detail": c.detail,
"where": c.where,
}
)
except Exception as exc:
checks.append({"id": "vm:probe", "ok": False, "detail": str(exc)[:200]})
return {
"ok": all(c.get("ok") for c in checks if c.get("id") != "warning"),
"phase": state.phase,
"ssh": True,
"checks": checks,
}
def collect_logs(
cfg: Config,
*,
unit: str | None = None,
lines: int = 80,
) -> dict[str, Any]:
state = load_state()
if not ssh_ready(cfg, state):
return _ssh_fail(state.phase)
host = ssh_host(state)
assert host is not None
try:
from gpu_rent.vm_logs import fetch_logs_for_cli
text = fetch_logs_for_cli(cfg, host, unit=unit, lines=lines)
return {
"ok": True,
"unit": unit or "all",
"lines": lines,
"text": text or "",
}
except ValueError as exc:
return {"ok": False, "error": str(exc)}
except Exception as exc:
return {"ok": False, "error": str(exc)[:200], "phase": state.phase}
def collect_diag(cfg: Config) -> dict[str, Any]:
state = load_state()
if not ssh_ready(cfg, state):
return _ssh_fail(state.phase)
host = ssh_host(state)
assert host is not None
chunks: list[str] = []
def _log(msg: str) -> None:
if msg.startswith("\r"):
return
chunks.append(msg)
try:
from gpu_rent.ready import collect_swarm_diagnostics
report = collect_swarm_diagnostics(cfg, host, _log)
return {"ok": True, "report": report or "\n".join(chunks), "log": chunks}
except Exception as exc:
return {"ok": False, "error": str(exc)[:200], "log": chunks}
def collect_gpu(cfg: Config) -> dict[str, Any]:
state = load_state()
if not ssh_ready(cfg, state):
return _ssh_fail(state.phase)
host = ssh_host(state)
assert host is not None
try:
from importlib.resources import files
from gpu_rent.ssh_ops import run_python
script = files("gpu_rent.remote").joinpath("stack_env_probe.py").read_text(
encoding="utf-8"
)
swarm_flag = "1" if bool(getattr(cfg, "enable_swarmui", True)) else "0"
out = run_python(
cfg,
host,
script,
remote_path="/tmp/gpu-rent-stack_env_probe.py",
timeout=90,
log=None,
env={"GPU_RENT_CHECK_SWARM": swarm_flag},
)
data: dict[str, Any] = {}
for line in reversed((out or "").splitlines()):
line = line.strip()
if line.startswith("{"):
try:
data = json.loads(line)
break
except json.JSONDecodeError:
continue
if not data:
return {"ok": False, "error": "нет JSON от stack_env_probe", "raw": (out or "")[-400:]}
checks = data.get("checks") if isinstance(data.get("checks"), list) else []
ok = all(bool(c.get("ok")) for c in checks if isinstance(c, dict) and c.get("required", True))
return {"ok": ok, "probe": data}
except Exception as exc:
return {"ok": False, "error": str(exc)[:200]}
def _http_json(url: str, *, method: str = "GET", body: dict | None = None, timeout: float = 8.0) -> tuple[bool, Any]:
data = None
headers = {}
if body is not None:
data = json.dumps(body).encode("utf-8")
headers["Content-Type"] = "application/json"
req = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
raw = resp.read().decode("utf-8", "replace")
try:
return True, json.loads(raw)
except json.JSONDecodeError:
return True, raw[:500]
except Exception as exc:
return False, str(exc)[:200]
def collect_swarm(cfg: Config) -> dict[str, Any]:
state = load_state()
if not bool(getattr(cfg, "enable_swarmui", True)):
return {"ok": True, "enabled": False, "detail": "ENABLE_SWARMUI=false"}
# Prefer local tunnel; fall back to SSH remote poll script.
port = int(cfg.swarmui_local_port)
if _port_open(port):
base = f"http://127.0.0.1:{port}"
ok_s, sess = _http_json(f"{base}/API/GetNewSession", method="POST", body={})
if not ok_s or not isinstance(sess, dict) or not sess.get("session_id"):
return {"ok": False, "via": "local", "error": sess}
sid = str(sess["session_id"])
ok_st, status = _http_json(
f"{base}/API/GetCurrentStatus",
method="POST",
body={"session_id": sid},
)
backends = None
ok_b, be = _http_json(
f"{base}/API/ListBackends",
method="POST",
body={"session_id": sid, "nonreal": False, "full_data": True},
)
if ok_b and isinstance(be, dict):
summary = {}
for key, val in be.items():
if not isinstance(val, dict):
continue
summary[key] = {
"id": val.get("id"),
"type": val.get("type"),
"status": val.get("status"),
"enabled": val.get("enabled"),
"title": val.get("title"),
"current_model": val.get("current_model"),
}
backends = summary
return {
"ok": bool(ok_st),
"via": "local",
"backend_status": (status or {}).get("backend_status") if isinstance(status, dict) else None,
"queue": (status or {}).get("status") if isinstance(status, dict) else None,
"backends": backends,
"error": None if ok_st else status,
}
if not ssh_ready(cfg, state):
return _ssh_fail(state.phase)
host = ssh_host(state)
assert host is not None
try:
from gpu_rent.ssh_ops import run_ssh
script = r"""
import json, urllib.request
def post(path, payload):
req = urllib.request.Request(
"http://127.0.0.1:7801" + path,
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req, timeout=8) as resp:
return json.loads(resp.read().decode())
sess = post("/API/GetNewSession", {})
sid = sess.get("session_id")
st = post("/API/GetCurrentStatus", {"session_id": sid})
be = post("/API/ListBackends", {"session_id": sid, "nonreal": False, "full_data": True})
summary = {}
for k, v in (be or {}).items():
if isinstance(v, dict):
summary[k] = {
"id": v.get("id"), "type": v.get("type"), "status": v.get("status"),
"enabled": v.get("enabled"), "title": v.get("title"),
"current_model": v.get("current_model"),
}
print(json.dumps({
"backend_status": st.get("backend_status"),
"queue": st.get("status"),
"backends": summary,
}, ensure_ascii=False))
"""
out = run_ssh(
cfg,
host,
"python3 - <<'PY'\n" + script + "\nPY",
check=False,
timeout=40,
).strip()
data = json.loads(out.splitlines()[-1])
return {"ok": True, "via": "ssh", **data}
except Exception as exc:
return {"ok": False, "via": "ssh", "error": str(exc)[:200]}
def collect_ollama(cfg: Config) -> dict[str, Any]:
state = load_state()
runtime = normalize_runtime(getattr(cfg, "llm_runtime", "none"))
if runtime != "ollama":
return {"ok": True, "enabled": False, "detail": f"LLM_RUNTIME={runtime}"}
port = int(cfg.ollama_local_port)
if _port_open(port):
base = f"http://127.0.0.1:{port}"
ok_t, tags = _http_json(f"{base}/api/tags")
ok_p, ps = _http_json(f"{base}/api/ps")
models = []
if isinstance(tags, dict) and isinstance(tags.get("models"), list):
for m in tags["models"]:
if isinstance(m, dict) and m.get("name"):
models.append(str(m["name"]))
hint = None
if ok_t and not models:
hint = (
"tags empty — часто NUL/sparse blobs на диске; "
"см. gpu-rent logs --unit ollama"
)
return {
"ok": ok_t,
"via": "local",
"models": models,
"tags": tags if ok_t else None,
"ps": ps if ok_p else None,
"hint": hint,
"error": None if ok_t else tags,
}
if not ssh_ready(cfg, state):
return _ssh_fail(state.phase)
host = ssh_host(state)
assert host is not None
try:
from gpu_rent.ssh_ops import run_ssh
unit = run_ssh(
cfg,
host,
"systemctl is-active gpu-rent-ollama 2>/dev/null || "
"systemctl is-active ollama 2>/dev/null || echo inactive",
check=False,
timeout=15,
).strip()
out = run_ssh(
cfg,
host,
"python3 - <<'PY'\n"
"import json,urllib.request\n"
"def get(u):\n"
" try:\n"
" with urllib.request.urlopen(u, timeout=5) as r:\n"
" return json.loads(r.read().decode())\n"
" except Exception as e:\n"
" return {'_error': str(e)}\n"
"print(json.dumps({'tags': get('http://127.0.0.1:11434/api/tags'),"
"'ps': get('http://127.0.0.1:11434/api/ps')}))\n"
"PY",
check=False,
timeout=25,
).strip()
data = json.loads(out.splitlines()[-1])
tags = data.get("tags") if isinstance(data, dict) else None
models = []
if isinstance(tags, dict) and isinstance(tags.get("models"), list):
for m in tags["models"]:
if isinstance(m, dict) and m.get("name"):
models.append(str(m["name"]))
hint = None
if not models and isinstance(tags, dict) and "_error" not in tags:
hint = "tags empty — часто NUL/sparse blobs"
return {
"ok": "_error" not in (tags or {}),
"via": "ssh",
"unit": unit,
"models": models,
"tags": tags,
"ps": data.get("ps") if isinstance(data, dict) else None,
"hint": hint,
}
except Exception as exc:
return {"ok": False, "via": "ssh", "error": str(exc)[:200]}
def collect_assistent(cfg: Config) -> dict[str, Any]:
state = load_state()
local: dict[str, Any] = {"personas_dir": None, "extensions_yaml": None}
try:
from gpu_rent.paths import assistent_personas_dir
pdir = assistent_personas_dir()
local["personas_dir"] = {
"path": str(pdir),
"exists": pdir.is_dir(),
"entries": sorted(x.name for x in pdir.iterdir())[:40] if pdir.is_dir() else [],
}
except Exception as exc:
local["personas_dir"] = {"error": str(exc)[:120]}
try:
from gpu_rent.manifests import parse_extensions, repo_dirname
repos = parse_extensions(cfg.extensions_manifest)
has = any(
"assistent" in repo_dirname(r).lower() or "assistent" in (r.url or "").lower()
for r in repos
)
local["extensions_yaml"] = {
"has_swarm_assistent": has,
"manifest": str(cfg.extensions_manifest),
}
except Exception as exc:
local["extensions_yaml"] = {"error": str(exc)[:120]}
ollama = collect_ollama(cfg)
out: dict[str, Any] = {
"ok": True,
"local": local,
"ollama_models": ollama.get("models") if ollama.get("ok") else [],
"ollama": {
"ok": ollama.get("ok"),
"enabled": ollama.get("enabled", True),
"hint": ollama.get("hint"),
"error": ollama.get("error"),
},
}
if not ssh_ready(cfg, state):
out["vm"] = _ssh_fail(state.phase)
out["ok"] = False
out["error"] = SSH_UNAVAILABLE
return out
host = ssh_host(state)
assert host is not None
try:
from gpu_rent.provision import count_wanted_models_on_vm
from gpu_rent.ssh_ops import run_ssh
remote = run_ssh(
cfg,
host,
"python3 - <<'PY'\n"
"from pathlib import Path\n"
"import json, os\n"
"DATA = Path('/mnt/swarm_data')\n"
"ext_roots = [\n"
" DATA / 'Data' / 'Extensions',\n"
" Path('/opt/swarmui/src/BuiltinExtensions'),\n"
" Path('/opt/swarmui/src/Extensions'),\n"
"]\n"
"found = []\n"
"for root in ext_roots:\n"
" if not root.is_dir():\n"
" continue\n"
" for p in root.iterdir():\n"
" if 'assistent' in p.name.lower():\n"
" found.append(str(p))\n"
"personas = DATA / 'Assistent'\n"
"entries = sorted(x.name for x in personas.iterdir())[:40] if personas.is_dir() else []\n"
"print(json.dumps({\n"
" 'extension_paths': found,\n"
" 'assistent_dir': str(personas),\n"
" 'assistent_exists': personas.is_dir(),\n"
" 'assistent_entries': entries,\n"
"}, ensure_ascii=False))\n"
"PY",
check=False,
timeout=30,
).strip()
vm = json.loads(remote.splitlines()[-1])
wanted = count_wanted_models_on_vm(cfg, host)
vm["wanted_models"] = wanted
out["vm"] = vm
if not vm.get("extension_paths"):
out["ok"] = False
out["hint"] = "swarm-assistent не найден на VM — seed-extensions / extensions.yaml"
elif not (ollama.get("models") or []):
out["ok"] = False
out["hint"] = "Ollama без моделей — вкладка Assistent будет пустой"
except Exception as exc:
out["vm"] = {"error": str(exc)[:200]}
out["ok"] = False
return out
def collect_snapshot(
cfg: Config,
*,
progress: dict[str, Any] | None = None,
events_tail: list[dict[str, Any]] | None = None,
) -> dict[str, Any]:
"""Cheap aggregate for agents — no full diag."""
state = load_state()
checks = collect_checks(cfg)
status = collect_status(cfg)
return {
"ok": bool(checks.get("ok")) and not access_card_warnings(state),
"ts": _now_iso(),
"phase": state.phase,
"progress": progress,
"warnings": access_card_warnings(state),
"status": {
"server_id": status.get("server_id"),
"floating_ip": status.get("floating_ip"),
"tunnel": status.get("tunnel"),
"idle_killer": status.get("idle_killer"),
"disk": status.get("disk"),
},
"checks": checks.get("checks"),
"events_tail": events_tail or [],
"ssh": ssh_ready(cfg, state),
}
def collect_health(
cfg: Config,
*,
started_at: float,
progress: dict[str, Any] | None = None,
) -> dict[str, Any]:
state = load_state()
checks = collect_checks(cfg)
hard = [
c
for c in (checks.get("checks") or [])
if isinstance(c, dict) and not c.get("ok") and c.get("id") != "warning"
]
return {
"ok": len(hard) == 0,
"phase": state.phase,
"uptime_sec": int(time.time() - started_at),
"progress": progress,
"failed_checks": hard[:20],
"warnings": access_card_warnings(state),
"ssh": ssh_ready(cfg, state),
"ts": _now_iso(),
}
+6 -1
View File
@@ -6,6 +6,7 @@ class _Cfg:
llm_runtime = "ollama" llm_runtime = "ollama"
ollama_local_port = 17811 ollama_local_port = 17811
enable_swarmui = True enable_swarmui = True
debug_local_port = 17821
def test_collect_links_swarm_and_ollama(monkeypatch): def test_collect_links_swarm_and_ollama(monkeypatch):
@@ -15,16 +16,20 @@ def test_collect_links_swarm_and_ollama(monkeypatch):
) )
links = collect_access_links(_Cfg(), tunneled=True) links = collect_access_links(_Cfg(), tunneled=True)
labels = [x.label for x in links] labels = [x.label for x in links]
assert "Debug API" in labels
assert "SwarmUI UI" in labels assert "SwarmUI UI" in labels
assert "Assistent" in labels assert "Assistent" in labels
assert "SwarmUI MCP" in labels assert "SwarmUI MCP" in labels
assert "Ollama API" in labels assert "Ollama API" in labels
assert any(x.label == "Assistent" and "вкладка" in x.note for x in links) assert any(x.label == "Assistent" and "вкладка" in x.note for x in links)
dbg = next(x for x in links if x.label == "Debug API")
assert dbg.url == "http://127.0.0.1:17821/"
def test_collect_links_no_tunnel(): def test_collect_links_no_tunnel():
links = collect_access_links(_Cfg(), tunneled=False) links = collect_access_links(_Cfg(), tunneled=False)
assert links[0].url.startswith("gpu-rent tunnel") assert links[0].label == "Debug API"
assert any(x.url.startswith("gpu-rent tunnel") for x in links)
def test_mcp_snippet_json(): def test_mcp_snippet_json():
+170
View File
@@ -0,0 +1,170 @@
"""Unit tests for localhost Debug API sidecar."""
from __future__ import annotations
import json
import time
import urllib.request
from dataclasses import replace
from pathlib import Path
import pytest
from gpu_rent import debug_api, debug_checks
from gpu_rent.config import load_config
from gpu_rent.state import SessionState
def _auth_env(monkeypatch, tmp_path: Path) -> None:
monkeypatch.setenv("HOME", str(tmp_path))
monkeypatch.setenv("USERPROFILE", str(tmp_path))
monkeypatch.chdir(tmp_path)
for key, val in {
"OS_AUTH_URL": "https://example/v3",
"OS_USER_DOMAIN_NAME": "default",
"OS_USERNAME": "u",
"OS_PASSWORD": "secret-password",
"OS_PROJECT_ID": "proj",
"OS_REGION_NAME": "ru-7",
"GPU_RENT_AZ": "ru-7a",
"CIVITAI_API_TOKEN": "civ-secret",
"HF_TOKEN": "hf-secret",
"GIT_TOKEN": "git-secret",
"SELECTEL_API_TOKEN": "sel-secret",
"DEBUG_LOCAL_PORT": "0", # overridden per-test with free port
}.items():
monkeypatch.setenv(key, val)
def test_redact_config_hides_secrets(monkeypatch, tmp_path):
_auth_env(monkeypatch, tmp_path)
cfg = load_config(require_auth=True)
red = debug_checks.redact_config(cfg)
assert red["os_password"] == "***"
assert red["civitai_api_token"] == "***"
assert red["hf_token"] == "***"
assert red["git_token"] == "***"
assert red["selectel_api_token"] == "***"
assert red["os_username"] == "u"
assert cfg.os_password == "secret-password" # original untouched
def test_redact_state_notes(monkeypatch, tmp_path):
_auth_env(monkeypatch, tmp_path)
st = SessionState(phase="provisioning", floating_ip="1.2.3.4")
st.notes = {"idle_killer": "armed", "api_token": "leak"}
red = debug_checks.redact_state(st)
assert red["phase"] == "provisioning"
assert red["notes"]["api_token"] == "***"
assert red["notes"]["idle_killer"] == "armed"
def test_ssh_unavailable_without_fip(monkeypatch, tmp_path):
_auth_env(monkeypatch, tmp_path)
cfg = load_config(require_auth=True)
monkeypatch.setattr(debug_checks, "load_state", lambda: SessionState(phase="idle"))
out = debug_checks.collect_logs(cfg)
assert out["ok"] is False
assert out["error"] == debug_checks.SSH_UNAVAILABLE
assert out["phase"] == "idle"
def test_debug_server_routes(monkeypatch, tmp_path):
_auth_env(monkeypatch, tmp_path)
cfg = load_config(require_auth=True)
# Free ephemeral port
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("127.0.0.1", 0))
port = sock.getsockname()[1]
sock.close()
cfg = replace(cfg, debug_local_port=port)
logs: list[str] = []
srv = debug_api.start_debug_server(cfg, port=port, log=logs.append, print_urls=True)
assert srv is not None
assert any("openapi.json" in x for x in logs)
try:
tee = srv.wrap_log(lambda m: None)
tee("жду ready backend…")
srv.set_step("wait_backend_idle")
time.sleep(0.15)
def get(path: str) -> tuple[int, dict | str]:
with urllib.request.urlopen(f"http://127.0.0.1:{port}{path}", timeout=5) as resp:
raw = resp.read().decode("utf-8")
code = getattr(resp, "status", 200)
if path == "/":
return code, raw
return code, json.loads(raw)
code, body = get("/")
assert code == 200
assert "snapshot" in body
code, data = get("/openapi.json")
assert code == 200
assert "/health" in data["paths"]
code, data = get("/progress")
assert data["ok"] is True
assert data["step"] in {"wait_backend_idle", "starting"} or "backend" in data["step"]
code, data = get("/events?since=0")
assert data["ok"] is True
assert any("ready backend" in e["msg"] for e in data["events"])
code, data = get("/config")
assert data["config"]["os_password"] == "***"
code, data = get("/state")
assert "state" in data
code, data = get("/snapshot")
assert "phase" in data
assert "checks" in data
code, data = get("/health")
assert "uptime_sec" in data
code, data = get("/logs")
assert data["error"] == debug_checks.SSH_UNAVAILABLE
with pytest.raises(Exception):
urllib.request.urlopen(f"http://127.0.0.1:{port}/nope", timeout=2)
finally:
debug_api.stop_debug_server(srv)
def test_bind_fail_returns_none(monkeypatch, tmp_path):
_auth_env(monkeypatch, tmp_path)
cfg = load_config(require_auth=True)
import socket
blocker = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
blocker.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
blocker.bind(("127.0.0.1", 0))
port = blocker.getsockname()[1]
# Keep bound so second bind fails on Windows/Linux
try:
warns: list[str] = []
# On some platforms SO_REUSEADDR allows double-bind; force fail via invalid host trick
# by patching ThreadingHTTPServer
class Boom:
def __init__(self, *a, **k):
raise OSError("Address already in use")
monkeypatch.setattr(debug_api, "ThreadingHTTPServer", Boom)
srv = debug_api.start_debug_server(cfg, port=port, log=warns.append)
assert srv is None
assert any("не стартовал" in w for w in warns)
finally:
blocker.close()
def test_config_debug_port_default(monkeypatch, tmp_path):
_auth_env(monkeypatch, tmp_path)
monkeypatch.delenv("DEBUG_LOCAL_PORT", raising=False)
cfg = load_config(require_auth=True)
assert cfg.debug_local_port == 17821