Add diagnostics command and enhance error handling for backend states

- Introduced a new CLI command `diag` to collect diagnostics from SwarmUI/Comfy, including API, journal, and paths.
- Enhanced the `ensure_swarm_comfy_installed` function to include diagnostic script handling for errored backends.
- Updated `wait_backend_idle` to trigger diagnostics when backends are in an errored state, improving error recovery.
- Implemented fallback mechanisms for diagnostics in the `run_diagnostics` function, ensuring better visibility into backend issues.
- Added tests to validate the new diagnostic functionality and error handling, ensuring robustness in backend management.
This commit is contained in:
Leonid Pershin
2026-08-21 10:17:47 +03:00
parent 1785ab369c
commit ec4663c04f
7 changed files with 443 additions and 11 deletions
+7
View File
@@ -161,11 +161,18 @@ def _bind_access(
ensure_swarm_comfy_installed(cfg, ip, log)
except Exception as exc:
log(f"SwarmUI Comfy install: {exc}")
try:
from gpu_rent.ready import collect_swarm_diagnostics
collect_swarm_diagnostics(cfg, ip, log)
except Exception as diag_exc:
log(f"diag: {diag_exc}")
raise
clock.mark("comfy-install", log)
try:
wait_backend_idle(cfg, ip, log)
except CloudError as exc:
# wait_backend_idle already collected diag on errored/timeout
log(f"ready: {exc}")
raise
clock.mark("Idle", log)