Add diagnostics command and enhance error handling for backend states
- Introduced a new CLI command `diag` to collect diagnostics from SwarmUI/Comfy, including API, journal, and paths. - Enhanced the `ensure_swarm_comfy_installed` function to include diagnostic script handling for errored backends. - Updated `wait_backend_idle` to trigger diagnostics when backends are in an errored state, improving error recovery. - Implemented fallback mechanisms for diagnostics in the `run_diagnostics` function, ensuring better visibility into backend issues. - Added tests to validate the new diagnostic functionality and error handling, ensuring robustness in backend management.
This commit is contained in:
@@ -161,11 +161,18 @@ def _bind_access(
|
||||
ensure_swarm_comfy_installed(cfg, ip, log)
|
||||
except Exception as exc:
|
||||
log(f"SwarmUI Comfy install: {exc}")
|
||||
try:
|
||||
from gpu_rent.ready import collect_swarm_diagnostics
|
||||
|
||||
collect_swarm_diagnostics(cfg, ip, log)
|
||||
except Exception as diag_exc:
|
||||
log(f"diag: {diag_exc}")
|
||||
raise
|
||||
clock.mark("comfy-install", log)
|
||||
try:
|
||||
wait_backend_idle(cfg, ip, log)
|
||||
except CloudError as exc:
|
||||
# wait_backend_idle already collected diag on errored/timeout
|
||||
log(f"ready: {exc}")
|
||||
raise
|
||||
clock.mark("Idle", log)
|
||||
|
||||
Reference in New Issue
Block a user