Implement UP_STOP_ON_FAIL option to manage GPU state on installation failure

- Added a new configuration option `UP_STOP_ON_FAIL` to control whether the GPU should be stopped automatically if the `up` command fails, enhancing user control over resource management.
- Updated the CLI to include a `--keep-on-fail` flag, allowing users to prevent GPU shutdown during installation errors.
- Enhanced the installation scripts and documentation to reflect these changes, providing clearer guidance on the new behavior and configuration options.
- Improved error handling in the CLI to ensure proper cleanup of resources in case of failure, preventing unexpected billing for unused GPU resources.
This commit is contained in:
Leonid Pershin
2026-08-21 08:32:33 +03:00
parent 5081a6bc1e
commit 6871c511c4
9 changed files with 179 additions and 14 deletions
+11 -6
View File
@@ -22,6 +22,7 @@
# IDLE_MINUTES=60
# IDLE_GRACE_MINUTES=90
# NOTIFY_READY=true
# UP_STOP_ON_FAIL=true # при падении up гасить GPU (default); false / --keep-on-fail
# Prefer auto /32 (leave unset). Spike/WARP only:
# GPU_RENT_SSH_CIDR=0.0.0.0/0
@@ -30,23 +31,27 @@
# Подробнее: docs/llm.md
# ---------------------------------------------------------------------------
# --- A) Быстрый default: SwarmUI + llama.cpp, Ubuntu Vulkan prebuilt ---
# --- A) Default: SwarmUI + llama.cpp (auto: CUDA если nvcc на VM, иначе Vulkan) ---
# LLM_RUNTIME=llamacpp
# ENABLE_SWARMUI=true
# LLAMACPP_TAG=b10545
# (GGUF — llamacpp-models.yaml; бинарь без compile)
# LLAMACPP_BACKEND=auto
# --- B) Только LLM, без SwarmUI (дешевле по времени bootstrap / VRAM) ---
# --- B) Только LLM, без SwarmUI ---
# WORKLOAD=llm
# LLM_RUNTIME=llamacpp
# LLAMACPP_TAG=b10545
# --- C) Максимальная скорость инференса: CUDA-сборка (5–15 мин первый раз) ---
# --- C) Явно CUDA (и снести старый Vulkan-бинарь) ---
# LLM_RUNTIME=llamacpp
# LLAMACPP_TAG=b10545
# LLAMACPP_BUILD_CUDA=1
# LLAMACPP_BACKEND=cuda
# LLAMACPP_FORCE_REINSTALL=1
# (FORCE — снести старый Vulkan/битый бинарь и переустановить)
# --- C2) Явно быстрый Vulkan, без compile ---
# LLM_RUNTIME=llamacpp
# LLAMACPP_BACKEND=vulkan
# LLAMACPP_BUILD_CUDA=0
# --- D) Свой бинарь / pin URL (supply-chain) ---
# LLM_RUNTIME=llamacpp