- Introduced a new `loading_fail_sec` parameter in the `wait_backend_idle` function to handle prolonged loading states, improving error handling for backend readiness.
- Updated the `ensure_dlbackend_bind` function to stop SwarmUI before remounting, preventing target busy errors and ensuring consistent data mounts.
- Enhanced the `recover_errored_backends` function to account for the new remount logic, improving backend recovery processes.
- Refactored tests to validate the new loading failure conditions and ensure proper handling of backend states during diagnostics.
- Introduced remounting logic for data directories to ensure correct bindings before restarting services, improving reliability during performance tuning.
- Added functions to sanitize and ensure absolute paths for backend scripts, preventing issues with relative paths in bind mounts.
- Enhanced the `patch_backends_extra_args` function to clean up corrupted entries and ensure proper configuration of backend parameters.
- Updated tests to validate the new sanitization and path handling functionalities, ensuring robustness in backend management.
- Added a `--update` option to the `up` command in the CLI for forced updates during warm ACTIVE states.
- Improved the idle-killer logic with new functions to check for reusable credentials and refresh settings on existing remote credentials.
- Enhanced the `seed_civitai` function to implement caching for model jobs, optimizing the provisioning process.
- Updated the installation script for Ollama to prevent unnecessary restarts when the service configuration has not changed.
- Added diagnostics for model job caching and improved error handling in various functions.
- Introduced a new CLI command `diag` to collect diagnostics from SwarmUI/Comfy, including API, journal, and paths.
- Enhanced the `ensure_swarm_comfy_installed` function to include diagnostic script handling for errored backends.
- Updated `wait_backend_idle` to trigger diagnostics when backends are in an errored state, improving error recovery.
- Implemented fallback mechanisms for diagnostics in the `run_diagnostics` function, ensuring better visibility into backend issues.
- Added tests to validate the new diagnostic functionality and error handling, ensuring robustness in backend management.
- Added a new function to ensure headless installation of ComfyUI when backends are empty, enhancing the setup process for SwarmUI.
- Updated documentation to clarify the installation flow and backend readiness checks, ensuring users understand the requirements for a successful setup.
- Enhanced backend status checks to differentiate between 'empty' and 'idle' states, improving error handling and user feedback during provisioning.
- Adjusted logging messages to provide clearer insights into the installation and backend status processes.
- Updated the `_print_checks` function to replace console prints with logging functions for better traceability.
- Introduced timing functionality in the `doctor`, `dry_run`, and `up` functions to log the duration of preflight checks.
- Modified the `wait_ssh` function to accept a logging callback, improving SSH wait feedback.
- Enhanced the `mark` method in `PhaseTimes` to log phase durations, aiding in performance analysis.
- Updated various remote scripts to ensure error messages are printed to stderr for better error handling.
- Updated the `_log_default` function to handle messages starting with `\r` for in-place updates.
- Introduced `split_ssh_stream` to differentiate between line and progress events in SSH output.
- Implemented `feed_ssh_log` to dispatch log messages based on event type.
- Refactored `_stream_pty_output` to utilize the new logging and event handling functions.
- Enhanced the `log` function in `term.py` to support progress updates, ensuring a cleaner output experience.
- Updated `DownloadProgress` classes in `civitai_fetch.py` and `llamacpp_fetch.py` to print progress in place, improving user feedback during downloads.
- Updated CLI documentation to reflect the new handling of `CIVITAI_API_TOKEN`, which is now automatically passed to SwarmUI user settings during startup.
- Improved the `render_access_panel` function to include additional warnings for idle-killer failures and stack errors, enhancing user feedback.
- Introduced a new function `seed_swarmui_api_keys` to manage API key injection into SwarmUI, ensuring seamless integration with the Model Downloader.
- Enhanced GPU environment verification logic to include fail-fast checks for critical components like CUDA, improving error handling and user notifications.
- Updated tests to validate the new API key handling and access panel behavior, ensuring robustness in the integration process.
- Added a new function `verify_gpu_env` to check GPU stack readiness, including nvidia-smi, CUDA, and torch in the Comfy virtual environment when SwarmUI is enabled.
- Updated the session management to call `verify_gpu_env`, capturing GPU environment status and errors in the state notes.
- Enhanced documentation in `cli.md` to reflect the new GPU environment verification process.
- Added tests for `verify_gpu_env` to ensure proper functionality and error handling during GPU checks.
- Added support for balance tracking using `SELECTEL_API_TOKEN` in the configuration.
- Introduced new balance notification logic in the local watchdog, alerting users on balance changes based on defined thresholds.
- Updated documentation to include instructions for setting up balance notifications and the required environment variables.
- Enhanced the `ready` and `session` modules to initialize balance state and handle notifications during GPU operations.
- Refactored the CLI and related components to support the new balance monitoring features, ensuring a seamless user experience.
- Updated `env.example` and `gpu-rent.vars.example` to include new variables for LLM runtime and SwarmUI options.
- Refactored CLI commands to support interactive selection of LLM runtime and workload type (SwarmUI, LLM, or both).
- Improved access link generation to handle cases where SwarmUI is disabled, providing clearer user feedback.
- Enhanced provisioning logic to conditionally bootstrap SwarmUI based on user configuration, allowing for LLM-only setups.
- Updated documentation across multiple files to reflect changes in LLM integration, CLI usage, and configuration management.
- Added support for `llamacpp-models.yaml` in `.gitignore` and implemented logic to copy it in `gpu-rent.ps1` and `gpu-rent.sh`.
- Enhanced CLI to prompt for llama.cpp model presets during setup and execution, improving user experience.
- Updated configuration handling to include `llamacpp_models_manifest` and related functions for managing llama.cpp models.
- Improved documentation in `cli.md` and `llm.md` to reflect changes in llama.cpp integration and model management.
- Refactored provisioning logic to handle llama.cpp model downloads and configurations effectively.
- Introduced GPU probing functionality to gather and store GPU specifications in `/mnt/swarm_data/.gpu-rent-gpu.json`, aiding in performance tuning.
- Updated `install_ollama.sh` and `install_llamacpp.sh` to utilize GPU information for configuring optimal runtime parameters.
- Enhanced `provision.py` to include GPU probing and performance tuning logic, ensuring better resource allocation for LLM operations.
- Improved documentation in `decisions.md`, `llm.md`, and `swarmui.md` to reflect changes in GPU handling and performance tuning processes.
- Added new tests to validate the GPU probing and model resolution logic, ensuring robustness in handling various GPU configurations.
- Updated `resolve_llm_runtime` to prioritize live configuration over legacy notes, ensuring accurate runtime resolution.
- Enhanced `tunnel_forwards` to prefer current configuration for LLM runtime, improving tunnel setup logic.
- Improved idle-killer logic to handle stale markers and provide clearer warnings in the status output.
- Updated CLI documentation in `cli.md` to reflect changes in command behavior and runtime handling.
- Enhanced tests to validate new runtime resolution logic and ensure proper handling of configuration states.
- Integrated `print_access_card` functionality into the `up` command for both tunneled and non-tunneled scenarios.
- Removed the deprecated `print_mcp_snippet` function from the session management flow.
- Updated the `tunnel_forwards` function to streamline port handling for LLM runtimes.
- Enhanced test cases to reflect changes in access card printing and MCP snippet logging.
- Added `ollama-models.yaml` to .gitignore and implemented logic to copy it in gpu-rent.ps1 and gpu-rent.sh.
- Enhanced env.example to include new variables for LLM runtime options and local watchdog configuration.
- Updated CLI commands to support LLM options during setup and execution, including new flags for Ollama and llama.cpp.
- Improved documentation in cli.md and README.md to reflect changes in LLM integration and local watchdog functionality.
- Adjusted architecture and decisions documentation to clarify the role of LLMs and local watchdog in the system.
- Updated .gitignore to include gpu-rent.vars.
- Modified env.example to introduce the UPDATE_GIT variable for controlling git updates during execution.
- Implemented Import-GpuRentVars function in gpu-rent.ps1 to load environment variables from gpu-rent.vars.
- Enhanced gpu-rent.sh to support loading variables from gpu-rent.vars and added logic for handling default and extra arguments.
- Updated CLI documentation to reflect the new gpu-rent.vars file and its usage in configuration.
- Improved bootstrap and provisioning logic to conditionally perform git updates based on the new configuration.
- Introduced `probe_ssh` function for quick SSH checks, improving the handling of server bootstrap scenarios.
- Updated `cmd_up` to utilize `probe_ssh`, enhancing the logic for managing server states based on SSH availability.
- Improved logging for server state transitions and error handling during SSH connection attempts.
- Adjusted timeout settings in `wait_ssh` for better performance and reliability in SSH key acceptance checks.
- Added functions to generate SSH user data in cloud-config format and encode it in Base64 for server creation.
- Updated the `create_gpu_server` function to include public key handling for SSH access.
- Adjusted the default boot volume size from 40GB to 30GB in `os_client.py`.
- Enhanced the `cmd_up` function to manage SSH key injection and server state more effectively.
- Improved logging for SSH connection attempts and error handling in `wait_ssh` to provide clearer feedback on authentication issues.
- Added optional SSH security-group CIDR configuration in `env.example` to allow broader access if needed.
- Improved `guess_operator_cidr` function to support multiple IP retrieval services and allow overriding CIDR via environment variable.
- Refactored security group management to ensure idempotency and proper logging of security group rules.
- Introduced helper function `_get_volume_or_none` for better volume management and error handling.
- Updated `wait_ssh` function to enhance SSH connection handling with improved timeout settings and logging.
- Added state reset logic in `cmd_up` to handle scenarios where the server is not found in the cloud.
- Introduced SCAN_POOLS configuration to specify pools for GPU flavor scanning.
- Updated the `flavors` command to scan specified pools and list available GPU flavors based on `FLAVOR_PREFERENCE`.
- Revised CLI documentation to reflect changes in the `flavors` command behavior.
- Enhanced decision documentation to include details about GPU pool scanning.
- Updated setup instructions to guide users on selecting appropriate GPU pools.
- Updated version number in pyproject.toml and __init__.py to 0.2.0.
- Revised README.md to reflect the current state of the project, including usage instructions and setup steps.
- Improved CLI documentation in cli.md, adding details about new commands and their functionalities.
- Enhanced the quick start section in README.md for better clarity on initial setup.
- Updated local folder documentation to clarify file handling and commands.
- Added a new command for listing GPU flavors and improved error handling in the CLI.
- Implemented a watchdog feature in the tunnel to manage server states effectively.
- Added package data configuration for the 'gpu_rent' package in pyproject.toml.
- Updated README.md to include usage instructions for Windows and Unix launchers.
- Enhanced CLI documentation in cli.md to reflect new commands and their functionalities.
- Revised setup.md to clarify installation steps and environment setup.
- Improved error handling and command descriptions in the CLI implementation.
- Added new functions for model version handling and flavor resolution in the codebase.
- Updated state management to include additional properties for better tracking.
Leonid Pershin