- Revised README, architecture, CLI, and other documentation to specify that both `Ctrl+C` and `Ctrl+D` now stop the GPU while preserving disk data, correcting previous inaccuracies.
- Enhanced access card and setup instructions to reflect the updated command behavior for better user understanding.
- Updated tests to ensure the new command behaviors are validated and documented correctly, improving overall clarity in GPU management.
- Enhanced the `OllamaTune` class to include a new `context_length` attribute, improving the configuration for different GPU tiers.
- Updated performance tuning logic to set appropriate context lengths for low, mid, high, and ultra tiers, ensuring optimal resource allocation.
- Modified installation scripts to reflect the new context length settings, enhancing the installation process for Ollama.
- Revised documentation to include context length details in the GPU performance table, providing clearer guidance for users.
- Added tests to validate the correct context length settings in various scenarios, ensuring robustness in performance tuning.
- Updated the `provision_llm` function to utilize the `/api/tags` endpoint for verifying available models, improving accuracy in model management.
- Introduced a new `already_have_ollama_tag` function to ensure exact tag matching, preventing mismatches during model checks.
- Enhanced the `pull_stream` function to require a successful status from the API before proceeding, ensuring reliable model downloads.
- Added logic to handle unwritten blob files, improving the robustness of the model pulling process.
- Updated documentation and tests to reflect these changes, ensuring clarity and reliability in Ollama model operations.
- Clarified the behavior of `Ctrl+C` and `Ctrl+D` in the README and other documentation, specifying that `Ctrl+C` only stops the tunnel while keeping the GPU active, and `Ctrl+D` stops the GPU while preserving disk data.
- Enhanced the CLI documentation to reflect these changes, ensuring users understand the implications of these commands during GPU operations.
- Improved the handling of data bindings and remounting logic in the codebase to prevent issues with empty model tabs in the UI.
- Added tests to validate the new command behaviors and ensure proper documentation alignment.
- Updated the `gpu-rent` logic to ensure the `IsInstalled` flag is accurately set based on the presence of ComfyUI, preventing conflicts during installation.
- Improved the handling of the `/mnt/swarm_data` directory to skip creation if it does not exist, enhancing robustness in provisioning.
- Added tests to validate the new installation state checks and ensure proper diagnostics during the installation process, including scenarios for clearing and setting the `IsInstalled` flag.
- Added a loop to check for the presence of ComfyUI in specified candidate paths, enhancing the logic for determining if ComfyUI is installed on disk.
- This change improves the accuracy of installation state reporting by expanding the search for ComfyUI binaries.
- Introduced `comfy_on_disk` and `sync_is_installed` functions to manage the `IsInstalled` flag based on the presence of ComfyUI, ensuring accurate installation state reporting.
- Updated `install_swarm_comfy.py` to clear the `IsInstalled` flag when necessary, allowing the installation process to proceed without conflicts.
- Improved logic to prevent setting `IsInstalled` to true on first boot or empty data volumes, addressing issues with the InstallConfirmWS process.
- Enhanced diagnostics and logging for installation state changes, providing clearer feedback during the installation process.
- Introduced a new `loading_fail_sec` parameter in the `wait_backend_idle` function to handle prolonged loading states, improving error handling for backend readiness.
- Updated the `ensure_dlbackend_bind` function to stop SwarmUI before remounting, preventing target busy errors and ensuring consistent data mounts.
- Enhanced the `recover_errored_backends` function to account for the new remount logic, improving backend recovery processes.
- Refactored tests to validate the new loading failure conditions and ensure proper handling of backend states during diagnostics.
- Introduced the `clean_extra_args` function to filter out unsafe CLI flags and comment garbage from `ExtraArgs`, improving backend configuration integrity.
- Updated the `repair_extra_args_on_disk` function to rewrite corrupted `ExtraArgs` lines in the FDS file, ensuring cleaner backend settings.
- Enhanced the `recover_errored_backends` function to utilize the new sanitization logic, preserving valid `ExtraArgs` during backend reconfiguration.
- Improved the `sanitize_backends_fds` function to drop comment garbage and retain valid entries, enhancing overall backend management.
- Added tests to validate the new sanitization behavior, ensuring robustness in handling `ExtraArgs` during backend operations.
- Updated the `recover_errored_backends` function to preserve `ExtraArgs` during backend reconfiguration, improving error recovery.
- Refined the `sanitize_backends_fds` function to prevent newline merging issues with bare `\\x` entries, ensuring correct FDS formatting.
- Enhanced the `configure_comfy_backend` function to accept and manage `ExtraArgs`, preventing loss of important configuration during recovery.
- Added tests to validate the new sanitization behavior and ensure that bare `\\x` entries remain intact, improving overall backend management.
- Introduced remounting logic for data directories to ensure correct bindings before restarting services, improving reliability during performance tuning.
- Added functions to sanitize and ensure absolute paths for backend scripts, preventing issues with relative paths in bind mounts.
- Enhanced the `patch_backends_extra_args` function to clean up corrupted entries and ensure proper configuration of backend parameters.
- Updated tests to validate the new sanitization and path handling functionalities, ensuring robustness in backend management.
- Introduced functions to ensure the `IsInstalled` flag is set in `Settings.fds`, preventing the UI from displaying the /Install wizard.
- Added logic to create or patch `Settings.fds` with installation details, including the installation date and version.
- Updated the `install_swarm_comfy.py` script to call the new function, ensuring the installation state is correctly managed during backend operations.
- Added tests to verify the presence of the `IsInstalled` flag in the relevant scripts and ensure proper functionality during installation checks.
- Added a `--update` option to the `up` command in the CLI for forced updates during warm ACTIVE states.
- Improved the idle-killer logic with new functions to check for reusable credentials and refresh settings on existing remote credentials.
- Enhanced the `seed_civitai` function to implement caching for model jobs, optimizing the provisioning process.
- Updated the installation script for Ollama to prevent unnecessary restarts when the service configuration has not changed.
- Added diagnostics for model job caching and improved error handling in various functions.
- Introduced a new `delete_backend` function to facilitate backend removal during recovery processes.
- Improved the `recover_errored_backends` function to handle backend reconfiguration and recreation more effectively, including enhanced logging for diagnostics.
- Added a new `add_comfy_selfstart` function to streamline the addition of new backends.
- Updated the `comfy_start_script` function to prefer absolute paths for better reliability.
- Enhanced diagnostics in `swarm_diag.py` to include detailed backend settings and mount status.
- Added tests to validate the new backend management features and ensure robust error handling.
- Updated the `wait_backend_idle` function to handle disabled backends, introducing a new failure condition and diagnostics collection for better error handling.
- Refactored the `install_swarm_comfy` script to include a new `configure_comfy_backend` function, streamlining backend configuration and enabling better management of backend states.
- Added tests to validate the new functionality for handling disabled backends and ensuring robust diagnostics, improving overall backend management.
- Improved the `install_swarm_comfy` function to handle empty backend states more effectively, introducing recovery mechanisms and enhanced logging for better visibility.
- Updated the `tune_swarm_perf` function to always sanitize backend FDS corruption, ensuring consistent performance tuning.
- Added new tests to validate the functionality of backend recovery and FDS sanitization, ensuring robustness in backend management.
- Introduced the `file_size_bytes` function to convert Civitai model sizes from kilobytes to bytes, improving data handling.
- Updated the `seed_civitai` function to include file size in job definitions, enhancing model processing efficiency.
- Enhanced the `should_skip` function to utilize expected size for faster decision-making during job processing.
- Added tests for new functionality, ensuring robustness in file size handling and job processing logic.
- Enhanced the `with_token` function to support Gitea and Forgejo URLs by constructing the appropriate authentication format for generic HTTPS.
- Updated tests to validate the new functionality, ensuring correct token injection for Gitea URLs while maintaining existing behavior for GitHub and GitLab.
- Improved overall robustness of token handling across different platforms.
- Introduced the `pick_preview_image` function to extract the first usable preview image URL and its suffix from model version data.
- Updated the `seed_civitai` function to include preview image URLs and destinations in job definitions, improving model handling.
- Implemented the `ensure_preview` function to download missing preview images during job processing, enhancing user experience.
- Added tests for `pick_preview_image` to ensure correct functionality across various scenarios, ensuring robustness in image handling.
- Introduced functions `swarm_git_ok`, `clone_swarmui`, and `update_swarmui` to streamline the cloning and updating process of SwarmUI.
- Improved error handling during the update process, ensuring a fallback to cloning if the update fails.
- Enhanced logging messages for better visibility during cloning and updating operations.
- Updated tests to verify the presence of new functions and logging messages in the bootstrap script.
- Added `wget` to the package installation list for improved script functionality.
- Implemented a fallback mechanism using `curl` for .NET installation if the primary script fails, enhancing robustness.
- Updated ownership change for the installation directory to ensure proper permissions for the user.
- Enhanced logging for error scenarios during .NET installation, providing clearer feedback for troubleshooting.
- Added tests to verify the new fallback functionality and ownership changes in the bootstrap script.
- Introduced a new CLI command `diag` to collect diagnostics from SwarmUI/Comfy, including API, journal, and paths.
- Enhanced the `ensure_swarm_comfy_installed` function to include diagnostic script handling for errored backends.
- Updated `wait_backend_idle` to trigger diagnostics when backends are in an errored state, improving error recovery.
- Implemented fallback mechanisms for diagnostics in the `run_diagnostics` function, ensuring better visibility into backend issues.
- Added tests to validate the new diagnostic functionality and error handling, ensuring robustness in backend management.
- Updated the idle-killer logic to treat SwarmUI `empty` and `disabled` states as busy, preventing unnecessary idle time during provisioning.
- Enhanced the `wait_backend_idle` function to recognize suspended backends as ready, improving resource utilization and user feedback.
- Refined the `install_swarm_comfy` script to skip installation when backends are already present, streamlining the setup process.
- Improved the `resolve_llm_runtime` function to prioritize live configuration over stale state notes, ensuring accurate runtime detection.
- Added tests to validate the new backend status handling and idle management logic, ensuring robustness and reliability.
- Replaced direct calls to `venv/bin/pip` with `python -m pip` to avoid execution errors on network mounts and improve compatibility.
- Introduced a new function `find_comfy_python` to locate the appropriate Python executable for ComfyUI installations.
- Updated the `pip_install_sage` function to utilize the new Python handling, enhancing installation reliability.
- Added tests to verify the new behavior of Python environment detection and installation processes, ensuring robustness.
- Enhanced documentation to clarify the transition from 'Idle' to 'ready (running)' for backend states, improving user understanding of system readiness.
- Updated logging messages in the notification system to reflect the new backend status terminology, ensuring accurate feedback during operations.
- Refined access link collection logic to better handle tunneled and non-tunneled scenarios, enhancing user experience.
- Improved tests to validate the new backend status handling and ensure accurate reporting of access links and notifications.
- Updated the `collect_access_links` function to provide clearer user-facing endpoint labels and notes, particularly for non-tunneled scenarios.
- Improved logging messages in the provisioning process to reflect the status of the SwarmUI and Ollama API, enhancing user feedback during setup.
- Added human-readable status messages for backend loading and running states, improving clarity during the waiting process.
- Updated tests to verify the new behavior and ensure accurate reporting of access links and backend statuses.
- Added functions for formatting byte sizes, displaying progress bars, and calculating directory sizes to improve user feedback during installation.
- Implemented a stage detection mechanism to provide real-time updates on the installation process of ComfyUI.
- Enhanced logging to capture recent journal lines related to ComfyUI installation, improving visibility into the installation status.
- Updated tests to verify the presence of new functionality in the installation script, ensuring robustness and reliability.
- Added a new function to ensure headless installation of ComfyUI when backends are empty, enhancing the setup process for SwarmUI.
- Updated documentation to clarify the installation flow and backend readiness checks, ensuring users understand the requirements for a successful setup.
- Enhanced backend status checks to differentiate between 'empty' and 'idle' states, improving error handling and user feedback during provisioning.
- Adjusted logging messages to provide clearer insights into the installation and backend status processes.
- Modified the `load_config` function to allow launch preferences in `gpu-rent.vars` to override `.env` settings for non-secret variables, enhancing user control.
- Updated the list of Python candidates in `stack_env_probe.py` to include additional paths for ComfyUI, improving the detection of Python environments.
- Added new pip candidates in `tune_swarm_perf.py` to support various ComfyUI installations, ensuring better compatibility with different setups.
- Removed the direct injection of environment variables into the script, preventing issues with `from __future__` imports.
- Updated the `run_python` function to accept an `env` parameter for passing runtime environment variables, enhancing script execution control.
- Adjusted tests to verify the new behavior, ensuring that environment variables are correctly set without modifying the script content.
- Revised model descriptions in `ollama-models.example.yaml` to emphasize uncensored and abliterated requirements, enhancing user understanding.
- Updated documentation in `llm.md` to reflect changes in model tags and their meanings, ensuring accurate guidance for users.
- Modified CLI help messages to clarify the nature of presets, reinforcing that all options are abliterate models with Russian support.
- Enhanced the `llm_runtime.py` file to align preset labels with the updated model descriptions, improving consistency across the codebase.
- Removed references to llamacpp from configuration files, scripts, and documentation, streamlining the LLM setup process to focus solely on Ollama.
- Updated environment variables and paths to eliminate llamacpp-related entries, ensuring clarity in the configuration.
- Adjusted CLI commands and help messages to reflect the removal of llamacpp, enhancing user experience and reducing confusion.
- Revised documentation to provide clear guidance on using Ollama exclusively, including updates to setup instructions and runtime options.
- Reduced default values for IDLE_MINUTES from 60 to 30 and IDLE_GRACE_MINUTES from 90 to 45 across configuration files and documentation.
- Updated related documentation to reflect the new default idle time settings, ensuring consistency in user guidance and system behavior.
- Added a new configuration option `UP_STOP_ON_FAIL` to control whether the GPU should be stopped automatically if the `up` command fails, enhancing user control over resource management.
- Updated the CLI to include a `--keep-on-fail` flag, allowing users to prevent GPU shutdown during installation errors.
- Enhanced the installation scripts and documentation to reflect these changes, providing clearer guidance on the new behavior and configuration options.
- Improved error handling in the CLI to ensure proper cleanup of resources in case of failure, preventing unexpected billing for unused GPU resources.
- Introduced a new variable to determine the type of archive being processed, enhancing the logic for handling different file types.
- Updated the installation process to ensure proper extraction of both zip and tar files, improving compatibility with various archive formats.
- Enhanced logging to provide clearer feedback on the type of archive being processed, aiding in troubleshooting.
- Added error handling to ensure the installation process returns a failure status if the archive does not contain the expected binaries.
- Added new environment variables in `env.example` and `gpu-rent.vars.example` for fine-tuning LLM settings, including CUDA build options and model version pinning.
- Updated `llm.md` documentation to include detailed descriptions of new configuration options and usage cases for LLM setups.
- Enhanced the `provision.py` script to forward new environment variables during remote installations, improving the installation process for LLM components.
- Modified the `install_llamacpp.sh` script to support conditional CUDA builds and asset URL overrides, ensuring better compatibility with various environments.
- Improved logging in the installation scripts to provide clearer feedback during the setup process.
- Updated the `_print_checks` function to replace console prints with logging functions for better traceability.
- Introduced timing functionality in the `doctor`, `dry_run`, and `up` functions to log the duration of preflight checks.
- Modified the `wait_ssh` function to accept a logging callback, improving SSH wait feedback.
- Enhanced the `mark` method in `PhaseTimes` to log phase durations, aiding in performance analysis.
- Updated various remote scripts to ensure error messages are printed to stderr for better error handling.
- Added logging statements to the `doctor`, `dry_run`, and `up` functions to indicate the start of preflight checks, improving user feedback during execution.
- Ensured consistent logging messages across different functions to enhance traceability and debugging capabilities.
- Added a new function `pick_llamacpp_linux_asset_url` to select appropriate Linux release assets, prioritizing Ubuntu CUDA and Vulkan options while excluding Windows and macOS binaries.
- Updated the installation script to build `llama-server` from source when Linux CUDA binaries are unavailable, improving compatibility and flexibility.
- Revised documentation to reflect changes in asset handling and installation procedures.
- Added tests to validate the new asset selection logic, ensuring correct behavior in various scenarios.
- Revised model URLs and descriptions in `llamacpp-models.example.yaml` and `ollama-models.example.yaml` to reflect new recommendations and vision capabilities.
- Updated the LLM runtime logic to support vision projectors and improved model resolution handling.
- Enhanced the installation script to conditionally include vision projectors when available.
- Added tests to validate the inclusion of vision projectors in model presets and ensure proper URL remapping for deprecated models.
- Improved documentation to clarify model usage and configuration options.
- Introduced support for Hugging Face API integration, allowing fallback model resolution when Civitai fails.
- Updated configuration to include `HF_TOKEN` and `HF_TOKEN_PATH` for authentication.
- Enhanced model capture logic to differentiate between Civitai and Hugging Face sources.
- Improved error handling for model downloads, providing clearer messages for authentication issues.
- Updated documentation to reflect new environment variables and usage instructions for Hugging Face integration.
- Added tests to validate the new fallback mechanism and ensure robust model resolution.
- Increased default values for IDLE_MINUTES from 30 to 60 and IDLE_GRACE_MINUTES from 45 to 90 across configuration files and documentation.
- Updated related documentation to reflect the new default idle time settings, ensuring consistency in user guidance and system behavior.
- Changed the check for the existence of the .NET installation script to verify if the file exists instead of checking for execution permissions.
- Updated the command to run the .NET installation script from the repository root to ensure proper execution context.
- Added error handling to log a warning if the installation script fails and an error message if .NET is still not installed after the attempt, preventing SwarmUI from starting.
- Enhanced logging to confirm successful .NET installation, improving user feedback during the bootstrap process.
- Updated the `_log_default` function to handle messages starting with `\r` for in-place updates.
- Introduced `split_ssh_stream` to differentiate between line and progress events in SSH output.
- Implemented `feed_ssh_log` to dispatch log messages based on event type.
- Refactored `_stream_pty_output` to utilize the new logging and event handling functions.
- Enhanced the `log` function in `term.py` to support progress updates, ensuring a cleaner output experience.
- Updated `DownloadProgress` classes in `civitai_fetch.py` and `llamacpp_fetch.py` to print progress in place, improving user feedback during downloads.
- Introduced `fmt_bytes` function for human-readable byte size formatting.
- Added `DownloadProgress` class to track and display download progress with speed and completion percentage.
- Updated `download` functions in `civitai_fetch.py` and `llamacpp_fetch.py` to utilize progress tracking.
- Enhanced `ollama_pull.py` to support streaming progress updates during model pulls.
- Updated tests to validate new formatting and progress tracking functionalities, ensuring accurate output and user feedback.
- Updated CLI documentation to reflect the new handling of `CIVITAI_API_TOKEN`, which is now automatically passed to SwarmUI user settings during startup.
- Improved the `render_access_panel` function to include additional warnings for idle-killer failures and stack errors, enhancing user feedback.
- Introduced a new function `seed_swarmui_api_keys` to manage API key injection into SwarmUI, ensuring seamless integration with the Model Downloader.
- Enhanced GPU environment verification logic to include fail-fast checks for critical components like CUDA, improving error handling and user notifications.
- Updated tests to validate the new API key handling and access panel behavior, ensuring robustness in the integration process.
- Marked critical bugs as resolved in the review documentation, including changes to the `arm_idle_killer` function to raise errors on credential creation failures and ensure proper file permissions for JSON credentials.
- Introduced a new `_try_arm_idle_killer` function in `provision.py` to manage idle-killer state more effectively, ensuring it arms correctly during provisioning.
- Updated the `swarm_busy` function in `remote/idle_killer.py` to allow idle state after a specified duration of Swarm unavailability, preventing unnecessary billing.
- Enhanced performance tuning logic in `tune_swarm_perf.py` to ensure proper handling of pip installation success before applying extra arguments.
- Added tests to validate the new idle-killer behavior and swarm management logic, ensuring robustness in handling idle states and error conditions.
- Rearranged the order of verification steps in the CLI documentation for clarity.
- Enhanced the `test_verify_gpu_env` tests to improve logging and error assertions, ensuring accurate feedback on GPU stack checks.
- Added a new test to skip the torch requirement when only LLM is enabled, reflecting updated behavior in GPU environment verification.
- Added a new function `verify_gpu_env` to check GPU stack readiness, including nvidia-smi, CUDA, and torch in the Comfy virtual environment when SwarmUI is enabled.
- Updated the session management to call `verify_gpu_env`, capturing GPU environment status and errors in the state notes.
- Enhanced documentation in `cli.md` to reflect the new GPU environment verification process.
- Added tests for `verify_gpu_env` to ensure proper functionality and error handling during GPU checks.
- Added support for balance tracking using `SELECTEL_API_TOKEN` in the configuration.
- Introduced new balance notification logic in the local watchdog, alerting users on balance changes based on defined thresholds.
- Updated documentation to include instructions for setting up balance notifications and the required environment variables.
- Enhanced the `ready` and `session` modules to initialize balance state and handle notifications during GPU operations.
- Refactored the CLI and related components to support the new balance monitoring features, ensuring a seamless user experience.
Leonid Pershin