Add package data for GPU rent and update CLI documentation

- Added package data configuration for the 'gpu_rent' package in pyproject.toml.
- Updated README.md to include usage instructions for Windows and Unix launchers.
- Enhanced CLI documentation in cli.md to reflect new commands and their functionalities.
- Revised setup.md to clarify installation steps and environment setup.
- Improved error handling and command descriptions in the CLI implementation.
- Added new functions for model version handling and flavor resolution in the codebase.
- Updated state management to include additional properties for better tracking.
This commit is contained in:
Leonid Pershin
2026-08-21 03:06:51 +03:00
parent 167d07a733
commit 615cf81493
34 changed files with 2733 additions and 117 deletions
+5
View File
@@ -0,0 +1,5 @@
# Keep launcher line endings correct on Windows and Unix clones.
gpu-rent.sh text eol=lf
gpu-rent.bat text eol=crlf
gpu-rent.ps1 text eol=crlf
src/gpu_rent/remote/*.sh text eol=lf
+21 -10
View File
@@ -8,19 +8,30 @@
## Что уже можно запустить ## Что уже можно запустить
Windows (cmd или двойной клик): `gpu-rent.bat`
PowerShell: `.\gpu-rent.ps1`
Linux / macOS / Git Bash: `./gpu-rent.sh`
Скрипты сами создают `.venv`, ставят пакет и прокидывают аргументы. Активировать окружение не нужно.
```powershell ```powershell
python -m venv .venv .\gpu-rent.ps1 doctor
.\.venv\Scripts\activate .\gpu-rent.ps1 dry-run
python -m pip install -U pip .\gpu-rent.ps1 status
python -m pip install -e ".[dev]" # если doctor зелёный и квота GPU > 0:
copy env.example $env:USERPROFILE\.gpu-rent\.env .\gpu-rent.ps1 up --yes
# заполни OS_* по docs/setup.md .\gpu-rent.ps1 tunnel
gpu-rent doctor .\gpu-rent.ps1 stop
gpu-rent dry-run
gpu-rent status
``` ```
`up` / `stop` / `tunnel` пока не создают сервер: сначала `doctor` должен быть зелёным и квота GPU > 0. ```bash
chmod +x gpu-rent.sh # один раз на Unix
./gpu-rent.sh doctor
```
Ключи: скопируй `env.example` в `%USERPROFILE%\.gpu-rent\.env` (скрипт сделает это сам при первом запуске) и заполни `OS_*` по [`docs/setup.md`](docs/setup.md).
`up` поднимает compute и ставит SwarmUI нативно (`launch-linux.sh` + systemd). `tunnel` пробрасывает `localhost:17801`. `stop` гасит GPU, диски оставляет.
## Зачем какие ключи ## Зачем какие ключи
+3 -1
View File
@@ -35,6 +35,8 @@
Нет команды `generate`. Нет зеркала каталога локального SwarmUI — только папки приложения, см. [local-folders.md](local-folders.md). Нет команды `generate`. Нет зеркала каталога локального SwarmUI — только папки приложения, см. [local-folders.md](local-folders.md).
Сейчас в коде: `up` создаёт/unshelve compute, FIP, SSH, идемпотентный bootstrap SwarmUI (без Docker) и systemd. `tunnel` на 17801, Ctrl+C не гасит GPU. Ещё нет: extensions, autocomplete, Civitai seed, push, idle-killer, snapshot, EXPIRED-reconnect туннеля.
## `doctor` ## `doctor`
Не создаёт сервер, не тратит GPU. Нужен до первого `up` и когда «вчера работало». Не создаёт сервер, не тратит GPU. Нужен до первого `up` и когда «вчера работало».
@@ -166,6 +168,6 @@ Application credential для idle-killer CLI создаёт при bootstrap и
## Windows ## Windows
Пути через `Path.expanduser`. Венв: `.venv\Scripts\activate`. Ключ генерирует CLI: `~/.gpu-rent/id_ed25519`. Лаунчеры в корне: `gpu-rent.bat` (cmd / двойной клик), `.\gpu-rent.ps1` (PowerShell). На Unix — `./gpu-rent.sh`. Они создают `.venv` в репозитории и вызывают `python -m gpu_rent`. Пути в приложении через `Path.expanduser`. Ключ генерирует CLI: `~/.gpu-rent/id_ed25519`.
`NOTIFY_READY`: toast через WinRT / `win10toast` (что заработает на 10/11 без админ-прав) + системный звук. Если toast недоступен — только звук и лог, не падать. `NOTIFY_READY`: toast через WinRT / `win10toast` (что заработает на 10/11 без админ-прав) + системный звук. Если toast недоступен — только звук и лог, не падать.
+14 -14
View File
@@ -34,36 +34,36 @@
## 1. Каркас ## 1. Каркас
- [x] `pyproject.toml`, пакет `gpu_rent`, MIT - [x] `pyproject.toml`, пакет `gpu_rent`, MIT
- [x] config / state / Typer по [cli.md](cli.md): `doctor` / `dry-run` / `status` / `open` живые; остальные команды есть и честно говорят, что mutating ещё нет - [x] config / state / Typer: `doctor` / `dry-run` / `status` / `open` / `up` / `stop` / `destroy` / `ssh` / `logs`; `tunnel` / `hold` / seed ещё `NotReadyError`
- [x] Windows: `python -m gpu_rent` - [x] Windows: `gpu-rent.bat` / `gpu-rent.ps1`; Unix: `gpu-rent.sh`; также `python -m gpu_rent`
- [x] [setup.md](setup.md) — квота, сервисный пользователь, RC, Civitai - [x] [setup.md](setup.md) — квота, сервисный пользователь, RC, Civitai
## 2. OpenStack-клиент ## 2. OpenStack-клиент
- [x] Token через openstacksdk (`authorize`), inventory flavors/квота/volume type - [x] Token через openstacksdk (`authorize`), inventory flavors/квота/volume type
- [ ] Сеть find-or-create, SG, FIP, volumes, server, unshelve - [x] Сеть find-or-create, SG, FIP, volumes, server, unshelve
- [ ] Preemptible tag 2.72, BDM boot+data, `delete_on_termination=false` - [x] Preemptible tag 2.72, BDM boot+data, `delete_on_termination=false`
- [x] Flavor fallback по `FLAVOR_PREFERENCE` (ранжирование; create ещё нет) - [x] Flavor fallback по `FLAVOR_PREFERENCE` (ранжирование + выбор на `up`)
- [x] `doctor`: Keystone, квота, flavor, диск, Civitai `.red`, манифесты - [x] `doctor`: Keystone, квота, flavor, диск, Civitai `.red`, манифесты
- [x] Тесты с моками / без облака - [x] Тесты с моками / без облака
## 3. Сессия без туннеля ## 3. Сессия без туннеля
- [ ] `up` / `stop` / `status` / reconcile - [x] `up` / `stop` / `status` / reconcile (compute+диски+SSH+bootstrap SwarmUI)
- [ ] `status`: диск used/free, ₽/час, TTL preempt 24 ч, killer/hold - [ ] `status`: диск used/free, ₽/час, TTL preempt 24 ч, killer/hold
- [ ] Второй `up` не создаёт второй GPU - [x] Второй `up` не создаёт второй GPU
- [ ] `resize-data` вверх - [ ] `resize-data` вверх
- [ ] Генерация `~/.gpu-rent/id_ed25519` + keypair при первом `up` - [x] Генерация `~/.gpu-rent/id_ed25519` + keypair при первом `up`
- [ ] `seed-models` на живом диске (идемпотентно, SHA256) - [ ] `seed-models` на живом диске (идемпотентно, SHA256)
- [ ] `push` / `push-models`: `Models/`, `Wildcards/`, `CustomWorkflows/` - [ ] `push` / `push-models`: `Models/`, `Wildcards/`, `CustomWorkflows/`
- [ ] `pull-output` и `PULL_OUTPUT` на `stop` / повторный `up` - [ ] `pull-output` и `PULL_OUTPUT` на `stop` / повторный `up`
- [ ] `seed-extensions` + restart контейнера - [ ] `seed-extensions` + `systemctl restart swarmui`
- [ ] `destroy` только с флагом - [x] `destroy` только с флагом
## 4. Bootstrap SwarmUI + idle-killer ## 4. Bootstrap SwarmUI + idle-killer
- [ ] Идемпотентный first-boot, маркер FS - [x] Идемпотентный first-boot, маркер FS (`/mnt/swarm_data/.gpu-rent-ready`)
- [ ] Bind-mounts data volume → `/opt/swarmui`, systemd `swarmui`, auth - [x] Bind-mounts data volume → `/opt/swarmui`, systemd `swarmui` (auth SwarmUI ещё на spike)
- [ ] Clone `extensions.yaml` в Extensions / DLNodes до старта UI - [ ] Clone `extensions.yaml` в Extensions / DLNodes до старта UI
- [ ] Autocomplete: danbooru.csv + Settings.fds; на каждом up — GitHub sha - [ ] Autocomplete: danbooru.csv + Settings.fds; на каждом up — GitHub sha
- [ ] Civitai seed по манифесту; без токена — дефолт SwarmUI - [ ] Civitai seed по манифесту; без токена — дефолт SwarmUI
@@ -74,8 +74,8 @@
## 5. Туннель ## 5. Туннель
- [ ] `gpu-rent tunnel` → 17801, Ctrl+C не делает `stop` - [x] `gpu-rent tunnel` → 17801, Ctrl+C не делает `stop`
- [ ] `gpu-rent open` / `tunnel --open` - [x] `gpu-rent open` / `tunnel --open`
- [ ] EXPIRED → unshelve + reconnect, пока туннель жив - [ ] EXPIRED → unshelve + reconnect, пока туннель жив
- [ ] Сниппет MCP в stdout - [ ] Сниппет MCP в stdout
+20 -8
View File
@@ -10,20 +10,32 @@
Нужен **Python 3.11+**. На Windows при установке включи «Add python.exe to PATH». Нужен **Python 3.11+**. На Windows при установке включи «Add python.exe to PATH».
В корне репозитория: В корне репозитория достаточно лаунчера — venv и `pip install` он сделает сам:
```powershell
.\gpu-rent.ps1 --help
.\gpu-rent.ps1 doctor
```
```bat
gpu-rent.bat --help
```
```bash
chmod +x gpu-rent.sh
./gpu-rent.sh --help
```
Первый запуск копирует `env.example` в `%USERPROFILE%\.gpu-rent\.env` / `~/.gpu-rent/.env`, если файла ещё нет. Заполни `OS_*` (шаги ниже). После `git pull`, если изменился `pyproject.toml`, лаунчер переустановит пакет.
Ручной венв по желанию (для разработки / pytest):
```powershell ```powershell
python -m venv .venv python -m venv .venv
.\.venv\Scripts\activate .\.venv\Scripts\activate
python -m pip install -U pip python -m pip install -U pip
python -m pip install -e . python -m pip install -e ".[dev]"
```
Проверка:
```powershell
python -m gpu_rent --help python -m gpu_rent --help
gpu-rent --help
``` ```
--- ---
+15
View File
@@ -0,0 +1,15 @@
@echo off
setlocal EnableExtensions
cd /d "%~dp0"
chcp 65001 >nul
rem cmd.exe / Explorer: bypass ExecutionPolicy. Logic lives in gpu-rent.ps1.
if "%~1"=="" (
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "%~dp0gpu-rent.ps1"
echo.
pause
exit /b %ERRORLEVEL%
)
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "%~dp0gpu-rent.ps1" %*
exit /b %ERRORLEVEL%
+89
View File
@@ -0,0 +1,89 @@
#Requires -Version 5.1
# Windows launcher: venv + pip + python -m gpu_rent. Call via gpu-rent.bat if ExecutionPolicy blocks this file.
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$Root = Split-Path -Parent $MyInvocation.MyCommand.Path
Set-Location -LiteralPath $Root
try {
$utf8 = New-Object System.Text.UTF8Encoding $false
[Console]::OutputEncoding = $utf8
$script:OutputEncoding = $utf8
} catch {
}
function Test-Python311 {
param(
[Parameter(Mandatory = $true)][string]$Exe,
[string[]]$Prefix = @()
)
$cmd = Get-Command $Exe -ErrorAction SilentlyContinue
if (-not $cmd) {
return $false
}
try {
& $Exe @Prefix -c "import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)" 2>$null | Out-Null
return ($LASTEXITCODE -eq 0)
} catch {
return $false
}
}
$PyExe = $null
$PyPrefix = @()
$found = $false
foreach ($row in @(
@{ Exe = "py"; Prefix = @("-3.12") },
@{ Exe = "py"; Prefix = @("-3.11") },
@{ Exe = "py"; Prefix = @("-3") },
@{ Exe = "python3"; Prefix = @() },
@{ Exe = "python"; Prefix = @() }
)) {
if (Test-Python311 -Exe $row.Exe -Prefix $row.Prefix) {
$PyExe = $row.Exe
$PyPrefix = $row.Prefix
$found = $true
break
}
}
if (-not $found) {
Write-Host "gpu-rent: нужен Python 3.11+. Поставь с python.org и включи Add python.exe to PATH."
exit 1
}
$VenvPy = Join-Path $Root ".venv\Scripts\python.exe"
if (-not (Test-Path -LiteralPath $VenvPy)) {
Write-Host "gpu-rent: создаю .venv"
& $PyExe @PyPrefix -m venv (Join-Path $Root ".venv")
}
$Stamp = Join-Path $Root ".venv\.gpu-rent-installed"
$Pyproject = Join-Path $Root "pyproject.toml"
$needInstall = -not (Test-Path -LiteralPath $Stamp)
if (-not $needInstall -and (Test-Path -LiteralPath $Pyproject)) {
$needInstall = (Get-Item -LiteralPath $Pyproject).LastWriteTime -gt (Get-Item -LiteralPath $Stamp).LastWriteTime
}
if ($needInstall) {
Write-Host "gpu-rent: ставлю пакет в .venv"
& $VenvPy -m pip install -U pip
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
& $VenvPy -m pip install -e $Root
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
Get-Date -Format o | Set-Content -LiteralPath $Stamp -Encoding ascii
}
$GpuHome = Join-Path $env:USERPROFILE ".gpu-rent"
$EnvFile = Join-Path $GpuHome ".env"
$Example = Join-Path $Root "env.example"
if (-not (Test-Path -LiteralPath $EnvFile)) {
New-Item -ItemType Directory -Force -Path $GpuHome | Out-Null
if (Test-Path -LiteralPath $Example) {
Copy-Item -LiteralPath $Example -Destination $EnvFile
Write-Host "gpu-rent: created $EnvFile - fill OS_* (docs/setup.md)"
}
}
$ErrorActionPreference = "Continue"
& $VenvPy -m gpu_rent @args
exit $LASTEXITCODE
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/env bash
# Linux / macOS / Git Bash: venv + pip + python -m gpu_rent
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$ROOT"
ok_py() {
local exe="$1"
shift || true
command -v "$exe" >/dev/null 2>&1 || return 1
"$exe" "$@" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)' >/dev/null 2>&1
}
PY=()
if ok_py python3.12; then
PY=(python3.12)
elif ok_py python3.11; then
PY=(python3.11)
elif ok_py python3; then
PY=(python3)
elif ok_py python; then
PY=(python)
elif ok_py py -3.12; then
PY=(py -3.12)
elif ok_py py -3.11; then
PY=(py -3.11)
else
echo "gpu-rent: нужен Python 3.11+. Ubuntu: sudo apt install python3 python3-venv python3-pip" >&2
exit 1
fi
if [[ -x "$ROOT/.venv/Scripts/python.exe" ]]; then
VENV_PY="$ROOT/.venv/Scripts/python.exe"
elif [[ -x "$ROOT/.venv/bin/python" ]]; then
VENV_PY="$ROOT/.venv/bin/python"
else
echo "gpu-rent: создаю .venv"
"${PY[@]}" -m venv "$ROOT/.venv"
if [[ -x "$ROOT/.venv/Scripts/python.exe" ]]; then
VENV_PY="$ROOT/.venv/Scripts/python.exe"
else
VENV_PY="$ROOT/.venv/bin/python"
fi
fi
STAMP="$ROOT/.venv/.gpu-rent-installed"
if [[ ! -f "$STAMP" || "$ROOT/pyproject.toml" -nt "$STAMP" ]]; then
echo "gpu-rent: ставлю пакет в .venv"
"$VENV_PY" -m pip install -U pip
"$VENV_PY" -m pip install -e "$ROOT"
date -u +"%Y-%m-%dT%H:%M:%SZ" >"$STAMP"
fi
GPU_HOME="${HOME}/.gpu-rent"
if [[ ! -f "$GPU_HOME/.env" && -f "$ROOT/env.example" ]]; then
mkdir -p "$GPU_HOME"
cp "$ROOT/env.example" "$GPU_HOME/.env"
echo "gpu-rent: created $GPU_HOME/.env - fill OS_* (see docs/setup.md)"
fi
exec "$VENV_PY" -m gpu_rent "$@"
+3
View File
@@ -44,6 +44,9 @@ package-dir = { "" = "src" }
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
where = ["src"] where = ["src"]
[tool.setuptools.package-data]
gpu_rent = ["remote/*"]
[tool.pytest.ini_options] [tool.pytest.ini_options]
testpaths = ["tests"] testpaths = ["tests"]
pythonpath = ["src"] pythonpath = ["src"]
+33
View File
@@ -0,0 +1,33 @@
"""Upload and run the idempotent SwarmUI bootstrap on the VM."""
from __future__ import annotations
from collections.abc import Callable
from importlib.resources import files
from gpu_rent.config import Config
from gpu_rent.errors import CloudError
from gpu_rent.ssh_ops import run_script_sudo
Log = Callable[[str], None]
def bootstrap_script() -> str:
return files("gpu_rent.remote").joinpath("bootstrap.sh").read_text(encoding="utf-8")
def run_bootstrap(cfg: Config, host: str, log: Log) -> None:
log("bootstrap SwarmUI на VM (идемпотентно, без Docker)")
script = bootstrap_script()
out = run_script_sudo(
cfg,
host,
script,
remote_path="/tmp/gpu-rent-bootstrap.sh",
timeout=1800,
env={"SWARM_USER": cfg.ssh_user},
log=log,
)
if "bootstrap ok" not in out:
raise CloudError(f"bootstrap не подтвердил успех:\n{out[-800:]}")
log("диски и systemd unit готовы; дальше seed, потом start swarmui")
+50
View File
@@ -6,6 +6,8 @@ from dataclasses import dataclass
import httpx import httpx
from gpu_rent.errors import CloudError
ALLOWED_HOSTS = ("civitai.com", "civitai.red", "civitai.green") ALLOWED_HOSTS = ("civitai.com", "civitai.red", "civitai.green")
@@ -56,3 +58,51 @@ def probe_me(token: str, host: str, timeout: float = 15.0) -> CivitaiProbe:
status=response.status_code, status=response.status_code,
detail=response.text[:200] or response.reason_phrase, detail=response.text[:200] or response.reason_phrase,
) )
def pick_primary_file(version: dict) -> dict | None:
files = version.get("files") or []
if not isinstance(files, list):
return None
for item in files:
if isinstance(item, dict) and item.get("primary"):
return item
for item in files:
if not isinstance(item, dict):
continue
name = str(item.get("name") or "").lower()
if "safetensor" in name or name.endswith(".safetensors"):
return item
for item in files:
if isinstance(item, dict):
return item
return None
def fetch_model_version(token: str, host: str, version_id: int, timeout: float = 30.0) -> tuple[str, dict]:
"""GET /api/v1/model-versions/{id}; one retry on the other Civitai host."""
first = _normalize_host(host)
order = [first, other_host(first)]
last_error = "нет ответа"
seen: set[str] = set()
for candidate in order:
if candidate in seen or candidate not in ALLOWED_HOSTS:
continue
seen.add(candidate)
url = f"https://{candidate}/api/v1/model-versions/{version_id}"
try:
with httpx.Client(timeout=timeout, follow_redirects=True) as client:
response = client.get(url, headers={"Authorization": f"Bearer {token}"})
except httpx.HTTPError as exc:
last_error = str(exc)
continue
if response.status_code == 200:
data = response.json()
if isinstance(data, dict) and pick_primary_file(data):
return candidate, data
last_error = "пустой files[]"
continue
last_error = f"HTTP {response.status_code}"
if response.status_code not in {404, 400}:
break
raise CloudError(f"Civitai version {version_id}: {last_error} (хосты {', '.join(seen)})")
+142 -59
View File
@@ -18,7 +18,12 @@ from gpu_rent.config import load_config
from gpu_rent.doctor import blocking_failed, dry_run_plan, run_doctor from gpu_rent.doctor import blocking_failed, dry_run_plan, run_doctor
from gpu_rent.errors import GpuRentError, NotReadyError from gpu_rent.errors import GpuRentError, NotReadyError
from gpu_rent.os_client import connect, find_snapshot_by_name, find_tagged_servers from gpu_rent.os_client import connect, find_snapshot_by_name, find_tagged_servers
from gpu_rent.session import cmd_stop, cmd_up
from gpu_rent.ssh_ops import interactive_ssh, run_ssh
from gpu_rent.state import load_state, preempt_window_end from gpu_rent.state import load_state, preempt_window_end
from gpu_rent.provision import ensure_swarmui_running, seed_civitai, seed_extensions
from gpu_rent.sync_files import pull_tree, push_tree
from gpu_rent.tunnel import run_tunnel
if sys.platform == "win32": if sys.platform == "win32":
for _stream in (sys.stdout, sys.stderr): for _stream in (sys.stdout, sys.stderr):
@@ -55,10 +60,8 @@ def _die(exc: BaseException) -> None:
def _nyi(name: str) -> None: def _nyi(name: str) -> None:
raise NotReadyError( raise NotReadyError(
f"`{name}` ещё не создаёт/не гасит GPU. Сейчас работают: doctor, dry-run, status, open.\n" f"`{name}` ещё не готов. Уже работают: doctor, dry-run, status, open, up, stop, destroy, ssh, logs, tunnel, seed-*, push, pull-output.\n"
"1) Заполни ~/.gpu-rent/.env по docs/setup.md\n" "Ключи: docs/setup.md"
"2) gpu-rent doctor\n"
"3) Когда doctor зелёный и квота GPU > 0 — можно писать up."
) )
@@ -77,10 +80,18 @@ def _print_checks(checks) -> int:
if failed: if failed:
console.print("\n[red]Сессию начинать нельзя.[/red] См. docs/setup.md") console.print("\n[red]Сессию начинать нельзя.[/red] См. docs/setup.md")
return 1 return 1
console.print("\n[green]Можно идти дальше.[/green] mutating up пока не подключён.") console.print("\n[green]Можно идти дальше.[/green] Дальше: gpu-rent up --yes")
return 0 return 0
def _live():
cfg = load_config(require_auth=True)
state = load_state()
if not state.floating_ip:
raise GpuRentError("нет floating IP — сначала gpu-rent up")
return cfg, state.floating_ip
@app.command() @app.command()
def version() -> None: def version() -> None:
"""Версия пакета.""" """Версия пакета."""
@@ -185,14 +196,86 @@ def up(
yes: bool = typer.Option(False, "--yes", help="Без вопросов"), yes: bool = typer.Option(False, "--yes", help="Без вопросов"),
adopt: bool = typer.Option(False, "--adopt", help="Подхватить тег gpu-rent без state"), adopt: bool = typer.Option(False, "--adopt", help="Подхватить тег gpu-rent без state"),
) -> None: ) -> None:
"""Create/unshelve GPU. Пока: doctor, затем стоп — mutating ещё не подключён.""" """Create/unshelve GPU и bootstrap SwarmUI (без Docker)."""
del no_spot, flavor, yes, adopt
try: try:
checks = run_doctor() checks = run_doctor()
code = _print_checks(checks) code = _print_checks(checks)
if code != 0: if code != 0:
raise typer.Exit(1) raise typer.Exit(1)
_nyi("up") cfg = load_config(require_auth=True)
def confirm(msg: str) -> bool:
return typer.confirm(msg)
cmd_up(
cfg,
no_spot=no_spot,
flavor=flavor,
yes=yes,
adopt=adopt,
confirm=confirm,
log=lambda m: console.print(m),
)
except GpuRentError as exc:
_die(exc)
@app.command()
def stop(
no_pull: bool = typer.Option(False, "--no-pull"),
) -> None:
"""Удалить compute и FIP, диски оставить."""
try:
cfg = load_config(require_auth=True)
cmd_stop(cfg, no_pull=no_pull, log=lambda m: console.print(m))
except GpuRentError as exc:
_die(exc)
@app.command()
def destroy(
i_understand_data_loss: bool = typer.Option(False, "--i-understand-data-loss"),
) -> None:
"""stop + диски."""
if not i_understand_data_loss:
console.print("Нужен флаг --i-understand-data-loss")
raise typer.Exit(1)
try:
cfg = load_config(require_auth=True)
cmd_stop(cfg, destroy_disks=True, log=lambda m: console.print(m))
except GpuRentError as exc:
_die(exc)
@app.command()
def ssh() -> None:
"""Оболочка на VM (нужен живой compute и FIP)."""
try:
cfg = load_config(require_auth=True)
state = load_state()
if not state.floating_ip:
raise GpuRentError("нет floating IP в state — сначала gpu-rent up")
raise typer.Exit(interactive_ssh(cfg, state.floating_ip))
except GpuRentError as exc:
_die(exc)
@app.command()
def logs() -> None:
"""cloud-init / journalctl на VM."""
try:
cfg = load_config(require_auth=True)
state = load_state()
if not state.floating_ip:
raise GpuRentError("нет IP — VM не поднята")
out = run_ssh(
cfg,
state.floating_ip,
"sudo -n tail -n 80 /var/log/cloud-init-output.log 2>/dev/null; "
"systemctl is-active swarmui 2>/dev/null || true",
check=False,
)
console.print(out)
except GpuRentError as exc: except GpuRentError as exc:
_die(exc) _die(exc)
@@ -202,9 +285,17 @@ def tunnel(
open_browser: bool = typer.Option(False, "--open", help="Открыть браузер на 17801"), open_browser: bool = typer.Option(False, "--open", help="Открыть браузер на 17801"),
) -> None: ) -> None:
"""SSH localhost:17801 -> VM :7801. Ctrl+C закрывает туннель, GPU оставляет.""" """SSH localhost:17801 -> VM :7801. Ctrl+C закрывает туннель, GPU оставляет."""
del open_browser
try: try:
_nyi("tunnel") cfg = load_config(require_auth=True)
state = load_state()
if not state.floating_ip:
raise GpuRentError("нет floating IP — сначала gpu-rent up")
run_tunnel(
cfg,
state.floating_ip,
open_browser=open_browser,
log=lambda m: console.print(m),
)
except GpuRentError as exc: except GpuRentError as exc:
_die(exc) _die(exc)
@@ -223,84 +314,76 @@ def hold(
_die(exc) _die(exc)
@app.command()
def stop(
no_pull: bool = typer.Option(False, "--no-pull"),
) -> None:
"""Удалить compute и FIP, диски оставить."""
del no_pull
try:
_nyi("stop")
except GpuRentError as exc:
_die(exc)
@app.command()
def destroy(
i_understand_data_loss: bool = typer.Option(False, "--i-understand-data-loss"),
) -> None:
"""stop + диски."""
if not i_understand_data_loss:
console.print("Нужен флаг --i-understand-data-loss")
raise typer.Exit(1)
try:
_nyi("destroy")
except GpuRentError as exc:
_die(exc)
@app.command()
def logs() -> None:
try:
_nyi("logs")
except GpuRentError as exc:
_die(exc)
@app.command()
def ssh() -> None:
try:
_nyi("ssh")
except GpuRentError as exc:
_die(exc)
@app.command("seed-models") @app.command("seed-models")
def seed_models() -> None: def seed_models() -> None:
"""Докачать Civitai-манифест на живой диск."""
try: try:
_nyi("seed-models") cfg, host = _live()
seed_civitai(cfg, host, log=lambda m: console.print(m))
except GpuRentError as exc: except GpuRentError as exc:
_die(exc) _die(exc)
@app.command("push") @app.command("push")
def push_all() -> None: def push_all() -> None:
"""SFTP Models + Wildcards + CustomWorkflows."""
try: try:
_nyi("push") cfg, host = _live()
def log(msg: str) -> None:
console.print(msg)
push_tree(cfg, host, cfg.local_models_dir, "/mnt/swarm_data/Models", log, models=True)
push_tree(cfg, host, cfg.local_wildcards_dir, "/mnt/swarm_data/Data/Wildcards", log, models=False)
push_tree(cfg, host, cfg.local_workflows_dir, "/mnt/swarm_data/CustomWorkflows", log, models=False)
except GpuRentError as exc: except GpuRentError as exc:
_die(exc) _die(exc)
@app.command("push-models") @app.command("push-models")
def push_models() -> None: def push_models() -> None:
"""SFTP только ./Models."""
try: try:
_nyi("push-models") cfg, host = _live()
push_tree(
cfg,
host,
cfg.local_models_dir,
"/mnt/swarm_data/Models",
lambda m: console.print(m),
models=True,
)
except GpuRentError as exc: except GpuRentError as exc:
_die(exc) _die(exc)
@app.command("pull-output") @app.command("pull-output")
def pull_output() -> None: def pull_output_cmd() -> None:
"""Забрать новые файлы Output/ с VM."""
try: try:
_nyi("pull-output") cfg, host = _live()
pull_tree(
cfg,
host,
"/mnt/swarm_data/Output",
cfg.local_output_dir,
lambda m: console.print(m),
)
except GpuRentError as exc: except GpuRentError as exc:
_die(exc) _die(exc)
@app.command("seed-extensions") @app.command("seed-extensions")
def seed_extensions() -> None: def seed_extensions_cmd() -> None:
"""Clone/fetch extensions.yaml, затем restart swarmui."""
try: try:
_nyi("seed-extensions") cfg, host = _live()
def log(msg: str) -> None:
console.print(msg)
seed_extensions(cfg, host, log)
ensure_swarmui_running(cfg, host, log, restart=True)
except GpuRentError as exc: except GpuRentError as exc:
_die(exc) _die(exc)
+450
View File
@@ -0,0 +1,450 @@
"""Find-or-create Selectel OpenStack resources for one gpu-rent session."""
from __future__ import annotations
import time
from typing import Any, Callable
import httpx
from gpu_rent.errors import CloudError
from gpu_rent.os_client import (
BOOT_VOLUME_NAME,
BOOT_VOLUME_SIZE_GB,
DATA_VOLUME_NAME,
KEYPAIR_NAME,
NET_NAME,
PREEMPTIBLE_TAG,
RESOURCE_TAG,
ROUTER_NAME,
SERVER_NAME,
SG_NAME,
SUBNET_CIDR,
SUBNET_NAME,
find_snapshot_by_name,
find_tagged_servers,
find_volumes_by_name,
)
def _wrap(exc: BaseException, what: str) -> CloudError:
text = str(exc)
low = text.lower()
if "402" in text or "payment" in low:
return CloudError(f"{what}: нет средств / 402. {text}")
if "403" in text or "forbidden" in low or "quota" in low:
return CloudError(
f"{what}: квота или запрет (403). Часто GPU=0 — тикет в поддержку, docs/setup.md. {text}"
)
return CloudError(f"{what}: {text}")
def _oid(obj: Any) -> str:
return str(getattr(obj, "id", "") or "")
def guess_operator_cidr() -> str:
try:
response = httpx.get("https://ifconfig.me/ip", timeout=5.0)
ip = response.text.strip()
if ip.count(".") == 3 and all(p.isdigit() for p in ip.split(".")):
return f"{ip}/32"
except httpx.HTTPError:
pass
return "0.0.0.0/0"
def wait_volume(conn, volume, status: str = "available", timeout: int = 900) -> Any:
deadline = time.time() + timeout
vid = _oid(volume)
last = None
while time.time() < deadline:
current = conn.block_storage.get_volume(vid)
last = getattr(current, "status", None)
if last == status:
return current
if last in {"error", "error_restoring"}:
raise CloudError(f"том {vid} статус {last}")
time.sleep(5)
raise CloudError(f"том {vid} не стал {status} (последний {last})")
def wait_server(conn, server, status: str = "ACTIVE", timeout: int = 900) -> Any:
deadline = time.time() + timeout
sid = _oid(server)
last = None
while time.time() < deadline:
current = conn.compute.get_server(sid)
last = (getattr(current, "status", None) or "").upper()
if last == status.upper():
return current
if last in {"ERROR"}:
fault = getattr(current, "fault", None)
raise CloudError(f"сервер {sid} ERROR{f' {fault}' if fault else ''}")
time.sleep(5)
raise CloudError(f"сервер {sid} не стал {status} (последний {last})")
def wait_gone(fetch: Callable[[], Any], timeout: int = 300) -> None:
deadline = time.time() + timeout
while time.time() < deadline:
try:
obj = fetch()
except Exception:
return
if obj is None:
return
time.sleep(4)
raise CloudError("ресурс не исчез вовремя")
def find_external_network(conn) -> Any:
for net in conn.network.networks():
if getattr(net, "is_router_external", False):
return net
for net in conn.network.networks():
name = (getattr(net, "name", "") or "").lower()
if name in {"external-network", "wan", "public"}:
return net
raise CloudError("нет внешней сети для floating IP / router gateway")
def ensure_network(conn, log: Callable[[str], None]) -> tuple[Any, Any]:
net = conn.network.find_network(NET_NAME)
if net:
subnet = next(iter(conn.network.subnets(network_id=net.id)), None)
if not subnet:
raise CloudError(f"сеть {NET_NAME} есть, subnet нет")
log(f"сеть {NET_NAME} уже есть")
return net, subnet
try:
net = conn.network.create_network(name=NET_NAME)
subnet = conn.network.create_subnet(
name=SUBNET_NAME,
network_id=net.id,
ip_version=4,
cidr=SUBNET_CIDR,
dns_nameservers=["1.1.1.1", "8.8.8.8"],
)
ext = find_external_network(conn)
router = conn.network.find_router(ROUTER_NAME)
if not router:
router = conn.network.create_router(
name=ROUTER_NAME,
external_gateway_info={"network_id": ext.id},
)
conn.network.add_interface_to_router(router, subnet_id=subnet.id)
log(f"создана сеть {NET_NAME} + router")
return net, subnet
except Exception as exc:
raise _wrap(exc, "сеть") from exc
def ensure_security_group(conn, cidr: str, log: Callable[[str], None]) -> Any:
sg = conn.network.find_security_group(SG_NAME)
if sg:
log(f"security group {SG_NAME} уже есть")
return sg
try:
sg = conn.network.create_security_group(
name=SG_NAME,
description="gpu-rent SSH only",
)
conn.network.create_security_group_rule(
security_group_id=sg.id,
direction="ingress",
ethertype="IPv4",
protocol="tcp",
port_range_min=22,
port_range_max=22,
remote_ip_prefix=cidr,
)
log(f"SG {SG_NAME}: TCP/22 с {cidr}")
return sg
except Exception as exc:
raise _wrap(exc, "security group") from exc
def ensure_keypair(conn, public_key: str, log: Callable[[str], None]) -> Any:
existing = conn.compute.find_keypair(KEYPAIR_NAME)
pub = public_key.strip()
if existing:
have = (getattr(existing, "public_key", "") or "").strip()
if have == pub:
log(f"keypair {KEYPAIR_NAME} совпадает")
return existing
log(f"keypair {KEYPAIR_NAME} другой — пересоздаём")
try:
conn.compute.delete_keypair(existing)
except Exception as exc:
raise _wrap(exc, "удалить keypair") from exc
try:
key = conn.compute.create_keypair(name=KEYPAIR_NAME, public_key=pub)
log(f"keypair {KEYPAIR_NAME} создан")
return key
except Exception as exc:
raise _wrap(exc, "keypair") from exc
def ensure_boot_volume(
conn,
*,
az: str,
volume_type: str | None,
image_id: str | None,
snapshot_name: str,
existing_id: str,
log: Callable[[str], None],
) -> Any:
if existing_id:
vol = conn.block_storage.get_volume(existing_id)
log(f"boot volume из state {existing_id}")
return vol
found = find_volumes_by_name(conn, BOOT_VOLUME_NAME)
if found:
log(f"boot volume {BOOT_VOLUME_NAME} уже в проекте")
return found[0]
snap = find_snapshot_by_name(conn, snapshot_name)
kwargs: dict[str, Any] = {
"name": BOOT_VOLUME_NAME,
"size": BOOT_VOLUME_SIZE_GB,
"availability_zone": az,
}
if volume_type:
kwargs["volume_type"] = volume_type
if snap:
kwargs["snapshot_id"] = snap.id
log(f"boot volume из snapshot {snapshot_name}")
elif image_id:
kwargs["image_id"] = image_id
log("boot volume из GPU-образа")
else:
raise CloudError("нет ни snapshot, ни image_id для boot volume")
try:
vol = conn.block_storage.create_volume(**kwargs)
return wait_volume(conn, vol)
except TypeError:
if "image_id" in kwargs:
kwargs["imageRef"] = kwargs.pop("image_id")
try:
vol = conn.block_storage.create_volume(**kwargs)
return wait_volume(conn, vol)
except Exception as exc:
raise _wrap(exc, "boot volume") from exc
except Exception as exc:
raise _wrap(exc, "boot volume") from exc
def ensure_data_volume(
conn,
*,
az: str,
volume_type: str | None,
size_gb: int,
existing_id: str,
log: Callable[[str], None],
) -> Any:
if existing_id:
vol = conn.block_storage.get_volume(existing_id)
log(f"data volume из state {existing_id}")
return vol
found = find_volumes_by_name(conn, DATA_VOLUME_NAME)
if found:
log(f"data volume {DATA_VOLUME_NAME} уже в проекте")
return found[0]
kwargs: dict[str, Any] = {
"name": DATA_VOLUME_NAME,
"size": size_gb,
"availability_zone": az,
}
if volume_type:
kwargs["volume_type"] = volume_type
try:
vol = conn.block_storage.create_volume(**kwargs)
log(f"data volume {size_gb} GB")
return wait_volume(conn, vol)
except Exception as exc:
raise _wrap(exc, "data volume") from exc
def _tag_server(conn, server, spot: bool) -> None:
tags = [RESOURCE_TAG]
if spot:
tags.append(PREEMPTIBLE_TAG)
try:
conn.compute.set_server_tags(server, tags)
except Exception:
try:
for tag in tags:
conn.compute.add_tag_to_server(server, tag)
except Exception:
pass
def create_gpu_server(
conn,
*,
flavor_id: str,
net_id: str,
sg_name: str,
boot_volume_id: str,
data_volume_id: str,
az: str,
spot: bool,
log: Callable[[str], None],
) -> Any:
bdm = [
{
"boot_index": 0,
"uuid": boot_volume_id,
"source_type": "volume",
"destination_type": "volume",
"delete_on_termination": False,
},
{
"boot_index": 1,
"uuid": data_volume_id,
"source_type": "volume",
"destination_type": "volume",
"delete_on_termination": False,
},
]
tags = [RESOURCE_TAG]
if spot:
tags.append(PREEMPTIBLE_TAG)
kwargs: dict[str, Any] = {
"name": SERVER_NAME,
"flavor_id": flavor_id,
"networks": [{"uuid": net_id}],
"key_name": KEYPAIR_NAME,
"availability_zone": az,
"block_device_mapping_v2": bdm,
"security_groups": [{"name": sg_name}],
"tags": tags,
}
try:
server = conn.compute.create_server(**kwargs)
except Exception as exc:
kwargs.pop("tags", None)
try:
server = conn.compute.create_server(**kwargs)
except Exception as exc2:
raise _wrap(exc2, "create server") from exc
log("ждём Nova ACTIVE (GPU create может занять несколько минут)")
server = wait_server(conn, server, "ACTIVE")
_tag_server(conn, server, spot)
return server
def unshelve(conn, server, log: Callable[[str], None]) -> Any:
log("unshelve (EXPIRED / shelved)")
try:
conn.compute.unshelve_server(server)
except Exception as exc:
raise _wrap(exc, "unshelve") from exc
return wait_server(conn, server, "ACTIVE", timeout=900)
def associate_floating_ip(conn, server, log: Callable[[str], None]) -> tuple[str, str]:
ports = list(conn.network.ports(device_id=server.id))
if not ports:
raise CloudError("у сервера нет neutron-порта — не к чему привязать FIP")
ext = find_external_network(conn)
try:
fip = conn.network.create_ip(floating_network_id=ext.id)
except Exception as exc:
raise _wrap(exc, "floating IP allocate") from exc
try:
fip = conn.network.update_ip(fip, port_id=ports[0].id)
except Exception as exc:
raise _wrap(exc, "floating IP associate") from exc
addr = getattr(fip, "floating_ip_address", None) or getattr(fip, "name", None)
log(f"floating IP {addr}")
return str(addr), _oid(fip)
def ensure_floating_ip(
conn,
server,
existing_id: str | None,
existing_addr: str | None,
log: Callable[[str], None],
) -> tuple[str, str]:
have = server_floating_ip(server)
if have:
return have, existing_id or ""
ports = list(conn.network.ports(device_id=server.id))
if existing_id and ports:
try:
fip = conn.network.update_ip(existing_id, port_id=ports[0].id)
addr = getattr(fip, "floating_ip_address", None) or existing_addr
log(f"вернули FIP {addr}")
return str(addr), existing_id
except Exception:
log("старый FIP не привязался — выделяем новый")
return associate_floating_ip(conn, server, log)
def delete_floating_ip(conn, fip_id: str | None, address: str | None, log: Callable[[str], None]) -> None:
if fip_id:
try:
conn.network.delete_ip(fip_id, ignore_missing=True)
log(f"удалён FIP {fip_id}")
return
except Exception:
pass
if address:
for ip in conn.network.ips():
if getattr(ip, "floating_ip_address", None) == address:
conn.network.delete_ip(ip, ignore_missing=True)
log(f"удалён FIP {address}")
return
def delete_server(conn, server, log: Callable[[str], None]) -> None:
sid = _oid(server)
log(f"удаляем compute {sid} (диски оставляем)")
try:
conn.compute.delete_server(server, ignore_missing=True)
except Exception as exc:
raise _wrap(exc, "delete server") from exc
def _get():
try:
current = conn.compute.get_server(sid)
except Exception:
return None
if current is None:
return None
if server_status(current) in {"DELETED", "SOFT_DELETED"}:
return None
return current
wait_gone(_get, timeout=420)
def server_status(server: Any) -> str:
return (getattr(server, "status", None) or "").upper()
def server_floating_ip(server: Any) -> str | None:
addrs = getattr(server, "addresses", None) or {}
if not isinstance(addrs, dict):
return None
for nets in addrs.values():
if not isinstance(nets, list):
continue
for item in nets:
if not isinstance(item, dict):
continue
if item.get("OS-EXT-IPS:type") == "floating":
return str(item.get("addr") or "") or None
return None
def pick_existing_server(conn) -> Any | None:
found = find_tagged_servers(conn)
if len(found) > 1:
ids = ", ".join(_oid(s) for s in found)
raise CloudError(f"несколько серверов gpu-rent: {ids}. Разбери вручную или --adopt один.")
return found[0] if found else None
+3 -23
View File
@@ -26,6 +26,7 @@ from gpu_rent.os_client import (
iter_volume_types, iter_volume_types,
volume_quotas, volume_quotas,
) )
from gpu_rent.payload import folder_bytes as _folder_bytes, has_payload as _has_payload
from gpu_rent.paths import env_path, home_dir from gpu_rent.paths import env_path, home_dir
from gpu_rent.ssh_keys import key_ready from gpu_rent.ssh_keys import key_ready
from gpu_rent.state import load_state from gpu_rent.state import load_state
@@ -39,28 +40,6 @@ class Check:
detail: str detail: str
SKIP = {".gitkeep", "README.md", ".gitignore"}
def _folder_bytes(root: Path) -> int:
if not root.is_dir():
return 0
total = 0
for path in root.rglob("*"):
if path.is_file() and path.name not in SKIP:
total += path.stat().st_size
return total
def _has_payload(root: Path) -> bool:
if not root.is_dir():
return False
for path in root.rglob("*"):
if path.is_file() and path.name not in SKIP:
return True
return False
def run_doctor() -> list[Check]: def run_doctor() -> list[Check]:
checks: list[Check] = [] checks: list[Check] = []
home_dir().mkdir(parents=True, exist_ok=True) home_dir().mkdir(parents=True, exist_ok=True)
@@ -383,7 +362,8 @@ def dry_run_plan(checks: list[Check]) -> list[str]:
f"preemptible: {cfg.default_spot} (обычный сервер: gpu-rent up --no-spot)", f"preemptible: {cfg.default_spot} (обычный сервер: gpu-rent up --no-spot)",
f"idle-killer: {cfg.idle_minutes} мин пустой очереди, льгота {cfg.idle_grace_minutes} мин", f"idle-killer: {cfg.idle_minutes} мин пустой очереди, льгота {cfg.idle_grace_minutes} мин",
f"туннель: localhost:{cfg.swarmui_local_port} -> VM :7801", f"туннель: localhost:{cfg.swarmui_local_port} -> VM :7801",
"сейчас mutating up/stop ещё не подключены — только doctor / dry-run / status / open", "gpu-rent up --yes создаст сеть/диски/compute и поставит SwarmUI (если doctor зелёный и квота GPU > 0)",
"после up: gpu-rent tunnel (Ctrl+C не гасит GPU)",
] ]
flavor = next((c.detail for c in checks if c.name == "flavor" and c.ok), None) flavor = next((c.detail for c in checks if c.name == "flavor" and c.ok), None)
if flavor: if flavor:
+28
View File
@@ -167,3 +167,31 @@ def pick_boot_image(images: list[Any]) -> Any | None:
return None return None
ranked.sort(key=lambda item: item[0], reverse=True) ranked.sort(key=lambda item: item[0], reverse=True)
return ranked[0][1] return ranked[0][1]
def resolve_flavor(
flavors: list[Any],
preference: tuple[str, ...],
*,
explicit: str | None = None,
default_id: str | None = None,
fallback: bool = True,
) -> FlavorInfo:
if explicit:
for flavor in flavors:
if _id(flavor) == explicit or _name(flavor) == explicit:
if is_disabled(flavor):
raise ValueError(f"flavor {explicit} disabled")
return flavor_info(flavor, label="explicit")
raise ValueError(f"flavor {explicit} не найден в регионе")
if not fallback:
if not default_id:
raise ValueError("FLAVOR_FALLBACK=false требует DEFAULT_FLAVOR_ID или --flavor")
return resolve_flavor(flavors, preference, explicit=default_id, fallback=True)
gpu = [f for f in flavors if looks_like_gpu(f)]
ranked = rank_flavors(gpu or flavors, preference)
if ranked:
return ranked[0]
if default_id:
return resolve_flavor(flavors, preference, explicit=default_id, fallback=True)
raise ValueError("нет доступного GPU flavor из FLAVOR_PREFERENCE")
+59
View File
@@ -0,0 +1,59 @@
"""Single-instance lock for up/stop."""
from __future__ import annotations
import os
import sys
from types import TracebackType
from gpu_rent.errors import GpuRentError
from gpu_rent.paths import lock_path
def _pid_alive(pid: int) -> bool:
if pid <= 0:
return False
if sys.platform == "win32":
import ctypes
handle = ctypes.windll.kernel32.OpenProcess(0x100000, False, pid)
if handle:
ctypes.windll.kernel32.CloseHandle(handle)
return True
return False
try:
os.kill(pid, 0)
return True
except OSError:
return False
class SessionLock:
def __enter__(self) -> SessionLock:
path = lock_path()
path.parent.mkdir(parents=True, exist_ok=True)
if path.is_file():
try:
old = int(path.read_text(encoding="utf-8").strip() or "0")
except ValueError:
old = 0
if _pid_alive(old) and old != os.getpid():
raise GpuRentError(
f"gpu-rent уже работает (pid {old}, {path}). Дождись окончания up/stop."
)
path.write_text(str(os.getpid()), encoding="utf-8")
self._path = path
self._pid = os.getpid()
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc: BaseException | None,
tb: TracebackType | None,
) -> None:
try:
if self._path.is_file() and self._path.read_text(encoding="utf-8").strip() == str(self._pid):
self._path.unlink()
except OSError:
pass
+57 -2
View File
@@ -2,9 +2,11 @@
from __future__ import annotations from __future__ import annotations
import re
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from urllib.parse import parse_qs, urlparse
import yaml import yaml
@@ -17,8 +19,24 @@ MODEL_TYPES = (
"embedding", "embedding",
"controlnet", "controlnet",
"upscaler", "upscaler",
"clip",
) )
MODEL_DIRS = {
"checkpoint": "Stable-Diffusion",
"lora": "Lora",
"vae": "VAE",
"embedding": "Embeddings",
"controlnet": "controlnet",
"upscaler": "upscale_models",
"clip": "clip",
}
_VERSION_QS = re.compile(r"modelVersionId=(\d+)", re.I)
_VERSION_PATH = re.compile(r"/model-versions/(\d+)", re.I)
_DOWNLOAD_PATH = re.compile(r"/api/download/models/(\d+)", re.I)
_SHA_REF = re.compile(r"^[0-9a-fA-F]{40}$")
@dataclass @dataclass
class ModelEntry: class ModelEntry:
@@ -71,7 +89,10 @@ def parse_models(path: Path) -> list[ModelEntry]:
if vid in (0, "0", None) and not url: if vid in (0, "0", None) and not url:
continue continue
version_id = int(vid) if vid not in (None, "", 0, "0") else None version_id = int(vid) if vid not in (None, "", 0, "0") else None
entries.append(ModelEntry(kind=kind, version_id=version_id, url=str(url) if url else None)) url_s = str(url) if url else None
if version_id is None and url_s:
version_id = extract_version_id(url_s)
entries.append(ModelEntry(kind=kind, version_id=version_id, url=url_s))
return entries return entries
@@ -94,9 +115,43 @@ def parse_extensions(path: Path) -> list[GitRepo]:
repos.append( repos.append(
GitRepo( GitRepo(
kind=kind, kind=kind,
url=str(item["url"]), url=str(item["url"]).strip(),
ref=str(item.get("ref") or "main"), ref=str(item.get("ref") or "main"),
directory=str(item["dir"]) if item.get("dir") else None, directory=str(item["dir"]) if item.get("dir") else None,
) )
) )
return repos return repos
def extract_version_id(url: str) -> int | None:
text = url.strip()
for rx in (_VERSION_QS, _VERSION_PATH, _DOWNLOAD_PATH):
match = rx.search(text)
if match:
return int(match.group(1))
parsed = urlparse(text)
ids = parse_qs(parsed.query).get("modelVersionId") or parse_qs(parsed.query).get("modelversionid")
if ids:
try:
return int(ids[0])
except ValueError:
return None
return None
def repo_dirname(repo: GitRepo) -> str:
if repo.directory:
return repo.directory
name = repo.url.rstrip("/").rsplit("/", 1)[-1]
if name.endswith(".git"):
name = name[:-4]
return name or "extension"
def is_commit_sha(ref: str) -> bool:
return bool(_SHA_REF.match(ref.strip()))
def remote_root_for(repo: GitRepo) -> str:
base = "/mnt/swarm_data/Extensions" if repo.kind == "swarmui" else "/mnt/swarm_data/DLNodes"
return f"{base}/{repo_dirname(repo)}"
+16
View File
@@ -9,8 +9,20 @@ from gpu_rent.config import Config
from gpu_rent.errors import CloudError from gpu_rent.errors import CloudError
RESOURCE_TAG = "gpu-rent" RESOURCE_TAG = "gpu-rent"
PREEMPTIBLE_TAG = "preemptible"
COMPUTE_MICROVERSION = "2.72" COMPUTE_MICROVERSION = "2.72"
SERVER_NAME = "gpu-rent"
KEYPAIR_NAME = "gpu-rent"
NET_NAME = "gpu-rent"
SUBNET_NAME = "gpu-rent-subnet"
ROUTER_NAME = "gpu-rent"
SG_NAME = "gpu-rent"
BOOT_VOLUME_NAME = "gpu-rent-boot"
DATA_VOLUME_NAME = "gpu-rent-data"
SUBNET_CIDR = "192.168.77.0/24"
BOOT_VOLUME_SIZE_GB = 40
def connect(cfg: Config): def connect(cfg: Config):
try: try:
@@ -34,6 +46,10 @@ def connect(cfg: Config):
app_version="0.1.0", app_version="0.1.0",
) )
conn.authorize() conn.authorize()
try:
conn.compute.default_microversion = COMPUTE_MICROVERSION
except Exception:
pass
except Exception as exc: except Exception as exc:
raise CloudError( raise CloudError(
"Keystone не выдал токен. Проверь OS_USERNAME / OS_PASSWORD / " "Keystone не выдал токен. Проверь OS_USERNAME / OS_PASSWORD / "
+93
View File
@@ -0,0 +1,93 @@
"""Local app trees: Models / Wildcards / CustomWorkflows / Output."""
from __future__ import annotations
import hashlib
from pathlib import Path
SKIP_NAMES = {".gitkeep", "README.md", "README.txt", ".gitignore"}
WEIGHT_SUFFIXES = {".safetensors", ".ckpt", ".pt", ".pth", ".bin", ".gguf", ".sft", ".onnx"}
META_SUFFIXES = {".json", ".civitai.json", ".swarm.json", ".preview.png", ".png", ".webp", ".jpg"}
def is_skipped(path: Path) -> bool:
name = path.name
if name in SKIP_NAMES or name.startswith("."):
return True
return False
def has_payload(root: Path) -> bool:
if not root.is_dir():
return False
for path in root.rglob("*"):
if path.is_file() and not is_skipped(path):
return True
return False
def folder_bytes(root: Path) -> int:
if not root.is_dir():
return 0
total = 0
for path in root.rglob("*"):
if path.is_file() and not is_skipped(path):
total += path.stat().st_size
return total
def iter_payload_files(root: Path) -> list[Path]:
if not root.is_dir():
return []
found = []
for path in sorted(root.rglob("*")):
if path.is_file() and not is_skipped(path):
found.append(path)
return found
def is_weight(path: Path) -> bool:
return path.suffix.lower() in WEIGHT_SUFFIXES
def sha256_file(path: Path, chunk: int = 1024 * 1024) -> str:
digest = hashlib.sha256()
with path.open("rb") as fh:
while True:
block = fh.read(chunk)
if not block:
break
digest.update(block)
return digest.hexdigest()
def sidecar_stem(name: str) -> str:
lower = name.lower()
for suffix in (
".civitai.json",
".swarm.json",
".preview.png",
".preview.webp",
".preview.jpg",
".json",
".png",
".webp",
".jpg",
):
if lower.endswith(suffix):
return name[: -len(suffix)]
return Path(name).stem
def model_push_set(root: Path) -> list[Path]:
"""Weights plus same-stem sidecars. Sidecar without weights is skipped."""
files = iter_payload_files(root)
weights = [p for p in files if is_weight(p)]
wanted: set[Path] = set(weights)
stems = {p.stem for p in weights}
for path in files:
if is_weight(path):
continue
if sidecar_stem(path.name) in stems:
wanted.add(path)
return sorted(wanted)
+269
View File
@@ -0,0 +1,269 @@
"""After OS bootstrap: extensions, autocomplete, Civitai, then start SwarmUI."""
from __future__ import annotations
import json
import shlex
from collections.abc import Callable
from datetime import datetime, timezone
from importlib.resources import files
from pathlib import Path
import httpx
from gpu_rent.civitai import fetch_model_version, pick_primary_file
from gpu_rent.config import Config
from gpu_rent.errors import CloudError, GpuRentError
from gpu_rent.manifests import (
MODEL_DIRS,
extract_version_id,
parse_extensions,
parse_models,
remote_root_for,
)
from gpu_rent.ssh_ops import put_text, remote_exists, run_python, run_ssh
from gpu_rent.sync_files import pull_tree, push_tree
Log = Callable[[str], None]
DATA = "/mnt/swarm_data"
def _pkg_text(name: str) -> str:
return files("gpu_rent.remote").joinpath(name).read_text(encoding="utf-8")
def seed_extensions(cfg: Config, host: str, log: Log) -> bool:
repos = parse_extensions(cfg.extensions_manifest)
if not repos:
log("extensions.yaml пуст — стоковый SwarmUI")
return False
jobs = []
for repo in repos:
jobs.append(
{
"kind": repo.kind,
"url": repo.url,
"ref": repo.ref,
"dest": remote_root_for(repo),
}
)
put_text(cfg, host, "/tmp/gpu-rent-ext.json", json.dumps(jobs, indent=2))
if cfg.git_token:
put_text(cfg, host, "/tmp/gpu-rent-git.token", cfg.git_token + "\n", mode=0o600)
log(f"clone {len(jobs)} git-реп на data volume")
out = run_python(
cfg,
host,
_pkg_text("clone_ext.py"),
remote_path="/tmp/gpu-rent-clone_ext.py",
timeout=1800,
log=log,
)
return "cloned " in out or "updated " in out
def _github_blob(cfg: Config) -> dict | None:
url = (
f"https://api.github.com/repos/{cfg.autocomplete_github_repo}/contents/"
f"{cfg.autocomplete_github_path}?ref={cfg.autocomplete_github_ref}"
)
headers = {"Accept": "application/vnd.github+json", "User-Agent": "gpu-rent"}
if cfg.git_token:
headers["Authorization"] = f"Bearer {cfg.git_token}"
try:
with httpx.Client(timeout=20.0, follow_redirects=True) as client:
response = client.get(url, headers=headers)
except httpx.HTTPError as exc:
log_skip = str(exc)
return {"error": log_skip}
if response.status_code == 403:
return {"error": "GitHub rate limit — autocomplete не обновляю"}
if response.status_code != 200:
return {"error": f"GitHub HTTP {response.status_code}"}
data = response.json()
if not isinstance(data, dict):
return {"error": "неожиданный JSON GitHub"}
return data
def seed_autocomplete(cfg: Config, host: str, log: Log) -> bool:
if not cfg.autocomplete_enabled:
log("autocomplete выключен")
return False
dest_dir = f"{DATA}/Data/Autocompletions"
dest = f"{dest_dir}/{cfg.autocomplete_filename}"
meta_path = f"{dest}.gpu-rent-meta.json"
blob = _github_blob(cfg)
if blob is None:
return False
if blob.get("error"):
log(str(blob["error"]))
return False
sha = str(blob.get("sha") or "")
download_url = str(blob.get("download_url") or "")
old_sha = ""
if remote_exists(cfg, host, meta_path):
raw = run_ssh(cfg, host, f"cat {meta_path}", check=False)
try:
old_sha = str(json.loads(raw).get("github_blob_sha") or "")
except json.JSONDecodeError:
old_sha = ""
changed = sha != old_sha or not remote_exists(cfg, host, dest)
if changed:
if not download_url:
log("GitHub не дал download_url")
return False
log(f"качаю {cfg.autocomplete_filename}")
run_ssh(
cfg,
host,
"mkdir -p {dir} && curl -fsSL -o {part} {url} && mv {part} {dest}".format(
dir=shlex.quote(dest_dir),
part=shlex.quote(dest + ".partial"),
url=shlex.quote(download_url),
dest=shlex.quote(dest),
),
timeout=180,
)
meta = {
"repo": cfg.autocomplete_github_repo,
"path": cfg.autocomplete_github_path,
"ref": cfg.autocomplete_github_ref,
"github_blob_sha": sha,
"filename": cfg.autocomplete_filename,
"fetched_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
"settings_applied": True,
}
put_text(cfg, host, meta_path, json.dumps(meta, indent=2) + "\n")
settings = f"{DATA}/Data/Settings.fds"
applied = False
if remote_exists(cfg, host, meta_path):
raw = run_ssh(cfg, host, f"cat {meta_path}", check=False)
try:
applied = bool(json.loads(raw).get("settings_applied"))
except json.JSONDecodeError:
applied = False
if not remote_exists(cfg, host, settings) or not applied:
fds = (
"DefaultUser:\n"
" AutoComplete:\n"
f" Source: {cfg.autocomplete_filename}\n"
" EscapeParens: true\n"
)
put_text(cfg, host, settings, fds)
log(f"Settings.fds AutoComplete.Source = {cfg.autocomplete_filename}")
return changed
def _download_url(host: str, version_id: int, file_info: dict) -> str:
raw = str(file_info.get("downloadUrl") or "")
if "civitai." in raw and "/api/download/" in raw:
# NSFW downloadUrl often points at .com — use the host that had files.
return f"https://{host}/api/download/models/{version_id}"
if raw.startswith("https://"):
return raw
return f"https://{host}/api/download/models/{version_id}"
def seed_civitai(cfg: Config, host: str, log: Log) -> None:
entries = parse_models(cfg.models_manifest)
if not cfg.civitai_api_token:
log("Civitai-seed пропущен: нет CIVITAI_API_TOKEN — дефолт SwarmUI")
return
if not entries:
log("Civitai-seed пропущен: манифест пуст — дефолт SwarmUI")
return
jobs = []
for entry in entries:
vid = entry.version_id or (extract_version_id(entry.url) if entry.url else None)
if not vid:
log(f"пропуск {entry.kind}: нет version_id")
continue
try:
api_host, version = fetch_model_version(cfg.civitai_api_token, cfg.civitai_api_host, vid)
except CloudError as exc:
log(str(exc))
continue
info = pick_primary_file(version)
if not info:
log(f"version {vid}: нет files[]")
continue
name = str(info.get("name") or f"{vid}.safetensors")
folder = MODEL_DIRS.get(entry.kind, entry.kind)
dest = f"{DATA}/Models/{folder}/{name}"
hashes = info.get("hashes") or {}
sha = str((hashes.get("SHA256") or hashes.get("sha256") or "")).lower()
stem = Path(name).stem
civitai_json = json.dumps(version, ensure_ascii=False, indent=2)
trained = version.get("trainedWords") or []
phrase = trained[0] if isinstance(trained, list) and trained else ""
swarm = {
"name": stem,
"title": version.get("name") or stem,
"description": (version.get("description") or "")[:2000],
"trigger_phrase": phrase,
"author": ((version.get("model") or {}) if isinstance(version.get("model"), dict) else {}).get("name"),
"tags": version.get("tags") or [],
}
jobs.append(
{
"dest": dest,
"url": _download_url(api_host, vid, info),
"sha256": sha,
"sidecars": {
f"{stem}.civitai.json": civitai_json,
f"{stem}.swarm.json": json.dumps(swarm, ensure_ascii=False, indent=2),
},
}
)
if not jobs:
log("Civitai-seed: ни одной скачиваемой строки")
return
if not any(e.kind == "checkpoint" for e in entries):
log("в манифесте нет checkpoint — генерация может не стартовать")
put_text(cfg, host, "/tmp/gpu-rent-civitai-jobs.json", json.dumps(jobs, indent=2))
put_text(cfg, host, "/tmp/gpu-rent-civitai.token", cfg.civitai_api_token + "\n", mode=0o600)
log(f"Civitai: качаю {len(jobs)} файл(ов) на VM")
run_python(
cfg,
host,
_pkg_text("civitai_fetch.py"),
remote_path="/tmp/gpu-rent-civitai_fetch.py",
timeout=7200,
log=log,
)
def ensure_swarmui_running(cfg: Config, host: str, log: Log, restart: bool) -> None:
active = run_ssh(cfg, host, "systemctl is-active swarmui 2>/dev/null || true", check=False).strip()
if restart and active == "active":
log("systemctl restart swarmui (новые extensions/autocomplete)")
run_ssh(cfg, host, "sudo -n systemctl restart swarmui", timeout=120)
return
if active != "active":
log("systemctl start swarmui")
run_ssh(cfg, host, "sudo -n systemctl start swarmui", timeout=120)
run_ssh(cfg, host, "sudo -n systemctl enable swarmui", check=False)
def provision_vm(cfg: Config, host: str, log: Log) -> None:
restart = False
try:
if seed_extensions(cfg, host, log):
restart = True
except GpuRentError as exc:
log(f"extensions: {exc}")
raise
try:
if seed_autocomplete(cfg, host, log):
restart = True
except GpuRentError as exc:
log(f"autocomplete: {exc}")
seed_civitai(cfg, host, log)
push_tree(cfg, host, cfg.local_models_dir, f"{DATA}/Models", log, models=True)
push_tree(cfg, host, cfg.local_wildcards_dir, f"{DATA}/Data/Wildcards", log, models=False)
push_tree(cfg, host, cfg.local_workflows_dir, f"{DATA}/CustomWorkflows", log, models=False)
if cfg.pull_output:
pull_tree(cfg, host, f"{DATA}/Output", cfg.local_output_dir, log)
ensure_swarmui_running(cfg, host, log, restart=restart)
log("SwarmUI слушает 127.0.0.1:7801 — gpu-rent tunnel")
+1
View File
@@ -0,0 +1 @@
"""Scripts uploaded to the GPU VM."""
+165
View File
@@ -0,0 +1,165 @@
#!/usr/bin/env bash
# Idempotent gpu-rent first-boot on the VM. No Docker. Do not apt-upgrade the kernel.
set -euo pipefail
SWARM_USER="${SWARM_USER:-ubuntu}"
SWARM_ROOT="/opt/swarmui"
DATA_ROOT="/mnt/swarm_data"
MARKER_DATA="${DATA_ROOT}/.gpu-rent-ready"
MARKER_BOOT="${SWARM_ROOT}/.gpu-rent-bootstrapped"
SWARM_REPO="https://github.com/mcmonkeyprojects/SwarmUI.git"
log() { echo "[gpu-rent] $*"; }
if [[ "$(id -u)" -ne 0 ]]; then
echo "нужен root (sudo -n bash $0)" >&2
exit 1
fi
export DEBIAN_FRONTEND=noninteractive
pick_data_disk() {
local root_src root_pk name type
root_src="$(findmnt -n -o SOURCE / || true)"
root_pk="$(lsblk -no PKNAME "$root_src" 2>/dev/null | head -n1 || true)"
if [[ -z "$root_pk" && -n "$root_src" ]]; then
root_pk="$(lsblk -no NAME "$root_src" 2>/dev/null | head -n1 | sed 's/[0-9]*$//' || true)"
fi
while read -r name type; do
[[ "$type" == "disk" ]] || continue
[[ -n "$root_pk" && "$name" == "$root_pk" ]] && continue
if findmnt "/dev/${name}" >/dev/null 2>&1; then
continue
fi
echo "/dev/${name}"
return 0
done < <(lsblk -dn -o NAME,TYPE)
return 1
}
ensure_data_mount() {
mkdir -p "$DATA_ROOT"
if findmnt "$DATA_ROOT" >/dev/null 2>&1; then
log "data volume уже на ${DATA_ROOT}"
return 0
fi
local disk fstype uuid
disk="$(pick_data_disk)" || {
log "нет второго диска — проверь BDM data volume"
exit 1
}
fstype="$(blkid -s TYPE -o value "$disk" 2>/dev/null || true)"
if [[ -z "$fstype" ]]; then
if [[ -e "$MARKER_DATA" ]]; then
log "маркер есть, а FS на ${disk} нет — не mkfs, разбери вручную"
exit 1
fi
log "mkfs.ext4 ${disk} (пустой data volume)"
mkfs.ext4 -F -L swarm-data "$disk"
fi
mount "$disk" "$DATA_ROOT"
uuid="$(blkid -s UUID -o value "$disk")"
if [[ -n "$uuid" ]] && ! grep -q "UUID=${uuid}" /etc/fstab; then
echo "UUID=${uuid} ${DATA_ROOT} ext4 defaults,nofail 0 2" >> /etc/fstab
fi
log "смонтирован ${disk} -> ${DATA_ROOT}"
}
ensure_bind() {
local src="$1" dst="$2"
mkdir -p "$src" "$dst"
if ! findmnt "$dst" >/dev/null 2>&1; then
mount --bind "$src" "$dst"
fi
if ! grep -Fq " ${dst} " /etc/fstab; then
echo "${src} ${dst} none bind,nofail 0 0" >> /etc/fstab
fi
}
log "пакеты (без upgrade ядра)"
apt-get update -qq
apt-get install -y -qq git python3 python3-venv python3-pip python3-full curl ca-certificates
ensure_data_mount
mkdir -p \
"${DATA_ROOT}/Models" \
"${DATA_ROOT}/Output" \
"${DATA_ROOT}/Data" \
"${DATA_ROOT}/Data/Autocompletions" \
"${DATA_ROOT}/Data/Wildcards" \
"${DATA_ROOT}/dlbackend" \
"${DATA_ROOT}/Extensions" \
"${DATA_ROOT}/DLNodes" \
"${DATA_ROOT}/CustomWorkflows"
if [[ ! -d "${SWARM_ROOT}/.git" ]]; then
log "clone SwarmUI -> ${SWARM_ROOT}"
mkdir -p "$(dirname "$SWARM_ROOT")"
git clone --depth 1 "$SWARM_REPO" "$SWARM_ROOT"
else
log "SwarmUI уже в ${SWARM_ROOT}"
fi
if [[ ! -x /usr/share/dotnet/dotnet && ! -x "/home/${SWARM_USER}/.dotnet/dotnet" ]]; then
if [[ -x "${SWARM_ROOT}/launchtools/linux-dotnet-install.sh" ]]; then
log "ставим .NET SDK (скрипт SwarmUI)"
sudo -u "$SWARM_USER" bash "${SWARM_ROOT}/launchtools/linux-dotnet-install.sh" || true
fi
fi
mkdir -p \
"${SWARM_ROOT}/Models" \
"${SWARM_ROOT}/Output" \
"${SWARM_ROOT}/Data" \
"${SWARM_ROOT}/dlbackend" \
"${SWARM_ROOT}/src/Extensions" \
"${SWARM_ROOT}/src/BuiltinExtensions/ComfyUIBackend/DLNodes" \
"${SWARM_ROOT}/src/BuiltinExtensions/ComfyUIBackend/CustomWorkflows"
ensure_bind "${DATA_ROOT}/Models" "${SWARM_ROOT}/Models"
ensure_bind "${DATA_ROOT}/Output" "${SWARM_ROOT}/Output"
ensure_bind "${DATA_ROOT}/Data" "${SWARM_ROOT}/Data"
ensure_bind "${DATA_ROOT}/dlbackend" "${SWARM_ROOT}/dlbackend"
ensure_bind "${DATA_ROOT}/Extensions" "${SWARM_ROOT}/src/Extensions"
ensure_bind "${DATA_ROOT}/DLNodes" "${SWARM_ROOT}/src/BuiltinExtensions/ComfyUIBackend/DLNodes"
ensure_bind "${DATA_ROOT}/CustomWorkflows" "${SWARM_ROOT}/src/BuiltinExtensions/ComfyUIBackend/CustomWorkflows"
chown -R "${SWARM_USER}:${SWARM_USER}" "$DATA_ROOT" "$SWARM_ROOT"
cat >/etc/systemd/system/swarmui.service <<EOF
[Unit]
Description=SwarmUI (gpu-rent)
After=network-online.target local-fs.target
Wants=network-online.target
[Service]
Type=simple
User=${SWARM_USER}
Group=${SWARM_USER}
WorkingDirectory=${SWARM_ROOT}
Environment=HOME=/home/${SWARM_USER}
Environment=DOTNET_ROOT=/home/${SWARM_USER}/.dotnet
Environment=DOTNET_CLI_HOME=/home/${SWARM_USER}
Environment=PATH=/home/${SWARM_USER}/.dotnet:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=${SWARM_ROOT}/launch-linux.sh --launch_mode none --host 127.0.0.1 --port 7801
Restart=on-failure
RestartSec=8
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable swarmui
# Не стартуем UI здесь: сначала extensions / autocomplete / Civitai / push.
date -u +"%Y-%m-%dT%H:%M:%SZ" >"$MARKER_DATA"
date -u +"%Y-%m-%dT%H:%M:%SZ" >"$MARKER_BOOT"
chown "${SWARM_USER}:${SWARM_USER}" "$MARKER_DATA" "$MARKER_BOOT"
if command -v nvidia-smi >/dev/null 2>&1; then
nvidia-smi -L || true
fi
log "bootstrap ok (unit готов; swarmui старт после seed)"
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Download Civitai files on the VM. Stdlib only. Token in a 600 file."""
from __future__ import annotations
import hashlib
import json
import sys
import urllib.parse
import urllib.request
from pathlib import Path
TOKEN_PATH = Path("/tmp/gpu-rent-civitai.token")
JOBS_PATH = Path("/tmp/gpu-rent-civitai-jobs.json")
MARKER = Path("/mnt/swarm_data/.gpu-rent-models-seeded")
def sha256_path(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as fh:
while True:
block = fh.read(1024 * 1024)
if not block:
break
digest.update(block)
return digest.hexdigest()
def download(url: str, dest: Path, token: str) -> None:
dest.parent.mkdir(parents=True, exist_ok=True)
partial = dest.with_suffix(dest.suffix + ".partial")
class StripAuthRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
new = urllib.request.HTTPRedirectHandler.redirect_request(
self, req, fp, code, msg, headers, newurl
)
if new is None:
return None
host = (urllib.parse.urlparse(new.full_url).hostname or "").lower()
if host.endswith("civitai.com") or host.endswith("civitai.red") or host.endswith("civitai.green"):
return new
# presigned S3 / CDN: token must not leave civitai
return urllib.request.Request(new.full_url, headers={"User-Agent": "gpu-rent/0.1"})
opener = urllib.request.build_opener(StripAuthRedirect)
req = urllib.request.Request(
url, headers={"Authorization": f"Bearer {token}", "User-Agent": "gpu-rent/0.1"}
)
with opener.open(req, timeout=600) as response, partial.open("wb") as out:
while True:
chunk = response.read(1024 * 1024)
if not chunk:
break
out.write(chunk)
partial.replace(dest)
def main() -> int:
if not TOKEN_PATH.is_file():
print("нет токена", file=sys.stderr)
return 1
token = TOKEN_PATH.read_text(encoding="utf-8").strip()
jobs = json.loads(JOBS_PATH.read_text(encoding="utf-8"))
failed = 0
for job in jobs:
dest = Path(job["dest"])
expect = (job.get("sha256") or "").lower()
if dest.is_file() and expect and sha256_path(dest).lower() == expect:
print(f"skip {dest}")
continue
try:
print(f"download {dest.name}")
download(job["url"], dest, token)
if expect:
got = sha256_path(dest).lower()
if got != expect:
dest.unlink(missing_ok=True)
raise RuntimeError(f"sha256 {got} != {expect}")
for extra_name, extra_text in (job.get("sidecars") or {}).items():
extra = dest.parent / extra_name
extra.write_text(extra_text, encoding="utf-8")
except Exception as exc:
failed += 1
print(f"FAIL {dest}: {exc}", file=sys.stderr)
TOKEN_PATH.unlink(missing_ok=True)
if failed:
return 1
MARKER.parent.mkdir(parents=True, exist_ok=True)
MARKER.write_text("ok\n", encoding="utf-8")
print("civitai seed ok")
return 0
if __name__ == "__main__":
sys.exit(main())
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env python3
"""Clone git extensions on the VM. Stdlib only. Token file optional."""
from __future__ import annotations
import json
import os
import subprocess
import sys
from pathlib import Path
from urllib.parse import urlsplit, urlunsplit
TOKEN_PATH = Path("/tmp/gpu-rent-git.token")
JOBS_PATH = Path("/tmp/gpu-rent-ext.json")
MARKER = Path("/mnt/swarm_data/.gpu-rent-extensions-seeded")
def strip_auth(url: str) -> str:
parts = urlsplit(url)
host = parts.hostname or ""
if parts.port:
host = f"{host}:{parts.port}"
return urlunsplit((parts.scheme, host, parts.path, parts.query, parts.fragment))
def with_token(url: str, token: str) -> str:
if not token:
return url
if url.startswith("https://github.com/"):
return "https://x-access-token:" + token + "@github.com/" + url[len("https://github.com/") :]
if url.startswith("https://gitlab.com/"):
return "https://oauth2:" + token + "@gitlab.com/" + url[len("https://gitlab.com/") :]
return url
def run(argv: list[str], cwd: str | None = None) -> None:
subprocess.check_call(argv, cwd=cwd)
def is_sha(ref: str) -> bool:
ref = ref.strip()
return len(ref) == 40 and all(c in "0123456789abcdefABCDEF" for c in ref)
def clone_one(job: dict, token: str) -> None:
dest = Path(job["dest"])
url = job["url"].strip()
ref = (job.get("ref") or "main").strip()
dest.parent.mkdir(parents=True, exist_ok=True)
authed = with_token(url, token)
if dest.is_dir() and (dest / ".git").is_dir():
origin = subprocess.check_output(["git", "-C", str(dest), "remote", "get-url", "origin"], text=True).strip()
if strip_auth(origin) != strip_auth(url):
print(f"FAIL origin mismatch {dest}: {origin} != {url}", file=sys.stderr)
raise SystemExit(2)
run(["git", "-C", str(dest), "fetch", "--recurse-submodules", "origin"])
run(["git", "-C", str(dest), "checkout", ref])
print(f"updated {dest}")
return
if dest.exists():
print(f"FAIL {dest} exists but is not a git repo", file=sys.stderr)
raise SystemExit(2)
if is_sha(ref):
run(["git", "clone", "--recurse-submodules", authed, str(dest)])
run(["git", "-C", str(dest), "fetch", "origin", ref])
run(["git", "-C", str(dest), "checkout", ref])
else:
try:
run(["git", "clone", "--recurse-submodules", "--depth", "1", "--branch", ref, authed, str(dest)])
except subprocess.CalledProcessError:
run(["git", "clone", "--recurse-submodules", authed, str(dest)])
run(["git", "-C", str(dest), "checkout", ref])
print(f"cloned {dest}")
def main() -> int:
token = TOKEN_PATH.read_text(encoding="utf-8").strip() if TOKEN_PATH.is_file() else ""
jobs = json.loads(JOBS_PATH.read_text(encoding="utf-8"))
failed = 0
for job in jobs:
try:
clone_one(job, token)
except Exception as exc:
failed += 1
print(f"FAIL {job.get('dest')}: {exc}", file=sys.stderr)
if TOKEN_PATH.is_file():
TOKEN_PATH.unlink()
if failed:
return 1
MARKER.parent.mkdir(parents=True, exist_ok=True)
MARKER.write_text("ok\n", encoding="utf-8")
print("extensions ok")
return 0
if __name__ == "__main__":
sys.exit(main())
+282
View File
@@ -0,0 +1,282 @@
"""up / stop / destroy: GPU lifetime; SwarmUI bootstrap after SSH."""
from __future__ import annotations
from collections.abc import Callable
from gpu_rent.cloud import (
create_gpu_server,
delete_floating_ip,
delete_server,
ensure_boot_volume,
ensure_data_volume,
ensure_floating_ip,
ensure_keypair,
ensure_network,
ensure_security_group,
guess_operator_cidr,
pick_existing_server,
server_status,
unshelve,
)
from gpu_rent.bootstrap import run_bootstrap
from gpu_rent.provision import provision_vm
from gpu_rent.config import Config
from gpu_rent.errors import CloudError, GpuRentError
from gpu_rent.inventory import (
gpu_quota_from_compute,
pick_boot_image,
pick_volume_type,
resolve_flavor,
)
from gpu_rent.lock import SessionLock
from gpu_rent.os_client import (
KEYPAIR_NAME,
SG_NAME,
compute_quotas,
connect,
iter_flavors,
iter_images,
iter_volume_types,
)
from gpu_rent.ssh_keys import ensure_ed25519
from gpu_rent.ssh_ops import wait_ssh
from gpu_rent.state import SessionState, load_state, save_state, utc_now
Log = Callable[[str], None]
def _log_default(msg: str) -> None:
print(msg)
def _require_gpu_quota(conn) -> None:
quota = compute_quotas(conn)
limit = gpu_quota_from_compute(quota)
if limit is not None and limit <= 0:
raise CloudError(
"квота GPU = 0. Напиши в поддержку Selectel (текст в docs/setup.md). "
"CLI не создаст сервер."
)
def _bind_access(conn, server, state: SessionState, cfg: Config, log: Log) -> SessionState:
ip, fip_id = ensure_floating_ip(
conn, server, state.floating_ip_id, state.floating_ip, log
)
state.floating_ip = ip
if fip_id:
state.floating_ip_id = fip_id
save_state(state)
wait_ssh(cfg, ip)
log(f"SSH {cfg.ssh_user}@{ip}")
run_bootstrap(cfg, ip, log)
provision_vm(cfg, ip, log)
state.bootstrapped = True
save_state(state)
log("gpu-rent tunnel — UI на localhost:17801")
log("gpu-rent stop — удалить compute, диски оставить")
return state
def adopt_server(cfg: Config, log: Log = _log_default) -> SessionState:
conn = connect(cfg)
server = pick_existing_server(conn)
if not server:
raise CloudError("нечего adopt: нет сервера с тегом/именем gpu-rent")
state = load_state()
state.server_id = server.id
state.server_name = getattr(server, "name", None)
state.flavor_id = getattr(server, "flavor_id", None) or (
(server.flavor or {}).get("id") if isinstance(getattr(server, "flavor", None), dict) else None
)
state.phase = "ready_cloud"
state.availability_zone = cfg.gpu_rent_az
save_state(state)
log(f"подхватили {server.id} статус {server_status(server)}")
if server_status(server) == "ACTIVE":
_bind_access(conn, server, state, cfg, log)
return state
def cmd_up(
cfg: Config,
*,
no_spot: bool = False,
flavor: str | None = None,
yes: bool = False,
adopt: bool = False,
confirm: Callable[[str], bool] | None = None,
log: Log = _log_default,
) -> SessionState:
with SessionLock():
if adopt:
return adopt_server(cfg, log=log)
conn = connect(cfg)
_require_gpu_quota(conn)
state = load_state()
existing = pick_existing_server(conn)
if existing:
status = server_status(existing)
state.server_id = existing.id
state.server_name = getattr(existing, "name", None)
if status == "ACTIVE":
log("сервер уже ACTIVE — второй GPU не создаём")
state.phase = "ready_cloud"
save_state(state)
_bind_access(conn, existing, state, cfg, log)
return state
if status in {"EXPIRED", "SHELVED", "SHELVED_OFFLOADED"}:
existing = unshelve(conn, existing, log)
state.server_id = existing.id
state.phase = "ready_cloud"
state.unshelved_at = utc_now()
save_state(state)
_bind_access(conn, existing, state, cfg, log)
return state
raise CloudError(f"сервер gpu-rent в статусе {status} — разбери в панели")
flavors = list(iter_flavors(conn))
try:
picked = resolve_flavor(
flavors,
cfg.flavor_preference,
explicit=flavor,
default_id=cfg.default_flavor_id or None,
fallback=cfg.flavor_fallback,
)
except ValueError as exc:
raise CloudError(str(exc)) from exc
image = pick_boot_image(list(iter_images(conn)))
if not image:
raise CloudError("нет GPU-образа Ubuntu Driver 580/535 в Glance")
vtype = pick_volume_type(list(iter_volume_types(conn)), cfg.gpu_rent_az)
spot = cfg.default_spot and not no_spot
prompt = (
f"Создать {'preemptible ' if spot else ''}GPU {picked.name} "
f"в {cfg.gpu_rent_az}, образ {getattr(image, 'name', image.id)}, "
f"data {cfg.data_volume_size_gb} GB. Диск тарифицируется всегда. Продолжить?"
)
if not yes:
ok = confirm(prompt) if confirm else False
if not ok:
raise GpuRentError("отменено")
private_path, pub = ensure_ed25519(cfg.ssh_private_key_path)
del private_path
public_key = pub.read_text(encoding="utf-8")
ensure_keypair(conn, public_key, log)
net, _subnet = ensure_network(conn, log)
cidr = guess_operator_cidr()
if cidr == "0.0.0.0/0":
log("не удалось узнать твой IP — SG откроет SSH с 0.0.0.0/0")
sg = ensure_security_group(conn, cidr, log)
boot = ensure_boot_volume(
conn,
az=cfg.gpu_rent_az,
volume_type=vtype,
image_id=image.id,
snapshot_name=cfg.boot_snapshot_name,
existing_id=state.boot_volume_id or cfg.boot_volume_id,
log=log,
)
data = ensure_data_volume(
conn,
az=cfg.gpu_rent_az,
volume_type=vtype,
size_gb=cfg.data_volume_size_gb,
existing_id=state.data_volume_id or cfg.data_volume_id,
log=log,
)
state.phase = "provisioning"
state.flavor_id = picked.id
state.flavor_name = picked.name
state.boot_volume_id = boot.id
state.data_volume_id = data.id
state.image_id = image.id
state.network_id = net.id
state.security_group_id = sg.id
state.availability_zone = cfg.gpu_rent_az
state.spot = spot
state.keypair_name = KEYPAIR_NAME
save_state(state)
server = create_gpu_server(
conn,
flavor_id=picked.id,
net_id=net.id,
sg_name=SG_NAME,
boot_volume_id=boot.id,
data_volume_id=data.id,
az=cfg.gpu_rent_az,
spot=spot,
log=log,
)
state.server_id = server.id
state.server_name = getattr(server, "name", None)
state.created_at = utc_now()
state.unshelved_at = None
state.phase = "ready_cloud"
save_state(state)
_bind_access(conn, server, state, cfg, log)
return state
def cmd_stop(
cfg: Config,
*,
destroy_disks: bool = False,
no_pull: bool = False,
log: Log = _log_default,
) -> SessionState:
with SessionLock():
conn = connect(cfg)
state = load_state()
if cfg.pull_output and not no_pull and state.floating_ip:
try:
from gpu_rent.sync_files import pull_tree
pull_tree(cfg, state.floating_ip, "/mnt/swarm_data/Output", cfg.local_output_dir, log)
except Exception as exc:
log(f"pull Output не удался: {exc}")
server = None
if state.server_id:
try:
server = conn.compute.get_server(state.server_id)
except Exception:
server = None
if server is None:
server = pick_existing_server(conn)
if server:
delete_server(conn, server, log)
else:
log("compute уже нет")
if not cfg.keep_floating_ip:
delete_floating_ip(conn, state.floating_ip_id, state.floating_ip, log)
state.floating_ip = None
state.floating_ip_id = None
if destroy_disks:
for vid in (state.data_volume_id, state.boot_volume_id):
if not vid:
continue
try:
conn.block_storage.delete_volume(vid, ignore_missing=True)
log(f"удалён том {vid}")
except Exception as exc:
log(f"том {vid} не удалился: {exc}")
state.boot_volume_id = None
state.data_volume_id = None
state.server_id = None
state.server_name = None
state.phase = "idle"
save_state(state)
log("фаза idle" + ("" if destroy_disks else " (диски на месте)"))
return state
+274
View File
@@ -0,0 +1,274 @@
"""Wait for SSH and run remote commands."""
from __future__ import annotations
from collections.abc import Callable
from pathlib import Path
import shlex
import socket
import subprocess
import time
import paramiko
from gpu_rent.config import Config
from gpu_rent.errors import CloudError
def wait_tcp(host: str, port: int, timeout: float = 300.0) -> None:
deadline = time.time() + timeout
last = None
while time.time() < deadline:
try:
with socket.create_connection((host, port), timeout=8):
return
except OSError as exc:
last = exc
time.sleep(4)
raise CloudError(f"TCP {host}:{port} не открылся за {int(timeout)} с ({last})")
def wait_ssh(cfg: Config, host: str, timeout: float = 420.0) -> None:
wait_tcp(host, 22, timeout=min(timeout, 240))
deadline = time.time() + timeout
key = str(cfg.ssh_private_key_path)
last = None
while time.time() < deadline:
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
client.connect(
hostname=host,
username=cfg.ssh_user,
key_filename=key,
timeout=12,
banner_timeout=12,
auth_timeout=12,
)
client.close()
return
except Exception as exc:
last = exc
try:
client.close()
except Exception:
pass
time.sleep(5)
raise CloudError(f"SSH {cfg.ssh_user}@{host} не принял ключ ({last})")
def ssh_argv(cfg: Config, host: str, remote: list[str] | None = None) -> list[str]:
cmd = [
"ssh",
"-i",
str(cfg.ssh_private_key_path),
"-o",
"StrictHostKeyChecking=accept-new",
"-o",
"IdentitiesOnly=yes",
f"{cfg.ssh_user}@{host}",
]
if remote:
cmd.append(" ".join(remote))
return cmd
def interactive_ssh(cfg: Config, host: str) -> int:
argv = ssh_argv(cfg, host)
try:
return subprocess.call(argv)
except FileNotFoundError:
raise CloudError(
"Нет клиента ssh в PATH. Windows: установи OpenSSH Client "
f"или зайди так: ssh -i {cfg.ssh_private_key_path} {cfg.ssh_user}@{host}"
) from None
def run_ssh(cfg: Config, host: str, command: str, timeout: int = 60, check: bool = True) -> str:
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
client.connect(
hostname=host,
username=cfg.ssh_user,
key_filename=str(cfg.ssh_private_key_path),
timeout=15,
)
_stdin, stdout, stderr = client.exec_command(command, timeout=timeout)
out = stdout.read().decode("utf-8", errors="replace")
err = stderr.read().decode("utf-8", errors="replace")
code = stdout.channel.recv_exit_status()
if check and code != 0:
raise CloudError(f"SSH `{command}` exit {code}: {err or out}")
return out
finally:
client.close()
def run_script_sudo(
cfg: Config,
host: str,
script: str,
*,
remote_path: str,
timeout: int = 1800,
env: dict[str, str] | None = None,
log: Callable[[str], None] | None = None,
) -> str:
"""Upload a script and run it with passwordless sudo, streaming stdout."""
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
chunks: list[str] = []
try:
client.connect(
hostname=host,
username=cfg.ssh_user,
key_filename=str(cfg.ssh_private_key_path),
timeout=20,
)
sftp = client.open_sftp()
with sftp.file(remote_path, "w") as fh:
fh.write(script)
sftp.chmod(remote_path, 0o755)
sftp.close()
env_s = " ".join(f"{key}={value}" for key, value in (env or {}).items())
prefix = f"sudo -n env {env_s} " if env_s else "sudo -n "
command = f"{prefix}bash {remote_path}"
_stdin, stdout, stderr = client.exec_command(command, timeout=timeout, get_pty=True)
while True:
line = stdout.readline()
if not line:
break
chunks.append(line)
if log:
log(line.rstrip("\n\r"))
code = stdout.channel.recv_exit_status()
err = stderr.read().decode("utf-8", errors="replace") if not stdout.channel.closed else ""
out = "".join(chunks)
if code != 0:
raise CloudError(f"remote script exit {code}: {err or out[-2000:]}")
return out
finally:
client.close()
def run_python(
cfg: Config,
host: str,
script: str,
*,
remote_path: str,
timeout: int = 1800,
log: Callable[[str], None] | None = None,
) -> str:
"""Upload a Python script and run it as SSH user (not root)."""
client = _connect(cfg, host)
chunks: list[str] = []
try:
sftp = client.open_sftp()
with sftp.file(remote_path, "w") as fh:
fh.write(script)
sftp.chmod(remote_path, 0o755)
sftp.close()
command = f"python3 {shlex.quote(remote_path)}"
_stdin, stdout, stderr = client.exec_command(command, timeout=timeout, get_pty=True)
while True:
line = stdout.readline()
if not line:
break
chunks.append(line)
if log:
log(line.rstrip("\n\r"))
code = stdout.channel.recv_exit_status()
err = stderr.read().decode("utf-8", errors="replace") if not stdout.channel.closed else ""
out = "".join(chunks)
if code != 0:
raise CloudError(f"remote python exit {code}: {err or out[-2000:]}")
return out
finally:
client.close()
def _connect(cfg: Config, host: str) -> paramiko.SSHClient:
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(
hostname=host,
username=cfg.ssh_user,
key_filename=str(cfg.ssh_private_key_path),
timeout=20,
)
return client
def put_text(cfg: Config, host: str, remote_path: str, text: str, mode: int = 0o644) -> None:
client = _connect(cfg, host)
try:
sftp = client.open_sftp()
_sftp_mkdirs(sftp, str(Path(remote_path).parent).replace("\\", "/"))
with sftp.file(remote_path, "w") as fh:
fh.write(text)
sftp.chmod(remote_path, mode)
sftp.close()
finally:
client.close()
def put_file(cfg: Config, host: str, local: Path, remote_path: str) -> None:
client = _connect(cfg, host)
try:
sftp = client.open_sftp()
_sftp_mkdirs(sftp, str(Path(remote_path).parent).replace("\\", "/"))
sftp.put(str(local), remote_path)
sftp.close()
finally:
client.close()
def get_file(cfg: Config, host: str, remote_path: str, local: Path) -> None:
local.parent.mkdir(parents=True, exist_ok=True)
client = _connect(cfg, host)
try:
sftp = client.open_sftp()
sftp.get(remote_path, str(local))
sftp.close()
finally:
client.close()
def remote_exists(cfg: Config, host: str, remote_path: str) -> bool:
client = _connect(cfg, host)
try:
sftp = client.open_sftp()
try:
sftp.stat(remote_path)
return True
except FileNotFoundError:
return False
finally:
sftp.close()
finally:
client.close()
def remote_sha256(cfg: Config, host: str, remote_path: str) -> str | None:
out = run_ssh(
cfg,
host,
f"sha256sum {shlex.quote(remote_path)} 2>/dev/null | awk '{{print $1}}'",
check=False,
).strip()
return out or None
def _sftp_mkdirs(sftp, remote_dir: str) -> None:
if not remote_dir or remote_dir == "/":
return
parts = [p for p in remote_dir.split("/") if p]
cur = ""
for part in parts:
cur += "/" + part
try:
sftp.stat(cur)
except FileNotFoundError:
sftp.mkdir(cur)
+7
View File
@@ -28,6 +28,13 @@ class SessionState:
created_at: str | None = None created_at: str | None = None
unshelved_at: str | None = None unshelved_at: str | None = None
notes: dict[str, Any] = field(default_factory=dict) notes: dict[str, Any] = field(default_factory=dict)
floating_ip_id: str | None = None
network_id: str | None = None
security_group_id: str | None = None
image_id: str | None = None
availability_zone: str | None = None
spot: bool = True
bootstrapped: bool = False
def to_dict(self) -> dict[str, Any]: def to_dict(self) -> dict[str, Any]:
return asdict(self) return asdict(self)
+76
View File
@@ -0,0 +1,76 @@
"""SFTP push/pull of local app folders. Never delete remote extras."""
from __future__ import annotations
from collections.abc import Callable
from pathlib import Path
from gpu_rent.config import Config
from gpu_rent.payload import has_payload, iter_payload_files, model_push_set, sha256_file
from gpu_rent.ssh_ops import get_file, put_file, remote_sha256, run_ssh
Log = Callable[[str], None]
def push_tree(
cfg: Config,
host: str,
local_root: Path,
remote_root: str,
log: Log,
*,
models: bool,
) -> int:
if not has_payload(local_root):
log(f"push {local_root.name}: пусто — skip")
return 0
files = model_push_set(local_root) if models else iter_payload_files(local_root)
sent = 0
for path in files:
rel = path.relative_to(local_root).as_posix()
remote = f"{remote_root.rstrip('/')}/{rel}"
local_hash = sha256_file(path)
remote_hash = remote_sha256(cfg, host, remote)
if remote_hash and remote_hash.lower() == local_hash.lower():
continue
if models and not _is_weight_name(path.name):
# sidecar: warn if we somehow got here without weight — still send
pass
log(f"push {rel}")
put_file(cfg, host, path, remote)
sent += 1
if sent == 0:
log(f"push {local_root.name}: всё уже на VM")
else:
log(f"push {local_root.name}: {sent} файл(ов)")
return sent
def _is_weight_name(name: str) -> bool:
lower = name.lower()
return lower.endswith((".safetensors", ".ckpt", ".pt", ".pth", ".bin", ".gguf", ".sft", ".onnx"))
def pull_tree(cfg: Config, host: str, remote_root: str, local_root: Path, log: Log) -> int:
listing = run_ssh(
cfg,
host,
f"find {remote_root} -type f 2>/dev/null | sed 's|^{remote_root}/||'",
check=False,
timeout=120,
)
names = [line.strip() for line in listing.splitlines() if line.strip()]
pulled = 0
for rel in names:
if rel.endswith("/.gitkeep") or rel.rsplit("/", 1)[-1] in {".gitkeep", "README.md"}:
continue
remote = f"{remote_root.rstrip('/')}/{rel}"
local = local_root / rel
remote_hash = remote_sha256(cfg, host, remote)
if local.is_file() and remote_hash and sha256_file(local).lower() == remote_hash.lower():
continue
log(f"pull {rel}")
get_file(cfg, host, remote, local)
pulled += 1
log(f"pull Output: {pulled} файл(ов)" if pulled else "pull Output: нечего забирать")
return pulled
+62
View File
@@ -0,0 +1,62 @@
"""SSH local forward. Closing the tunnel does not stop the GPU."""
from __future__ import annotations
import time
import webbrowser
from collections.abc import Callable
from gpu_rent.config import Config
from gpu_rent.errors import CloudError
Log = Callable[[str], None]
def run_tunnel(
cfg: Config,
host: str,
*,
open_browser: bool = False,
log: Log = print,
wait: Callable[[], None] | None = None,
) -> None:
try:
from sshtunnel import SSHTunnelForwarder
except ImportError as exc:
raise CloudError("Нет sshtunnel. Переустанови пакет: pip install -e .") from exc
local_port = cfg.swarmui_local_port
log(f"туннель 127.0.0.1:{local_port} -> {host}:7801")
log("Ctrl+C закрывает туннель, GPU оставляет. Стоп GPU: gpu-rent stop")
server = SSHTunnelForwarder(
(host, 22),
ssh_username=cfg.ssh_user,
ssh_pkey=str(cfg.ssh_private_key_path),
remote_bind_address=("127.0.0.1", 7801),
local_bind_address=("127.0.0.1", local_port),
set_keepalive=30,
)
try:
server.start()
except Exception as exc:
raise CloudError(
f"не открыть туннель на {local_port}: {exc}. Порт занят локальным SwarmUI? "
"17801 должен быть свободен."
) from exc
url = f"http://127.0.0.1:{local_port}"
log(f"UI {url}")
log(f"API {url}/API/")
log(f"MCP {url}/mcp")
if open_browser:
webbrowser.open(url)
try:
if wait is not None:
wait()
return
while server.is_active:
time.sleep(1)
except KeyboardInterrupt:
log("туннель закрыт. GPU жив.")
finally:
server.stop()
+16
View File
@@ -0,0 +1,16 @@
from gpu_rent.bootstrap import bootstrap_script
def test_bootstrap_script_is_native_swarmui():
script = bootstrap_script()
assert "docker.io" not in script
assert "docker run" not in script
assert "nvidia-container" not in script
assert "/opt/swarmui" in script
assert "/mnt/swarm_data" in script
assert "launch-linux.sh --launch_mode none --host 127.0.0.1 --port 7801" in script
assert "systemctl enable swarmui" in script
assert "mkfs.ext4" in script
assert "apt-get upgrade" not in script
assert ".gpu-rent-ready" in script
assert "src/BuiltinExtensions/ComfyUIBackend/DLNodes" in script
+43
View File
@@ -0,0 +1,43 @@
from gpu_rent.cloud import ensure_floating_ip, server_floating_ip, server_status
from gpu_rent.lock import SessionLock
from gpu_rent.paths import lock_path
class Obj:
def __init__(self, **kwargs):
self.__dict__.update(kwargs)
def test_server_floating_ip():
server = Obj(
status="ACTIVE",
addresses={"gpu-rent": [{"addr": "192.168.77.10", "OS-EXT-IPS:type": "fixed"}, {"addr": "203.0.113.9", "OS-EXT-IPS:type": "floating"}]},
)
assert server_status(server) == "ACTIVE"
assert server_floating_ip(server) == "203.0.113.9"
def test_lock_roundtrip():
with SessionLock():
assert lock_path().is_file()
assert not lock_path().is_file()
def test_ensure_floating_ip_reuses_existing():
class Net:
def ports(self, device_id=None):
return [Obj(id="p1")]
def update_ip(self, existing_id, port_id=None):
assert existing_id == "old-fip"
assert port_id == "p1"
return Obj(floating_ip_address="9.9.9.9")
def create_ip(self, floating_network_id=None):
raise AssertionError("must reuse FIP")
conn = Obj(network=Net())
server = Obj(id="s1", addresses={})
ip, fip_id = ensure_floating_ip(conn, server, "old-fip", "9.9.9.9", lambda m: None)
assert ip == "9.9.9.9"
assert fip_id == "old-fip"
+33
View File
@@ -4,6 +4,7 @@ from gpu_rent.inventory import (
match_label, match_label,
pick_boot_image, pick_boot_image,
rank_flavors, rank_flavors,
resolve_flavor,
) )
@@ -41,6 +42,38 @@ def test_gpu_quota_from_compute():
assert gpu_quota_from_compute({"GPU_limit": 2}) == 2 assert gpu_quota_from_compute({"GPU_limit": 2}) == 2
def test_resolve_flavor_explicit_and_preference():
flavors = [
FakeFlavor("d", "RTX 4090 24GB", disabled=True),
FakeFlavor("a5", "RTX A5000 24GB"),
FakeFlavor("ok", "GPU 1x RTX 4090 24GB"),
]
picked = resolve_flavor(flavors, ("4090-24", "a5000"))
assert picked.id == "ok"
explicit = resolve_flavor(flavors, ("a5000",), explicit="a5")
assert explicit.id == "a5"
def test_resolve_flavor_prefers_list_over_default_id():
flavors = [
FakeFlavor("a5", "RTX A5000 24GB"),
FakeFlavor("ok", "GPU 1x RTX 4090 24GB"),
]
picked = resolve_flavor(flavors, ("4090-24", "a5000"), default_id="a5", fallback=True)
assert picked.id == "ok"
def test_resolve_flavor_no_fallback_requires_default():
flavors = [FakeFlavor("ok", "GPU 1x RTX 4090 24GB")]
try:
resolve_flavor(flavors, ("4090-24",), fallback=False)
raise AssertionError("expected ValueError")
except ValueError:
pass
picked = resolve_flavor(flavors, ("a5000",), default_id="ok", fallback=False)
assert picked.id == "ok"
def test_pick_boot_image_prefers_24_580_without_docker(): def test_pick_boot_image_prefers_24_580_without_docker():
class Img: class Img:
def __init__(self, name): def __init__(self, name):
+19
View File
@@ -0,0 +1,19 @@
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
def test_launcher_files_exist():
for name in ("gpu-rent.bat", "gpu-rent.ps1", "gpu-rent.sh"):
path = ROOT / name
assert path.is_file(), name
text = path.read_text(encoding="utf-8")
assert ".venv" in text or "gpu-rent.ps1" in text
assert "gpu_rent" in text or "gpu-rent.ps1" in text
def test_launchers_require_python_311():
ps1 = (ROOT / "gpu-rent.ps1").read_text(encoding="utf-8")
sh = (ROOT / "gpu-rent.sh").read_text(encoding="utf-8")
assert "3, 11" in ps1 or "3.11" in ps1
assert "3, 11" in sh
+132
View File
@@ -0,0 +1,132 @@
import pytest
from gpu_rent.config import load_config
from gpu_rent.errors import CloudError, GpuRentError
from gpu_rent.lock import SessionLock
from gpu_rent.paths import lock_path
from gpu_rent.session import cmd_stop, cmd_up
from gpu_rent.state import SessionState, save_state
class Server:
def __init__(self, status="ACTIVE", server_id="s1"):
self.id = server_id
self.name = "gpu-rent"
self.status = status
self.flavor = {"id": "f1"}
self.addresses = {
"gpu-rent": [{"addr": "203.0.113.9", "OS-EXT-IPS:type": "floating"}]
}
def _cfg(monkeypatch):
monkeypatch.setenv("OS_AUTH_URL", "https://example.invalid/identity/v3")
monkeypatch.setenv("OS_USER_DOMAIN_NAME", "999")
monkeypatch.setenv("OS_USERNAME", "svc")
monkeypatch.setenv("OS_PASSWORD", "secret")
monkeypatch.setenv("OS_PROJECT_ID", "proj")
monkeypatch.setenv("OS_REGION_NAME", "ru-7")
monkeypatch.setenv("GPU_RENT_AZ", "ru-7a")
return load_config(require_auth=True)
def test_cmd_up_refuses_zero_gpu_quota(monkeypatch):
monkeypatch.setattr("gpu_rent.session.connect", lambda cfg: object())
monkeypatch.setattr("gpu_rent.session.compute_quotas", lambda conn: {"gpu": 0})
with pytest.raises(CloudError, match="квота GPU"):
cmd_up(_cfg(monkeypatch), yes=True)
def test_cmd_up_does_not_create_second_gpu(monkeypatch):
created = []
monkeypatch.setattr("gpu_rent.session.connect", lambda cfg: object())
monkeypatch.setattr("gpu_rent.session.compute_quotas", lambda conn: {"gpu": 1})
monkeypatch.setattr("gpu_rent.session.pick_existing_server", lambda conn: Server())
monkeypatch.setattr(
"gpu_rent.session.ensure_floating_ip",
lambda conn, server, existing_id, existing_addr, log: ("203.0.113.9", "fip1"),
)
monkeypatch.setattr("gpu_rent.session.wait_ssh", lambda cfg, host, timeout=420.0: None)
monkeypatch.setattr("gpu_rent.session.run_bootstrap", lambda cfg, host, log: None)
monkeypatch.setattr("gpu_rent.session.provision_vm", lambda cfg, host, log: None)
monkeypatch.setattr(
"gpu_rent.session.create_gpu_server",
lambda *a, **k: created.append("created") or Server(),
)
state = cmd_up(_cfg(monkeypatch), yes=True)
assert created == []
assert state.server_id == "s1"
assert state.phase == "ready_cloud"
assert state.floating_ip == "203.0.113.9"
assert state.bootstrapped is True
def test_cmd_up_unshelves_expired(monkeypatch):
unshelved = []
created = []
monkeypatch.setattr("gpu_rent.session.connect", lambda cfg: object())
monkeypatch.setattr("gpu_rent.session.compute_quotas", lambda conn: {"gpu": 1})
monkeypatch.setattr(
"gpu_rent.session.pick_existing_server", lambda conn: Server(status="EXPIRED")
)
monkeypatch.setattr(
"gpu_rent.session.unshelve",
lambda conn, server, log: unshelved.append(server.id) or Server(status="ACTIVE"),
)
monkeypatch.setattr(
"gpu_rent.session.ensure_floating_ip",
lambda conn, server, existing_id, existing_addr, log: ("203.0.113.9", "fip1"),
)
monkeypatch.setattr("gpu_rent.session.wait_ssh", lambda cfg, host, timeout=420.0: None)
monkeypatch.setattr("gpu_rent.session.run_bootstrap", lambda cfg, host, log: None)
monkeypatch.setattr("gpu_rent.session.provision_vm", lambda cfg, host, log: None)
monkeypatch.setattr(
"gpu_rent.session.create_gpu_server",
lambda *a, **k: created.append("created"),
)
state = cmd_up(_cfg(monkeypatch), yes=True)
assert unshelved == ["s1"]
assert created == []
assert state.unshelved_at
assert state.phase == "ready_cloud"
assert state.bootstrapped is True
def test_cmd_stop_deletes_compute_keeps_disks(monkeypatch):
deleted = []
monkeypatch.setattr("gpu_rent.session.connect", lambda cfg: object())
monkeypatch.setattr("gpu_rent.session.pick_existing_server", lambda conn: Server())
monkeypatch.setattr(
"gpu_rent.session.delete_server",
lambda conn, server, log: deleted.append(server.id),
)
monkeypatch.setattr(
"gpu_rent.session.delete_floating_ip",
lambda conn, fip_id, address, log: deleted.append("fip"),
)
save_state(
SessionState(server_id="s1", boot_volume_id="b1", data_volume_id="d1", floating_ip="1.1.1.1")
)
class Conn:
class compute:
@staticmethod
def get_server(sid):
return Server(server_id=sid)
monkeypatch.setattr("gpu_rent.session.connect", lambda cfg: Conn())
state = cmd_stop(_cfg(monkeypatch))
assert "s1" in deleted
assert state.phase == "idle"
assert state.server_id is None
assert state.boot_volume_id == "b1"
assert state.data_volume_id == "d1"
def test_lock_busy(monkeypatch):
lock_path().parent.mkdir(parents=True, exist_ok=True)
lock_path().write_text("1", encoding="utf-8")
monkeypatch.setattr("gpu_rent.lock._pid_alive", lambda pid: True)
monkeypatch.setattr("gpu_rent.lock.os.getpid", lambda: 99)
with pytest.raises(GpuRentError, match="уже работает"):
SessionLock().__enter__()