feat(api): library sources, scanner and catalog (M2)
ci / api / api (push) Successful in 26s
ci / mobile / mobile (Metro-бандл Android) (push) Failing after 22s
ci / contract / contract (сгенерированный код совпадает с openapi.yaml) (push) Successful in 37s
ci / ts / ts (packages + web) (push) Successful in 51s
ci / docker / docker (образы собираются — их сервер собирает сам при обновлении) (push) Successful in 53s

- schema for sources, path patterns, media files, artists, albums,
  tracks, genres (migration 00003)
- path patterns (RE2 with named groups, skip), merge with tags by
  source priority, fallbacks from the file name and folder, name
  normalization and multi-artist split
- scanner: walk, skip unchanged files by size and mtime, ffprobe in 8
  workers with a single catalog writer, renames by content hash keep
  the track, duplicates fold into one track with the best file as
  primary, missing files are kept 30 days
- storage-agent mounts NFS with validated server, export and an option
  whitelist, no shell; test mount counts audio files
- admin API: sources CRUD, connection test, patterns with preview,
  scan, library status, file errors, host folder picker; scheduled
  scans and health checks; mounts are restored on start
- catalog API: artist, artist albums, album with tracks, track
- api image is alpine with ffmpeg for ffprobe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mrleo1nidandClaude Opus 5.5 committed 2026-10-20 06:48:37 +03:00
1 parent ab092def54
commit c583991d14
24 files changed
+7465 -82

No files matched your search

+766
View File
@@ -12,6 +12,8 @@ tags:
- name: auth
- name: me
- name: admin
- name: library
- name: catalog
security:
- bearerAuth: []
paths:
@@ -369,8 +371,316 @@ paths:
$ref: "#/components/schemas/AuditPage"
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/sources:
get:
operationId: listSources
tags: [library]
summary: Хранилища (admin)
responses:
"200":
description: Список хранилищ
content:
application/json:
schema:
$ref: "#/components/schemas/SourceList"
default:
$ref: "#/components/responses/Problem"
post:
operationId: createSource
tags: [library]
summary: Подключить хранилище (admin)
description: |
`local` — подпапка в каталоге LIBRARY_HOST сервера; `nfs` — монтирует storage-agent.
Хранилище сохраняется, даже если подключиться не удалось: статус и сообщение — в ответе.
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/SourceInput"
responses:
"201":
description: Хранилище создано
content:
application/json:
schema:
$ref: "#/components/schemas/Source"
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/sources/{id}:
parameters:
- $ref: "#/components/parameters/SourceId"
get:
operationId: getSource
tags: [library]
summary: Хранилище (admin)
responses:
"200":
description: Хранилище
content:
application/json:
schema:
$ref: "#/components/schemas/Source"
default:
$ref: "#/components/responses/Problem"
patch:
operationId: updateSource
tags: [library]
summary: Изменить хранилище (admin)
description: Изменение адреса NFS или папки переподключает хранилище.
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/SourcePatch"
responses:
"200":
description: Хранилище
content:
application/json:
schema:
$ref: "#/components/schemas/Source"
default:
$ref: "#/components/responses/Problem"
delete:
operationId: deleteSource
tags: [library]
summary: Отключить и удалить хранилище (admin)
description: Треки, файлы которых были только в этом хранилище, удаляются из каталога.
responses:
"204":
description: Удалено
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/sources/{id}/test:
parameters:
- $ref: "#/components/parameters/SourceId"
post:
operationId: testSource
tags: [library]
summary: Проверить подключение (admin)
responses:
"200":
description: Результат проверки
content:
application/json:
schema:
$ref: "#/components/schemas/SourceTestResult"
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/sources/{id}/patterns:
parameters:
- $ref: "#/components/parameters/SourceId"
get:
operationId: getSourcePatterns
tags: [library]
summary: Шаблоны путей (admin)
responses:
"200":
description: Шаблоны по порядку
content:
application/json:
schema:
$ref: "#/components/schemas/PatternList"
default:
$ref: "#/components/responses/Problem"
put:
operationId: putSourcePatterns
tags: [library]
summary: Заменить шаблоны путей (admin)
description: Ошибка компиляции — `422 validation.pattern_invalid` с `params.index` и `params.reason`.
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/PatternList"
responses:
"200":
description: Сохранённые шаблоны
content:
application/json:
schema:
$ref: "#/components/schemas/PatternList"
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/sources/{id}/patterns/preview:
parameters:
- $ref: "#/components/parameters/SourceId"
post:
operationId: previewSourcePatterns
tags: [library]
summary: Предпросмотр разбора путей без сохранения (admin)
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/PatternPreviewRequest"
responses:
"200":
description: Разбор выборки файлов
content:
application/json:
schema:
$ref: "#/components/schemas/PatternPreview"
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/sources/{id}/scan:
parameters:
- $ref: "#/components/parameters/SourceId"
post:
operationId: scanSource
tags: [library]
summary: Запустить скан (admin)
description: Скан идёт в фоне; прогресс — в `GET /admin/library/status`. Уже идёт — `409 library.scan_running`.
requestBody:
required: false
content:
application/json:
schema:
$ref: "#/components/schemas/ScanRequest"
responses:
"202":
description: Скан запущен
content:
application/json:
schema:
$ref: "#/components/schemas/ScanProgress"
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/library/status:
get:
operationId: getLibraryStatus
tags: [library]
summary: Состояние библиотеки (admin, moderator)
responses:
"200":
description: Счётчики и сканы
content:
application/json:
schema:
$ref: "#/components/schemas/LibraryStatus"
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/library/errors:
get:
operationId: listLibraryErrors
tags: [library]
summary: Файлы, которые не удалось прочитать (admin)
parameters:
- $ref: "#/components/parameters/Limit"
- $ref: "#/components/parameters/Cursor"
responses:
"200":
description: Страница ошибок
content:
application/json:
schema:
$ref: "#/components/schemas/LibraryErrorPage"
default:
$ref: "#/components/responses/Problem"
/api/v1/admin/library/host-dirs:
get:
operationId: listHostDirs
tags: [library]
summary: Подпапки каталога LIBRARY_HOST — выбор папки для local-хранилища (admin)
parameters:
- name: path
in: query
description: Путь относительно корня, пусто — корень
schema:
type: string
responses:
"200":
description: Подпапки
content:
application/json:
schema:
$ref: "#/components/schemas/HostDirList"
default:
$ref: "#/components/responses/Problem"
/api/v1/artists/{id}:
parameters:
- $ref: "#/components/parameters/EntityId"
get:
operationId: getArtist
tags: [catalog]
summary: Исполнитель
responses:
"200":
description: Исполнитель
content:
application/json:
schema:
$ref: "#/components/schemas/Artist"
default:
$ref: "#/components/responses/Problem"
/api/v1/artists/{id}/albums:
parameters:
- $ref: "#/components/parameters/EntityId"
get:
operationId: listArtistAlbums
tags: [catalog]
summary: Альбомы исполнителя (свои и сборники с его треками)
responses:
"200":
description: Альбомы исполнителя (свои и сборники с его треками)
content:
application/json:
schema:
$ref: "#/components/schemas/AlbumSummaryList"
default:
$ref: "#/components/responses/Problem"
/api/v1/albums/{id}:
parameters:
- $ref: "#/components/parameters/EntityId"
get:
operationId: getAlbum
tags: [catalog]
summary: Альбом с треками
responses:
"200":
description: Альбом с треками
content:
application/json:
schema:
$ref: "#/components/schemas/Album"
default:
$ref: "#/components/responses/Problem"
/api/v1/tracks/{id}:
parameters:
- $ref: "#/components/parameters/EntityId"
get:
operationId: getTrack
tags: [catalog]
summary: Трек
responses:
"200":
description: Трек
content:
application/json:
schema:
$ref: "#/components/schemas/Track"
default:
$ref: "#/components/responses/Problem"
components:
parameters:
SourceId:
name: id
in: path
required: true
schema:
type: string
format: uuid
EntityId:
name: id
in: path
required: true
schema:
type: string
format: uuid
Limit:
name: limit
in: query
@@ -695,6 +1005,462 @@ components:
next_cursor:
type: string
nullable: true
SourceType:
type: string
enum: [nfs, local]
SourceStatus:
type: string
enum: [pending, mounted, error, disabled]
TagPriority:
type: string
enum: [tags_first, path_first]
Source:
type: object
required: [id, name, type, nfs_server, nfs_export, nfs_options, local_path, tag_priority, scan_interval_minutes, status, status_message, last_scan_at, created_at, scan]
properties:
id:
type: string
format: uuid
name:
type: string
type:
$ref: "#/components/schemas/SourceType"
nfs_server:
type: string
nullable: true
nfs_export:
type: string
nullable: true
nfs_options:
type: string
local_path:
type: string
nullable: true
description: Подпапка относительно LIBRARY_HOST, пусто — весь каталог
tag_priority:
$ref: "#/components/schemas/TagPriority"
scan_interval_minutes:
type: integer
nullable: true
description: Автоматический скан, null — только вручную
status:
$ref: "#/components/schemas/SourceStatus"
status_message:
type: string
nullable: true
last_scan_at:
type: string
format: date-time
nullable: true
created_at:
type: string
format: date-time
scan:
nullable: true
allOf:
- $ref: "#/components/schemas/ScanProgress"
SourceList:
type: object
required: [items]
properties:
items:
type: array
items:
$ref: "#/components/schemas/Source"
SourceInput:
type: object
required: [name, type]
properties:
name:
type: string
maxLength: 100
type:
$ref: "#/components/schemas/SourceType"
nfs_server:
type: string
nfs_export:
type: string
nfs_options:
type: string
local_path:
type: string
tag_priority:
$ref: "#/components/schemas/TagPriority"
scan_interval_minutes:
type: integer
minimum: 5
nullable: true
SourcePatch:
type: object
properties:
name:
type: string
maxLength: 100
nfs_server:
type: string
nfs_export:
type: string
nfs_options:
type: string
local_path:
type: string
tag_priority:
$ref: "#/components/schemas/TagPriority"
scan_interval_minutes:
type: integer
description: 0 — отключить автоматический скан
enabled:
type: boolean
description: false — отключить хранилище (треки становятся недоступны), true — подключить снова
SourceTestResult:
type: object
required: [ok, message, audio_files]
properties:
ok:
type: boolean
message:
type: string
audio_files:
type: integer
description: Сколько аудиофайлов найдено (проверка останавливается на 1000)
PathPattern:
type: object
required: [regex, is_enabled]
properties:
regex:
type: string
maxLength: 1000
is_enabled:
type: boolean
PatternList:
type: object
required: [patterns]
properties:
patterns:
type: array
maxItems: 50
items:
$ref: "#/components/schemas/PathPattern"
PatternPreviewRequest:
type: object
required: [patterns]
properties:
patterns:
type: array
maxItems: 50
items:
$ref: "#/components/schemas/PathPattern"
sample:
type: integer
minimum: 1
maximum: 200
default: 50
TrackMeta:
type: object
required: [artist, album_artist, album, title, genres, year, track, disc]
properties:
artist:
type: string
album_artist:
type: string
album:
type: string
title:
type: string
genres:
type: array
items:
type: string
year:
type: integer
nullable: true
track:
type: integer
nullable: true
disc:
type: integer
nullable: true
PreviewItem:
type: object
required: [rel_path, pattern_index, skipped, path_fields, tags, result, error]
properties:
rel_path:
type: string
pattern_index:
type: integer
nullable: true
description: Номер совпавшего шаблона, null — ни один
skipped:
type: boolean
path_fields:
type: object
additionalProperties:
type: string
tags:
type: object
additionalProperties:
type: string
result:
$ref: "#/components/schemas/TrackMeta"
error:
type: string
nullable: true
PatternPreview:
type: object
required: [items, unmatched]
properties:
items:
type: array
items:
$ref: "#/components/schemas/PreviewItem"
unmatched:
type: integer
ScanRequest:
type: object
properties:
full:
type: boolean
description: Перечитать все файлы, а не только новые и изменённые
ScanProgress:
type: object
required: [running, full, started_at, finished_at, found, to_process, processed, added, updated, moved, missing, errors, error]
properties:
running:
type: boolean
full:
type: boolean
started_at:
type: string
format: date-time
finished_at:
type: string
format: date-time
nullable: true
found:
type: integer
to_process:
type: integer
processed:
type: integer
added:
type: integer
updated:
type: integer
moved:
type: integer
missing:
type: integer
errors:
type: integer
error:
type: string
nullable: true
LibraryStatus:
type: object
required: [artists, albums, tracks, files, missing_files, file_errors, sources]
properties:
artists:
type: integer
albums:
type: integer
tracks:
type: integer
files:
type: integer
missing_files:
type: integer
file_errors:
type: integer
sources:
type: array
items:
$ref: "#/components/schemas/SourceStatusItem"
SourceStatusItem:
type: object
required: [id, name, status, status_message, last_scan_at, scan]
properties:
id:
type: string
format: uuid
name:
type: string
status:
$ref: "#/components/schemas/SourceStatus"
status_message:
type: string
nullable: true
last_scan_at:
type: string
format: date-time
nullable: true
scan:
nullable: true
allOf:
- $ref: "#/components/schemas/ScanProgress"
LibraryError:
type: object
required: [file_id, source_id, rel_path, error]
properties:
file_id:
type: string
format: uuid
source_id:
type: string
format: uuid
rel_path:
type: string
error:
type: string
LibraryErrorPage:
type: object
required: [items, next_cursor]
properties:
items:
type: array
items:
$ref: "#/components/schemas/LibraryError"
next_cursor:
type: string
nullable: true
HostDirList:
type: object
required: [path, dirs]
properties:
path:
type: string
dirs:
type: array
items:
type: string
ArtistRef:
type: object
required: [id, name]
properties:
id:
type: string
format: uuid
name:
type: string
AlbumRef:
type: object
required: [id, title, year, cover_url, cover_colors]
properties:
id:
type: string
format: uuid
title:
type: string
year:
type: integer
nullable: true
cover_url:
type: string
nullable: true
cover_colors:
type: array
items:
type: string
Track:
type: object
required: [id, title, duration_ms, artists, album, genres, disc_no, track_no, available]
properties:
id:
type: string
format: uuid
title:
type: string
duration_ms:
type: integer
artists:
type: array
items:
$ref: "#/components/schemas/ArtistRef"
album:
$ref: "#/components/schemas/AlbumRef"
genres:
type: array
items:
type: string
disc_no:
type: integer
nullable: true
track_no:
type: integer
nullable: true
available:
type: boolean
description: Есть доступный файл (хранилище подключено, файл на месте)
Artist:
type: object
required: [id, name, album_count, track_count]
properties:
id:
type: string
format: uuid
name:
type: string
album_count:
type: integer
track_count:
type: integer
AlbumSummary:
type: object
required: [id, title, year, cover_url, cover_colors, artist, track_count]
properties:
id:
type: string
format: uuid
title:
type: string
year:
type: integer
nullable: true
cover_url:
type: string
nullable: true
cover_colors:
type: array
items:
type: string
artist:
$ref: "#/components/schemas/ArtistRef"
track_count:
type: integer
AlbumSummaryList:
type: object
required: [items]
properties:
items:
type: array
items:
$ref: "#/components/schemas/AlbumSummary"
Album:
type: object
required: [id, title, year, cover_url, cover_colors, artist, duration_ms, tracks]
properties:
id:
type: string
format: uuid
title:
type: string
year:
type: integer
nullable: true
cover_url:
type: string
nullable: true
cover_colors:
type: array
items:
type: string
artist:
$ref: "#/components/schemas/ArtistRef"
duration_ms:
type: integer
tracks:
type: array
items:
$ref: "#/components/schemas/Track"
Session:
type: object
required: [id, device_name, client_type, created_at, last_used_at, current]
+3 -2
View File
@@ -197,7 +197,7 @@ CREATE TABLE albums (
id uuid PRIMARY KEY,
title text NOT NULL,
title_norm text NOT NULL,
album_artist_id uuid REFERENCES artists(id),
album_artist_id uuid NOT NULL REFERENCES artists(id), -- без тегов — исполнитель трека или «Unknown artist»
year int,
cover_path text, -- путь в кэше обложек (/cache/covers/<hash>.webp)
cover_colors text[], -- палитра обложки (3 цвета: доминирующий, второй, акцент) — для ambient-фона,
@@ -208,7 +208,8 @@ CREATE TABLE albums (
CREATE TABLE tracks (
id uuid PRIMARY KEY,
title text NOT NULL,
album_id uuid REFERENCES albums(id),
title_norm text NOT NULL, -- ключ поиска дублей: (album_id, title_norm, duration ±2 с)
album_id uuid NOT NULL REFERENCES albums(id),
disc_no int, track_no int,
year int,
duration_ms int NOT NULL,
+878
View File
@@ -322,6 +322,265 @@ export interface paths {
patch?: never;
trace?: never;
};
"/api/v1/admin/sources": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/** Хранилища (admin) */
get: operations["listSources"];
put?: never;
/**
* Подключить хранилище (admin)
* @description `local` — подпапка в каталоге LIBRARY_HOST сервера; `nfs` — монтирует storage-agent.
* Хранилище сохраняется, даже если подключиться не удалось: статус и сообщение — в ответе.
*/
post: operations["createSource"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/admin/sources/{id}": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
/** Хранилище (admin) */
get: operations["getSource"];
put?: never;
post?: never;
/**
* Отключить и удалить хранилище (admin)
* @description Треки, файлы которых были только в этом хранилище, удаляются из каталога.
*/
delete: operations["deleteSource"];
options?: never;
head?: never;
/**
* Изменить хранилище (admin)
* @description Изменение адреса NFS или папки переподключает хранилище.
*/
patch: operations["updateSource"];
trace?: never;
};
"/api/v1/admin/sources/{id}/test": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
get?: never;
put?: never;
/** Проверить подключение (admin) */
post: operations["testSource"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/admin/sources/{id}/patterns": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
/** Шаблоны путей (admin) */
get: operations["getSourcePatterns"];
/**
* Заменить шаблоны путей (admin)
* @description Ошибка компиляции — `422 validation.pattern_invalid` с `params.index` и `params.reason`.
*/
put: operations["putSourcePatterns"];
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/admin/sources/{id}/patterns/preview": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
get?: never;
put?: never;
/** Предпросмотр разбора путей без сохранения (admin) */
post: operations["previewSourcePatterns"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/admin/sources/{id}/scan": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
get?: never;
put?: never;
/**
* Запустить скан (admin)
* @description Скан идёт в фоне; прогресс — в `GET /admin/library/status`. Уже идёт — `409 library.scan_running`.
*/
post: operations["scanSource"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/admin/library/status": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/** Состояние библиотеки (admin, moderator) */
get: operations["getLibraryStatus"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/admin/library/errors": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/** Файлы, которые не удалось прочитать (admin) */
get: operations["listLibraryErrors"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/admin/library/host-dirs": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/** Подпапки каталога LIBRARY_HOST — выбор папки для local-хранилища (admin) */
get: operations["listHostDirs"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/artists/{id}": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["EntityId"];
};
cookie?: never;
};
/** Исполнитель */
get: operations["getArtist"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/artists/{id}/albums": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["EntityId"];
};
cookie?: never;
};
/** Альбомы исполнителя (свои и сборники с его треками) */
get: operations["listArtistAlbums"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/albums/{id}": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["EntityId"];
};
cookie?: never;
};
/** Альбом с треками */
get: operations["getAlbum"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/v1/tracks/{id}": {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["EntityId"];
};
cookie?: never;
};
/** Трек */
get: operations["getTrack"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
}
export type webhooks = Record<string, never>;
export interface components {
@@ -491,6 +750,217 @@ export interface components {
items: components["schemas"]["AuditEntry"][];
next_cursor: string | null;
};
/** @enum {string} */
SourceType: "nfs" | "local";
/** @enum {string} */
SourceStatus: "pending" | "mounted" | "error" | "disabled";
/** @enum {string} */
TagPriority: "tags_first" | "path_first";
Source: {
/** Format: uuid */
id: string;
name: string;
type: components["schemas"]["SourceType"];
nfs_server: string | null;
nfs_export: string | null;
nfs_options: string;
/** @description Подпапка относительно LIBRARY_HOST, пусто — весь каталог */
local_path: string | null;
tag_priority: components["schemas"]["TagPriority"];
/** @description Автоматический скан, null — только вручную */
scan_interval_minutes: number | null;
status: components["schemas"]["SourceStatus"];
status_message: string | null;
/** Format: date-time */
last_scan_at: string | null;
/** Format: date-time */
created_at: string;
scan: components["schemas"]["ScanProgress"] | null;
};
SourceList: {
items: components["schemas"]["Source"][];
};
SourceInput: {
name: string;
type: components["schemas"]["SourceType"];
nfs_server?: string;
nfs_export?: string;
nfs_options?: string;
local_path?: string;
tag_priority?: components["schemas"]["TagPriority"];
scan_interval_minutes?: number | null;
};
SourcePatch: {
name?: string;
nfs_server?: string;
nfs_export?: string;
nfs_options?: string;
local_path?: string;
tag_priority?: components["schemas"]["TagPriority"];
/** @description 0 — отключить автоматический скан */
scan_interval_minutes?: number;
/** @description false — отключить хранилище (треки становятся недоступны), true — подключить снова */
enabled?: boolean;
};
SourceTestResult: {
ok: boolean;
message: string;
/** @description Сколько аудиофайлов найдено (проверка останавливается на 1000) */
audio_files: number;
};
PathPattern: {
regex: string;
is_enabled: boolean;
};
PatternList: {
patterns: components["schemas"]["PathPattern"][];
};
PatternPreviewRequest: {
patterns: components["schemas"]["PathPattern"][];
/** @default 50 */
sample: number;
};
TrackMeta: {
artist: string;
album_artist: string;
album: string;
title: string;
genres: string[];
year: number | null;
track: number | null;
disc: number | null;
};
PreviewItem: {
rel_path: string;
/** @description Номер совпавшего шаблона, null — ни один */
pattern_index: number | null;
skipped: boolean;
path_fields: {
[key: string]: string;
};
tags: {
[key: string]: string;
};
result: components["schemas"]["TrackMeta"];
error: string | null;
};
PatternPreview: {
items: components["schemas"]["PreviewItem"][];
unmatched: number;
};
ScanRequest: {
/** @description Перечитать все файлы, а не только новые и изменённые */
full?: boolean;
};
ScanProgress: {
running: boolean;
full: boolean;
/** Format: date-time */
started_at: string;
/** Format: date-time */
finished_at: string | null;
found: number;
to_process: number;
processed: number;
added: number;
updated: number;
moved: number;
missing: number;
errors: number;
error: string | null;
};
LibraryStatus: {
artists: number;
albums: number;
tracks: number;
files: number;
missing_files: number;
file_errors: number;
sources: components["schemas"]["SourceStatusItem"][];
};
SourceStatusItem: {
/** Format: uuid */
id: string;
name: string;
status: components["schemas"]["SourceStatus"];
status_message: string | null;
/** Format: date-time */
last_scan_at: string | null;
scan: components["schemas"]["ScanProgress"] | null;
};
LibraryError: {
/** Format: uuid */
file_id: string;
/** Format: uuid */
source_id: string;
rel_path: string;
error: string;
};
LibraryErrorPage: {
items: components["schemas"]["LibraryError"][];
next_cursor: string | null;
};
HostDirList: {
path: string;
dirs: string[];
};
ArtistRef: {
/** Format: uuid */
id: string;
name: string;
};
AlbumRef: {
/** Format: uuid */
id: string;
title: string;
year: number | null;
cover_url: string | null;
cover_colors: string[];
};
Track: {
/** Format: uuid */
id: string;
title: string;
duration_ms: number;
artists: components["schemas"]["ArtistRef"][];
album: components["schemas"]["AlbumRef"];
genres: string[];
disc_no: number | null;
track_no: number | null;
/** @description Есть доступный файл (хранилище подключено, файл на месте) */
available: boolean;
};
Artist: {
/** Format: uuid */
id: string;
name: string;
album_count: number;
track_count: number;
};
AlbumSummary: {
/** Format: uuid */
id: string;
title: string;
year: number | null;
cover_url: string | null;
cover_colors: string[];
artist: components["schemas"]["ArtistRef"];
track_count: number;
};
AlbumSummaryList: {
items: components["schemas"]["AlbumSummary"][];
};
Album: {
/** Format: uuid */
id: string;
title: string;
year: number | null;
cover_url: string | null;
cover_colors: string[];
artist: components["schemas"]["ArtistRef"];
duration_ms: number;
tracks: components["schemas"]["Track"][];
};
Session: {
/** Format: uuid */
id: string;
@@ -540,6 +1010,8 @@ export interface components {
};
};
parameters: {
SourceId: string;
EntityId: string;
Limit: number;
/** @description Непрозрачный курсор из `next_cursor` предыдущей страницы */
Cursor: string;
@@ -997,4 +1469,410 @@ export interface operations {
default: components["responses"]["Problem"];
};
};
listSources: {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Список хранилищ */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["SourceList"];
};
};
default: components["responses"]["Problem"];
};
};
createSource: {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
requestBody: {
content: {
"application/json": components["schemas"]["SourceInput"];
};
};
responses: {
/** @description Хранилище создано */
201: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Source"];
};
};
default: components["responses"]["Problem"];
};
};
getSource: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Хранилище */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Source"];
};
};
default: components["responses"]["Problem"];
};
};
deleteSource: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Удалено */
204: {
headers: {
[name: string]: unknown;
};
content?: never;
};
default: components["responses"]["Problem"];
};
};
updateSource: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
requestBody: {
content: {
"application/json": components["schemas"]["SourcePatch"];
};
};
responses: {
/** @description Хранилище */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Source"];
};
};
default: components["responses"]["Problem"];
};
};
testSource: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Результат проверки */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["SourceTestResult"];
};
};
default: components["responses"]["Problem"];
};
};
getSourcePatterns: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Шаблоны по порядку */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["PatternList"];
};
};
default: components["responses"]["Problem"];
};
};
putSourcePatterns: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
requestBody: {
content: {
"application/json": components["schemas"]["PatternList"];
};
};
responses: {
/** @description Сохранённые шаблоны */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["PatternList"];
};
};
default: components["responses"]["Problem"];
};
};
previewSourcePatterns: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
requestBody: {
content: {
"application/json": components["schemas"]["PatternPreviewRequest"];
};
};
responses: {
/** @description Разбор выборки файлов */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["PatternPreview"];
};
};
default: components["responses"]["Problem"];
};
};
scanSource: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["SourceId"];
};
cookie?: never;
};
requestBody?: {
content: {
"application/json": components["schemas"]["ScanRequest"];
};
};
responses: {
/** @description Скан запущен */
202: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["ScanProgress"];
};
};
default: components["responses"]["Problem"];
};
};
getLibraryStatus: {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Счётчики и сканы */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["LibraryStatus"];
};
};
default: components["responses"]["Problem"];
};
};
listLibraryErrors: {
parameters: {
query?: {
limit?: components["parameters"]["Limit"];
/** @description Непрозрачный курсор из `next_cursor` предыдущей страницы */
cursor?: components["parameters"]["Cursor"];
};
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Страница ошибок */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["LibraryErrorPage"];
};
};
default: components["responses"]["Problem"];
};
};
listHostDirs: {
parameters: {
query?: {
/** @description Путь относительно корня, пусто — корень */
path?: string;
};
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Подпапки */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["HostDirList"];
};
};
default: components["responses"]["Problem"];
};
};
getArtist: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["EntityId"];
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Исполнитель */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Artist"];
};
};
default: components["responses"]["Problem"];
};
};
listArtistAlbums: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["EntityId"];
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Альбомы исполнителя (свои и сборники с его треками) */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["AlbumSummaryList"];
};
};
default: components["responses"]["Problem"];
};
};
getAlbum: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["EntityId"];
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Альбом с треками */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Album"];
};
};
default: components["responses"]["Problem"];
};
};
getTrack: {
parameters: {
query?: never;
header?: never;
path: {
id: components["parameters"]["EntityId"];
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Трек */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Track"];
};
};
default: components["responses"]["Problem"];
};
};
}
+4 -1
View File
@@ -14,8 +14,11 @@ RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache
go build -trimpath -ldflags "-s -w -X pipu/api/internal/buildinfo.Version=${VERSION}" -o /out/api ./cmd/api && \
go build -trimpath -ldflags "-s -w -X pipu/api/internal/buildinfo.Version=${VERSION}" -o /out/storage-agent ./cmd/storage-agent
FROM gcr.io/distroless/static-debian12:nonroot AS api
# api вызывает ffprobe (сканер) и ffmpeg (транскодинг, M3) — нужен не distroless, а alpine с ffmpeg.
FROM alpine:3.22 AS api
RUN apk add --no-cache ffmpeg ca-certificates tzdata
COPY --from=build /out/api /usr/local/bin/api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/api"]
+12
View File
@@ -9,12 +9,14 @@ import (
"os"
"os/signal"
"syscall"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/redis/go-redis/v9"
"pipu/api/internal/auth"
"pipu/api/internal/buildinfo"
"pipu/api/internal/library"
"pipu/api/internal/platform/config"
"pipu/api/internal/platform/db"
"pipu/api/internal/platform/httpx"
@@ -78,11 +80,21 @@ func run() error {
"redis": system.CheckerFunc(func(ctx context.Context) error { return rdb.Ping(ctx).Err() }),
})
libStore := library.NewStore(pool)
prober := library.FFprobe{Bin: cfg.FFprobePath, Timeout: time.Minute}
board := library.NewProgressBoard()
scanner := library.NewScanner(libStore, prober, board, log)
libSvc := library.NewService(libStore, scanner, board, library.NewHTTPAgent(cfg.StorageAgentURL, cfg.InternalToken),
prober, library.Config{LibraryRoot: cfg.LibraryRoot, HostRoot: cfg.LibraryHostRoot}, log)
go libSvc.Restore(ctx)
go libSvc.RunSchedule(ctx)
srv := httpx.NewServer(cfg.HTTPAddr, server.NewRouter(server.Deps{
Log: log,
System: sys,
Auth: auth.NewHandler(authSvc, cfg.CookieSecure),
AuthService: authSvc,
Library: library.NewHandler(libSvc, libStore),
}))
log.Info("starting", "version", buildinfo.Version)
return httpx.Serve(ctx, log, srv, cfg.ShutdownTimeout)
File diff suppressed because it is too large. Load diff
+105
View File
@@ -0,0 +1,105 @@
package library
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"time"
"github.com/google/uuid"
"pipu/api/internal/storageagent"
)
// Agent — клиент storage-agent (монтирование NFS).
type Agent interface {
Mount(ctx context.Context, sourceID uuid.UUID, spec storageagent.NFSSpec) (storageagent.MountResult, error)
Unmount(ctx context.Context, sourceID uuid.UUID) error
Test(ctx context.Context, spec storageagent.NFSSpec) (storageagent.MountResult, error)
}
// HTTPAgent ходит в storage-agent по внутренней сети с общим секретом.
type HTTPAgent struct {
BaseURL string
Token string
Client *http.Client
}
// NewHTTPAgent создаёт клиента агента.
func NewHTTPAgent(baseURL, token string) *HTTPAgent {
return &HTTPAgent{BaseURL: baseURL, Token: token, Client: &http.Client{Timeout: 40 * time.Second}}
}
// AgentError — агент отклонил запрос (например, недопустимые параметры NFS).
type AgentError struct {
Status int
Field string
Reason string
}
func (e *AgentError) Error() string {
return fmt.Sprintf("storage-agent: %d %s %s", e.Status, e.Field, e.Reason)
}
func (a *HTTPAgent) do(ctx context.Context, method, path string, body any, out any) error {
var rd *bytes.Reader
if body != nil {
raw, err := json.Marshal(body)
if err != nil {
return err
}
rd = bytes.NewReader(raw)
} else {
rd = bytes.NewReader(nil)
}
req, err := http.NewRequestWithContext(ctx, method, a.BaseURL+path, rd)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+a.Token)
req.Header.Set("Content-Type", "application/json")
res, err := a.Client.Do(req)
if err != nil {
return fmt.Errorf("storage-agent: %w", err)
}
defer func() { _ = res.Body.Close() }()
if res.StatusCode >= 400 {
var p struct {
Detail string `json:"detail"`
Errors []struct {
Field string `json:"field"`
} `json:"errors"`
}
_ = json.NewDecoder(res.Body).Decode(&p)
ae := &AgentError{Status: res.StatusCode, Reason: p.Detail}
if len(p.Errors) > 0 {
ae.Field = p.Errors[0].Field
}
return ae
}
if out != nil {
return json.NewDecoder(res.Body).Decode(out)
}
return nil
}
// Mount реализует Agent.
func (a *HTTPAgent) Mount(ctx context.Context, id uuid.UUID, spec storageagent.NFSSpec) (storageagent.MountResult, error) {
var res storageagent.MountResult
err := a.do(ctx, http.MethodPost, "/internal/mounts", storageagent.MountRequest{SourceID: id.String(), NFSSpec: spec}, &res)
return res, err
}
// Unmount реализует Agent.
func (a *HTTPAgent) Unmount(ctx context.Context, id uuid.UUID) error {
return a.do(ctx, http.MethodDelete, "/internal/mounts/"+id.String(), nil, nil)
}
// Test реализует Agent.
func (a *HTTPAgent) Test(ctx context.Context, spec storageagent.NFSSpec) (storageagent.MountResult, error) {
var res storageagent.MountResult
err := a.do(ctx, http.MethodPost, "/internal/mounts/test", spec, &res)
return res, err
}
+191
View File
@@ -0,0 +1,191 @@
package library
import (
"context"
"encoding/json"
"errors"
"fmt"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
// Ref — ссылка на исполнителя.
type Ref struct {
ID uuid.UUID `json:"id"`
Name string `json:"name"`
}
// AlbumRow — альбом для карточек и страниц.
type AlbumRow struct {
ID uuid.UUID
Title string
Year *int
CoverPath *string
CoverColors []string
Artist Ref
TrackCount int
}
// TrackRow — трек с исполнителями, альбомом и жанрами.
type TrackRow struct {
ID uuid.UUID
Title string
DurationMs int
DiscNo *int
TrackNo *int
Artists []Ref
Album AlbumRow
Genres []string
Available bool
}
// ArtistRow — исполнитель со счётчиками.
type ArtistRow struct {
ID uuid.UUID
Name string
AlbumCount int
TrackCount int
}
// trackSelect — общий SELECT треков; доступен трек, у которого есть основной файл
// в подключённом хранилище.
const trackSelect = `
SELECT t.id, t.title, t.duration_ms, t.disc_no, t.track_no,
COALESCE((SELECT json_agg(json_build_object('id', a.id, 'name', a.name) ORDER BY ta.role DESC, ta.position)
FROM track_artists ta JOIN artists a ON a.id = ta.artist_id WHERE ta.track_id = t.id), '[]'),
al.id, al.title, al.year, al.cover_path, COALESCE(al.cover_colors, '{}'), aa.id, aa.name,
COALESCE((SELECT array_agg(g.name ORDER BY g.name) FROM track_genres tg JOIN genres g ON g.id = tg.genre_id
WHERE tg.track_id = t.id), '{}'),
EXISTS (SELECT 1 FROM media_files f JOIN library_sources s ON s.id = f.source_id
WHERE f.id = t.primary_file_id AND s.status = 'mounted')
FROM tracks t
JOIN albums al ON al.id = t.album_id
JOIN artists aa ON aa.id = al.album_artist_id`
func scanTrack(row pgx.Row) (TrackRow, error) {
var (
t TrackRow
artists []byte
)
err := row.Scan(&t.ID, &t.Title, &t.DurationMs, &t.DiscNo, &t.TrackNo, &artists,
&t.Album.ID, &t.Album.Title, &t.Album.Year, &t.Album.CoverPath, &t.Album.CoverColors,
&t.Album.Artist.ID, &t.Album.Artist.Name, &t.Genres, &t.Available)
if errors.Is(err, pgx.ErrNoRows) {
return TrackRow{}, ErrNotFound
}
if err != nil {
return TrackRow{}, fmt.Errorf("library: scan track: %w", err)
}
if err := json.Unmarshal(artists, &t.Artists); err != nil {
return TrackRow{}, fmt.Errorf("library: track artists: %w", err)
}
return t, nil
}
func (st *Store) queryTracks(ctx context.Context, where string, args ...any) ([]TrackRow, error) {
rows, err := st.pool.Query(ctx, trackSelect+" "+where, args...)
if err != nil {
return nil, fmt.Errorf("library: tracks: %w", err)
}
defer rows.Close()
var out []TrackRow
for rows.Next() {
t, err := scanTrack(rows)
if err != nil {
return nil, err
}
out = append(out, t)
}
return out, rows.Err()
}
// Track — трек по id.
func (st *Store) Track(ctx context.Context, id uuid.UUID) (TrackRow, error) {
return scanTrack(st.pool.QueryRow(ctx, trackSelect+" WHERE t.id = $1", id))
}
// TracksByIDs — треки в порядке ids (недостающие пропускаются).
func (st *Store) TracksByIDs(ctx context.Context, ids []uuid.UUID) ([]TrackRow, error) {
rows, err := st.queryTracks(ctx, "WHERE t.id = ANY($1)", ids)
if err != nil {
return nil, err
}
byID := make(map[uuid.UUID]TrackRow, len(rows))
for _, t := range rows {
byID[t.ID] = t
}
out := make([]TrackRow, 0, len(ids))
for _, id := range ids {
if t, ok := byID[id]; ok {
out = append(out, t)
}
}
return out, nil
}
// AlbumTracks — треки альбома по дискам и номерам.
func (st *Store) AlbumTracks(ctx context.Context, albumID uuid.UUID) ([]TrackRow, error) {
return st.queryTracks(ctx, "WHERE t.album_id = $1 ORDER BY t.disc_no NULLS FIRST, t.track_no NULLS LAST, t.title", albumID)
}
const albumSelect = `
SELECT al.id, al.title, al.year, al.cover_path, COALESCE(al.cover_colors, '{}'), aa.id, aa.name,
(SELECT count(*) FROM tracks t WHERE t.album_id = al.id)
FROM albums al JOIN artists aa ON aa.id = al.album_artist_id`
func scanAlbum(row pgx.Row) (AlbumRow, error) {
var a AlbumRow
err := row.Scan(&a.ID, &a.Title, &a.Year, &a.CoverPath, &a.CoverColors, &a.Artist.ID, &a.Artist.Name, &a.TrackCount)
if errors.Is(err, pgx.ErrNoRows) {
return AlbumRow{}, ErrNotFound
}
if err != nil {
return AlbumRow{}, fmt.Errorf("library: scan album: %w", err)
}
return a, nil
}
// Album — альбом по id.
func (st *Store) Album(ctx context.Context, id uuid.UUID) (AlbumRow, error) {
return scanAlbum(st.pool.QueryRow(ctx, albumSelect+" WHERE al.id = $1", id))
}
// ArtistAlbums — альбомы исполнителя и сборники, где есть его треки; новые сверху.
func (st *Store) ArtistAlbums(ctx context.Context, artistID uuid.UUID) ([]AlbumRow, error) {
rows, err := st.pool.Query(ctx, albumSelect+`
WHERE al.album_artist_id = $1 OR EXISTS (
SELECT 1 FROM tracks t JOIN track_artists ta ON ta.track_id = t.id
WHERE t.album_id = al.id AND ta.artist_id = $1)
ORDER BY (al.album_artist_id = $1) DESC, al.year DESC NULLS LAST, al.title`, artistID)
if err != nil {
return nil, fmt.Errorf("library: artist albums: %w", err)
}
defer rows.Close()
var out []AlbumRow
for rows.Next() {
a, err := scanAlbum(rows)
if err != nil {
return nil, err
}
out = append(out, a)
}
return out, rows.Err()
}
// Artist — исполнитель со счётчиками альбомов и треков.
func (st *Store) Artist(ctx context.Context, id uuid.UUID) (ArtistRow, error) {
var a ArtistRow
err := st.pool.QueryRow(ctx, `
SELECT a.id, a.name,
(SELECT count(*) FROM albums al WHERE al.album_artist_id = a.id),
(SELECT count(DISTINCT ta.track_id) FROM track_artists ta WHERE ta.artist_id = a.id)
FROM artists a WHERE a.id = $1`, id).Scan(&a.ID, &a.Name, &a.AlbumCount, &a.TrackCount)
if errors.Is(err, pgx.ErrNoRows) {
return ArtistRow{}, ErrNotFound
}
if err != nil {
return ArtistRow{}, fmt.Errorf("library: artist: %w", err)
}
return a, nil
}
+378
View File
@@ -0,0 +1,378 @@
package library
import (
"context"
"encoding/base64"
"errors"
"time"
"pipu/api/internal/api/gen"
"pipu/api/internal/library/meta"
"pipu/api/internal/platform/apperr"
)
// Handler реализует library- и catalog-части gen.StrictServerInterface.
type Handler struct {
svc *Service
store *Store
}
// NewHandler создаёт обработчик.
func NewHandler(svc *Service, store *Store) *Handler { return &Handler{svc: svc, store: store} }
func (h *Handler) sourceDTO(s Source) gen.Source {
out := gen.Source{
Id: s.ID, Name: s.Name, Type: gen.SourceType(s.Type), NfsServer: s.NFSServer, NfsExport: s.NFSExport,
NfsOptions: s.NFSOptions, LocalPath: s.LocalPath, TagPriority: gen.TagsFirst,
Status: gen.SourceStatus(s.Status), StatusMessage: s.StatusMessage, LastScanAt: s.LastScanAt,
CreatedAt: s.CreatedAt,
}
if !s.TagsFirst {
out.TagPriority = gen.PathFirst
}
if s.ScanInterval != nil {
m := int(*s.ScanInterval / time.Minute)
out.ScanIntervalMinutes = &m
}
if p, ok := h.svc.Progress(s.ID); ok {
dto := progressDTO(p)
out.Scan = &dto
}
return out
}
func progressDTO(p Progress) gen.ScanProgress {
out := gen.ScanProgress{
Running: p.Running, Full: p.Full, StartedAt: p.StartedAt, FinishedAt: p.FinishedAt, Found: p.Found,
ToProcess: p.ToProcess, Processed: p.Processed, Added: p.Added, Updated: p.Updated, Moved: p.Moved,
Missing: p.Missing, Errors: p.Errors,
}
if p.Error != "" {
out.Error = &p.Error
}
return out
}
// ListSources реализует gen.StrictServerInterface.
func (h *Handler) ListSources(ctx context.Context, _ gen.ListSourcesRequestObject) (gen.ListSourcesResponseObject, error) {
sources, err := h.svc.ListSources(ctx)
if err != nil {
return nil, err
}
items := make([]gen.Source, 0, len(sources))
for _, s := range sources {
items = append(items, h.sourceDTO(s))
}
return gen.ListSources200JSONResponse{Items: items}, nil
}
func tagsFirst(p *gen.TagPriority) *bool {
if p == nil {
return nil
}
v := *p == gen.TagsFirst
return &v
}
// CreateSource реализует gen.StrictServerInterface.
func (h *Handler) CreateSource(ctx context.Context, req gen.CreateSourceRequestObject) (gen.CreateSourceResponseObject, error) {
b := req.Body
src, err := h.svc.CreateSource(ctx, SourceInput{
Name: &b.Name, Type: SourceType(b.Type), NFSServer: b.NfsServer, NFSExport: b.NfsExport, NFSOptions: b.NfsOptions,
LocalPath: b.LocalPath, TagsFirst: tagsFirst(b.TagPriority), ScanIntervalMinutes: b.ScanIntervalMinutes,
})
if err != nil {
return nil, err
}
return gen.CreateSource201JSONResponse(h.sourceDTO(src)), nil
}
// GetSource реализует gen.StrictServerInterface.
func (h *Handler) GetSource(ctx context.Context, req gen.GetSourceRequestObject) (gen.GetSourceResponseObject, error) {
src, err := h.svc.GetSource(ctx, req.Id)
if err != nil {
return nil, err
}
return gen.GetSource200JSONResponse(h.sourceDTO(src)), nil
}
// UpdateSource реализует gen.StrictServerInterface.
func (h *Handler) UpdateSource(ctx context.Context, req gen.UpdateSourceRequestObject) (gen.UpdateSourceResponseObject, error) {
b := req.Body
src, err := h.svc.UpdateSource(ctx, req.Id, SourceInput{
Name: b.Name, NFSServer: b.NfsServer, NFSExport: b.NfsExport, NFSOptions: b.NfsOptions, LocalPath: b.LocalPath,
TagsFirst: tagsFirst(b.TagPriority), ScanIntervalMinutes: b.ScanIntervalMinutes, Enabled: b.Enabled,
})
if err != nil {
return nil, err
}
return gen.UpdateSource200JSONResponse(h.sourceDTO(src)), nil
}
// DeleteSource реализует gen.StrictServerInterface.
func (h *Handler) DeleteSource(ctx context.Context, req gen.DeleteSourceRequestObject) (gen.DeleteSourceResponseObject, error) {
if err := h.svc.DeleteSource(ctx, req.Id); err != nil {
return nil, err
}
return gen.DeleteSource204Response{}, nil
}
// TestSource реализует gen.StrictServerInterface.
func (h *Handler) TestSource(ctx context.Context, req gen.TestSourceRequestObject) (gen.TestSourceResponseObject, error) {
r, err := h.svc.TestSource(ctx, req.Id)
if err != nil {
return nil, err
}
return gen.TestSource200JSONResponse{Ok: r.OK, Message: r.Message, AudioFiles: r.AudioFiles}, nil
}
func patternsDTO(rows []PatternRow) gen.PatternList {
out := gen.PatternList{Patterns: make([]gen.PathPattern, 0, len(rows))}
for _, r := range rows {
out.Patterns = append(out.Patterns, gen.PathPattern{Regex: r.Regex, IsEnabled: r.IsEnabled})
}
return out
}
func patternRows(in []gen.PathPattern) []PatternRow {
out := make([]PatternRow, 0, len(in))
for _, p := range in {
out = append(out, PatternRow{Regex: p.Regex, IsEnabled: p.IsEnabled})
}
return out
}
// GetSourcePatterns реализует gen.StrictServerInterface.
func (h *Handler) GetSourcePatterns(ctx context.Context, req gen.GetSourcePatternsRequestObject) (gen.GetSourcePatternsResponseObject, error) {
rows, err := h.svc.Patterns(ctx, req.Id)
if err != nil {
return nil, err
}
return gen.GetSourcePatterns200JSONResponse(patternsDTO(rows)), nil
}
// PutSourcePatterns реализует gen.StrictServerInterface.
func (h *Handler) PutSourcePatterns(ctx context.Context, req gen.PutSourcePatternsRequestObject) (gen.PutSourcePatternsResponseObject, error) {
rows, err := h.svc.PutPatterns(ctx, req.Id, patternRows(req.Body.Patterns))
if err != nil {
return nil, err
}
return gen.PutSourcePatterns200JSONResponse(patternsDTO(rows)), nil
}
func intPtr(n int) *int {
if n == 0 {
return nil
}
return &n
}
func metaDTO(m meta.Meta) gen.TrackMeta {
genres := m.Genres
if genres == nil {
genres = []string{}
}
return gen.TrackMeta{
Artist: m.Artist, AlbumArtist: m.AlbumArtist, Album: m.Album, Title: m.Title, Genres: genres,
Year: intPtr(m.Year), Track: intPtr(m.Track), Disc: intPtr(m.Disc),
}
}
// PreviewSourcePatterns реализует gen.StrictServerInterface.
func (h *Handler) PreviewSourcePatterns(ctx context.Context, req gen.PreviewSourcePatternsRequestObject) (gen.PreviewSourcePatternsResponseObject, error) {
sample := 50
if req.Body.Sample != nil {
sample = *req.Body.Sample
}
items, unmatched, err := h.svc.Preview(ctx, req.Id, patternRows(req.Body.Patterns), sample)
if err != nil {
return nil, err
}
out := gen.PreviewSourcePatterns200JSONResponse{Items: make([]gen.PreviewItem, 0, len(items)), Unmatched: unmatched}
for _, it := range items {
dto := gen.PreviewItem{
RelPath: it.RelPath, PatternIndex: it.PatternIndex, Skipped: it.Skipped, PathFields: it.PathFields,
Tags: it.Tags, Result: metaDTO(it.Result),
}
if it.Error != "" {
e := it.Error
dto.Error = &e
}
out.Items = append(out.Items, dto)
}
return out, nil
}
// ScanSource реализует gen.StrictServerInterface.
func (h *Handler) ScanSource(ctx context.Context, req gen.ScanSourceRequestObject) (gen.ScanSourceResponseObject, error) {
full := req.Body != nil && req.Body.Full != nil && *req.Body.Full
p, err := h.svc.StartScan(ctx, req.Id, full)
if err != nil {
return nil, err
}
return gen.ScanSource202JSONResponse(progressDTO(p)), nil
}
// GetLibraryStatus реализует gen.StrictServerInterface.
func (h *Handler) GetLibraryStatus(ctx context.Context, _ gen.GetLibraryStatusRequestObject) (gen.GetLibraryStatusResponseObject, error) {
st, err := h.svc.Status(ctx)
if err != nil {
return nil, err
}
c := st.Counts
out := gen.GetLibraryStatus200JSONResponse{
Artists: int(c.Artists), Albums: int(c.Albums), Tracks: int(c.Tracks), Files: int(c.Files),
MissingFiles: int(c.Missing), FileErrors: int(c.Errors), Sources: []gen.SourceStatusItem{},
}
for _, s := range st.Sources {
dto := h.sourceDTO(s)
out.Sources = append(out.Sources, gen.SourceStatusItem{
Id: s.ID, Name: s.Name, Status: dto.Status, StatusMessage: s.StatusMessage, LastScanAt: s.LastScanAt, Scan: dto.Scan,
})
}
return out, nil
}
// ListLibraryErrors реализует gen.StrictServerInterface.
func (h *Handler) ListLibraryErrors(ctx context.Context, req gen.ListLibraryErrorsRequestObject) (gen.ListLibraryErrorsResponseObject, error) {
limit := 50
if req.Params.Limit != nil {
limit = min(max(*req.Params.Limit, 1), 200)
}
after := ""
if req.Params.Cursor != nil {
raw, err := base64.RawURLEncoding.DecodeString(*req.Params.Cursor)
if err != nil {
return nil, apperr.Validation("cursor", "validation.cursor_invalid", "malformed cursor", nil)
}
after = string(raw)
}
rows, err := h.svc.FileErrors(ctx, after, limit+1)
if err != nil {
return nil, err
}
out := gen.ListLibraryErrors200JSONResponse{Items: []gen.LibraryError{}}
if len(rows) > limit {
rows = rows[:limit]
next := base64.RawURLEncoding.EncodeToString([]byte(rows[limit-1].RelPath))
out.NextCursor = &next
}
for _, r := range rows {
out.Items = append(out.Items, gen.LibraryError{FileId: r.ID, SourceId: r.SourceID, RelPath: r.RelPath, Error: r.ScanError})
}
return out, nil
}
// ListHostDirs реализует gen.StrictServerInterface.
func (h *Handler) ListHostDirs(ctx context.Context, req gen.ListHostDirsRequestObject) (gen.ListHostDirsResponseObject, error) {
rel := ""
if req.Params.Path != nil {
rel = *req.Params.Path
}
p, dirs, err := h.svc.HostDirs(ctx, rel)
if err != nil {
return nil, err
}
return gen.ListHostDirs200JSONResponse{Path: p, Dirs: dirs}, nil
}
// --- каталог ---
func catalogErr(err error) error {
if errors.Is(err, ErrNotFound) {
return apperr.NotFound(CodeNotFound, "not found")
}
return err
}
func albumRefDTO(a AlbumRow) gen.AlbumRef {
return gen.AlbumRef{Id: a.ID, Title: a.Title, Year: a.Year, CoverUrl: coverURL(a), CoverColors: colors(a.CoverColors)}
}
// coverURL — обложки появятся вместе с извлечением (следующий шаг M2).
func coverURL(AlbumRow) *string { return nil }
func colors(c []string) []string {
if c == nil {
return []string{}
}
return c
}
// TrackDTO переводит трек в схему Track.
func TrackDTO(t TrackRow) gen.Track {
artists := make([]gen.ArtistRef, 0, len(t.Artists))
for _, a := range t.Artists {
artists = append(artists, gen.ArtistRef{Id: a.ID, Name: a.Name})
}
genres := t.Genres
if genres == nil {
genres = []string{}
}
return gen.Track{
Id: t.ID, Title: t.Title, DurationMs: t.DurationMs, Artists: artists, Album: albumRefDTO(t.Album),
Genres: genres, DiscNo: t.DiscNo, TrackNo: t.TrackNo, Available: t.Available,
}
}
func albumSummaryDTO(a AlbumRow) gen.AlbumSummary {
return gen.AlbumSummary{
Id: a.ID, Title: a.Title, Year: a.Year, CoverUrl: coverURL(a), CoverColors: colors(a.CoverColors),
Artist: gen.ArtistRef{Id: a.Artist.ID, Name: a.Artist.Name}, TrackCount: a.TrackCount,
}
}
// GetArtist реализует gen.StrictServerInterface.
func (h *Handler) GetArtist(ctx context.Context, req gen.GetArtistRequestObject) (gen.GetArtistResponseObject, error) {
a, err := h.store.Artist(ctx, req.Id)
if err != nil {
return nil, catalogErr(err)
}
return gen.GetArtist200JSONResponse{Id: a.ID, Name: a.Name, AlbumCount: a.AlbumCount, TrackCount: a.TrackCount}, nil
}
// ListArtistAlbums реализует gen.StrictServerInterface.
func (h *Handler) ListArtistAlbums(ctx context.Context, req gen.ListArtistAlbumsRequestObject) (gen.ListArtistAlbumsResponseObject, error) {
if _, err := h.store.Artist(ctx, req.Id); err != nil {
return nil, catalogErr(err)
}
rows, err := h.store.ArtistAlbums(ctx, req.Id)
if err != nil {
return nil, err
}
out := gen.ListArtistAlbums200JSONResponse{Items: make([]gen.AlbumSummary, 0, len(rows))}
for _, a := range rows {
out.Items = append(out.Items, albumSummaryDTO(a))
}
return out, nil
}
// GetAlbum реализует gen.StrictServerInterface.
func (h *Handler) GetAlbum(ctx context.Context, req gen.GetAlbumRequestObject) (gen.GetAlbumResponseObject, error) {
a, err := h.store.Album(ctx, req.Id)
if err != nil {
return nil, catalogErr(err)
}
tracks, err := h.store.AlbumTracks(ctx, req.Id)
if err != nil {
return nil, err
}
out := gen.GetAlbum200JSONResponse{
Id: a.ID, Title: a.Title, Year: a.Year, CoverUrl: coverURL(a), CoverColors: colors(a.CoverColors),
Artist: gen.ArtistRef{Id: a.Artist.ID, Name: a.Artist.Name}, Tracks: make([]gen.Track, 0, len(tracks)),
}
for _, t := range tracks {
out.DurationMs += t.DurationMs
out.Tracks = append(out.Tracks, TrackDTO(t))
}
return out, nil
}
// GetTrack реализует gen.StrictServerInterface.
func (h *Handler) GetTrack(ctx context.Context, req gen.GetTrackRequestObject) (gen.GetTrackResponseObject, error) {
t, err := h.store.Track(ctx, req.Id)
if err != nil {
return nil, catalogErr(err)
}
return gen.GetTrack200JSONResponse(TrackDTO(t)), nil
}
+137
View File
@@ -0,0 +1,137 @@
package meta
import (
"path"
"regexp"
"strconv"
"strings"
)
// UnknownArtist — исполнитель, если его нет ни в тегах, ни в пути.
const UnknownArtist = "Unknown artist"
// Fields — значения метаданных из одного источника (теги файла или шаблон пути). Пусто — нет значения.
type Fields struct {
Artist string
AlbumArtist string
Album string
Title string
Genre string
Year string
Track string
Disc string
}
// FromPath собирает Fields из групп шаблона.
func FromPath(m map[string]string) Fields {
return Fields{
Artist: m["artist"], AlbumArtist: m["albumartist"], Album: m["album"], Title: m["title"],
Genre: m["genre"], Year: m["year"], Track: m["track"], Disc: m["disc"],
}
}
// Meta — итоговые метаданные файла.
type Meta struct {
Artist string // поле исполнителя целиком (разбивается SplitArtists)
AlbumArtist string
Album string
Title string
Genres []string
Year int
Track int
Disc int
}
// Merge сводит теги и поля из пути. tagsFirst: тег важнее пути (по умолчанию), иначе наоборот.
// Без обоих значений: title — имя файла без номера и расширения, artist — UnknownArtist,
// album — имя папки, albumartist — artist.
func Merge(tags, fromPath Fields, tagsFirst bool, relPath string) Meta {
first, second := tags, fromPath
if !tagsFirst {
first, second = fromPath, tags
}
pick := func(a, b string) string {
if v := Clean(a); v != "" {
return v
}
return Clean(b)
}
m := Meta{
Artist: pick(first.Artist, second.Artist),
AlbumArtist: pick(first.AlbumArtist, second.AlbumArtist),
Album: pick(first.Album, second.Album),
Title: pick(first.Title, second.Title),
Year: parseYear(pick(first.Year, second.Year)),
Track: parseNumber(pick(first.Track, second.Track)),
Disc: parseNumber(pick(first.Disc, second.Disc)),
}
if g := pick(first.Genre, second.Genre); g != "" {
m.Genres = splitGenres(g)
}
if m.Title == "" {
m.Title = titleFromFile(relPath)
}
if m.Artist == "" {
m.Artist = m.AlbumArtist
}
if m.Artist == "" {
m.Artist = UnknownArtist
}
if m.AlbumArtist == "" {
m.AlbumArtist = m.Artist
}
if m.Album == "" {
if dir := path.Base(path.Dir(relPath)); dir != "." && dir != "/" {
m.Album = Clean(dir)
}
}
return m
}
var yearRe = regexp.MustCompile(`\b(1[89]\d\d|20\d\d)\b`)
// parseYear достаёт год из «2019», «2019-05-01», «(1999)».
func parseYear(s string) int {
if y := yearRe.FindString(s); y != "" {
n, _ := strconv.Atoi(y)
return n
}
return 0
}
// parseNumber разбирает «3», «03», «3/12».
func parseNumber(s string) int {
s, _, _ = strings.Cut(s, "/")
n, err := strconv.Atoi(strings.TrimSpace(s))
if err != nil || n < 0 || n > 9999 {
return 0
}
return n
}
func splitGenres(s string) []string {
var out []string
seen := map[string]bool{}
for _, g := range splitAny(s, []string{";", "/", ","}) {
g = Clean(g)
if k := Key(g); k != "" && !seen[k] {
seen[k] = true
out = append(out, g)
}
}
return out
}
var leadingNumber = regexp.MustCompile(`^\s*(?:\d{1,2}[-.])?\d{1,3}\s*[-._)]*\s*`)
// titleFromFile: «03 - Song name.flac» → «Song name».
func titleFromFile(relPath string) string {
name := path.Base(relPath)
name = strings.TrimSuffix(name, path.Ext(name))
if t := Clean(leadingNumber.ReplaceAllString(name, "")); t != "" {
return t
}
return Clean(name)
}
@@ -0,0 +1,92 @@
package meta
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestKey(t *testing.T) {
assert.Equal(t, "the beatles", Key(" The_Beatles! "))
assert.Equal(t, Key("Ёлка"), Key("елка"))
assert.Equal(t, "ac dc", Key("AC/DC"))
assert.Equal(t, "", Key(" -- "))
}
func TestClean(t *testing.T) {
assert.Equal(t, "Hello World", Clean(" Hello__World "))
// «й» из двух кодовых точек (и + кратка) становится одной.
assert.Equal(t, "й", Clean("й"))
}
func TestSplitArtists(t *testing.T) {
known := func(k string) bool { return k == "simon garfunkel" }
names := func(cs []ArtistCredit) []string {
var out []string
for _, c := range cs {
out = append(out, c.Role+":"+c.Name)
}
return out
}
assert.Equal(t, []string{"main:Simon & Garfunkel"}, names(SplitArtists("Simon & Garfunkel", known)))
assert.Equal(t, []string{"main:Tom", "main:Jerry"}, names(SplitArtists("Tom & Jerry", known)))
assert.Equal(t, []string{"main:A", "main:B"}, names(SplitArtists("A; B", nil)))
assert.Equal(t, []string{"main:A", "feat:B", "feat:C"}, names(SplitArtists("A feat. B & C", nil)))
assert.Equal(t, []string{"main:A"}, names(SplitArtists("A / a", nil)), "duplicates by key are dropped")
assert.Empty(t, SplitArtists(" ", nil))
}
func TestPatterns(t *testing.T) {
patterns, err := CompilePatterns([]string{
`^(?P<skip>_incoming)/`,
`^(?P<artist>[^/]+)/(?P<year>\d{4}) - (?P<album>[^/]+)/(?:(?P<disc>\d)-)?(?P<track>\d{1,3}) - (?P<title>[^/]+)\.\w+$`,
`^(?P<genre>[^/]+)/(?P<artist>[^/]+)/(?P<album>.+?) \((?P<year>\d{4})\)/(?P<track>\d+)\. (?P<title>[^/]+)\.\w+$`,
})
require.NoError(t, err)
m := MatchPath(patterns, "Кино/1988 - Группа крови/1-01 - Группа крови.flac")
assert.Equal(t, 1, m.Index)
assert.Equal(t, map[string]string{
"artist": "Кино", "year": "1988", "album": "Группа крови", "disc": "1", "track": "01", "title": "Группа крови",
}, m.Fields)
m = MatchPath(patterns, "Rock/Queen/Jazz (1978)/03. Jealousy.mp3")
assert.Equal(t, 2, m.Index)
assert.Equal(t, "Rock", m.Fields["genre"])
assert.True(t, MatchPath(patterns, "_incoming/x.mp3").Skip)
assert.Equal(t, -1, MatchPath(patterns, "loose.mp3").Index)
}
func TestCompilePatternErrors(t *testing.T) {
for _, bad := range []string{`(?P<artist>[`, `^(?P<band>.+)$`, `^.+$`} {
_, err := CompilePatterns([]string{`^(?P<title>.+)$`, bad})
var pe *PatternError
require.ErrorAs(t, err, &pe, bad)
assert.Equal(t, 1, pe.Index)
}
}
func TestMerge(t *testing.T) {
tags := Fields{Title: "Tag title", Artist: "", Album: "Tag album", Year: "2019-05-01", Track: "3/12"}
path := Fields{Title: "Path title", Artist: "Path artist", Album: "Path album", Year: "2018", Track: "04"}
m := Merge(tags, path, true, "x/y.flac")
assert.Equal(t, "Tag title", m.Title)
assert.Equal(t, "Path artist", m.Artist, "empty tag falls back to the path")
assert.Equal(t, "Path artist", m.AlbumArtist, "album artist defaults to the artist")
assert.Equal(t, 2019, m.Year)
assert.Equal(t, 3, m.Track)
m = Merge(tags, path, false, "x/y.flac")
assert.Equal(t, "Path title", m.Title)
assert.Equal(t, 2018, m.Year)
assert.Equal(t, 4, m.Track)
m = Merge(Fields{Genre: "Rock; Pop/rock"}, Fields{}, true, "Some Folder/07 - My_Song.mp3")
assert.Equal(t, "My Song", m.Title)
assert.Equal(t, UnknownArtist, m.Artist)
assert.Equal(t, "Some Folder", m.Album)
assert.Equal(t, []string{"Rock", "Pop", "rock"}[:2], m.Genres[:2])
}
@@ -0,0 +1,112 @@
// Package meta — метаданные треков: нормализация имён, шаблоны путей, слияние с тегами
// (docs/library-and-storage.md).
package meta
import (
"strings"
"unicode"
"golang.org/x/text/unicode/norm"
)
// Clean приводит значение к виду для показа: NFC, `_` → пробел, без лишних пробелов.
func Clean(s string) string {
s = norm.NFC.String(s)
s = strings.ReplaceAll(s, "_", " ")
return strings.Join(strings.Fields(s), " ")
}
// Key — ключ сопоставления (name_norm): нижний регистр, ё → е, без пунктуации, пробелы схлопнуты.
// «The Beatles», «the beatles!» и «THE BEATLES» дают один ключ.
func Key(s string) string {
s = norm.NFKC.String(s)
var b strings.Builder
b.Grow(len(s))
space := false
for _, r := range strings.ToLower(s) {
switch {
case r == 'ё':
r = 'е'
case unicode.IsPunct(r) || unicode.IsSymbol(r) || unicode.IsSpace(r) || r == '_':
space = b.Len() > 0
continue
}
if space {
b.WriteByte(' ')
space = false
}
b.WriteRune(r)
}
return b.String()
}
// SortName — имя для сортировки исполнителей: без ведущего артикля «The ».
func SortName(name string) string {
if rest, ok := strings.CutPrefix(name, "The "); ok && rest != "" {
return rest
}
return name
}
// Разделители нескольких исполнителей в одном поле, кроме « & ».
var artistSeparators = []string{";", " / ", " feat. ", " feat ", " ft. ", " Feat. ", " Ft. ", " featuring "}
// ArtistCredit — исполнитель трека и его роль.
type ArtistCredit struct {
Name string
Role string // main | feat
}
// SplitArtists разбирает поле исполнителя на отдельных. « & » делит имя, только если составного
// исполнителя нет в библиотеке (known), — иначе «Simon & Garfunkel» распался бы на двоих.
// Всё после feat./ft. — приглашённые исполнители.
func SplitArtists(s string, known func(key string) bool) []ArtistCredit {
s = Clean(s)
if s == "" {
return nil
}
var out []ArtistCredit
seen := map[string]bool{}
add := func(name, role string) {
name = Clean(name)
k := Key(name)
if k == "" || seen[k] {
return
}
seen[k] = true
out = append(out, ArtistCredit{Name: name, Role: role})
}
main, feat := s, ""
for _, sep := range []string{" feat. ", " feat ", " ft. ", " Feat. ", " Ft. ", " featuring "} {
if i := strings.Index(main, sep); i >= 0 {
main, feat = main[:i], main[i+len(sep):]
break
}
}
for _, part := range splitAny(main, artistSeparators) {
if strings.Contains(part, " & ") && (known == nil || !known(Key(part))) {
for _, p := range strings.Split(part, " & ") {
add(p, "main")
}
continue
}
add(part, "main")
}
for _, part := range splitAny(feat, append(artistSeparators, " & ", ", ")) {
add(part, "feat")
}
return out
}
func splitAny(s string, seps []string) []string {
parts := []string{s}
for _, sep := range seps {
var next []string
for _, p := range parts {
next = append(next, strings.Split(p, sep)...)
}
parts = next
}
return parts
}
@@ -0,0 +1,92 @@
package meta
import (
"fmt"
"regexp"
"strings"
)
// MaxPatternLen — предел длины выражения (RE2 линеен, но огромные выражения бессмысленны).
const MaxPatternLen = 1000
// Группы шаблона и поля, которые они заполняют.
var knownGroups = map[string]bool{
"artist": true, "albumartist": true, "album": true, "year": true, "disc": true,
"track": true, "title": true, "genre": true, "skip": true,
}
// PatternError — ошибка компиляции выражения (для показа под полем ввода).
type PatternError struct {
Index int
Reason string
}
func (e *PatternError) Error() string { return fmt.Sprintf("pattern %d: %s", e.Index, e.Reason) }
// Pattern — скомпилированный шаблон пути.
type Pattern struct {
Source string
re *regexp.Regexp
}
// CompilePatterns проверяет и компилирует выражения по порядку.
func CompilePatterns(exprs []string) ([]Pattern, error) {
out := make([]Pattern, 0, len(exprs))
for i, e := range exprs {
if len(e) > MaxPatternLen {
return nil, &PatternError{Index: i, Reason: fmt.Sprintf("longer than %d characters", MaxPatternLen)}
}
re, err := regexp.Compile(e)
if err != nil {
return nil, &PatternError{Index: i, Reason: err.Error()}
}
named := 0
for _, name := range re.SubexpNames() {
if name == "" {
continue
}
if !knownGroups[name] {
return nil, &PatternError{Index: i, Reason: fmt.Sprintf("unknown group %q", name)}
}
named++
}
if named == 0 {
return nil, &PatternError{Index: i, Reason: "no named groups"}
}
out = append(out, Pattern{Source: e, re: re})
}
return out, nil
}
// PathMatch — результат применения шаблонов к пути.
type PathMatch struct {
Index int // номер совпавшего выражения, -1 — ни одно
Fields map[string]string // извлечённые группы (непустые)
Skip bool // совпала группа skip — файл игнорируется
}
// MatchPath применяет шаблоны к пути относительно корня источника (разделитель `/`).
// Побеждает первое совпавшее выражение.
func MatchPath(patterns []Pattern, relPath string) PathMatch {
for i, p := range patterns {
m := p.re.FindStringSubmatch(relPath)
if m == nil {
continue
}
// Выражение с группой skip описывает игнорируемые файлы: совпало — пропускаем.
if p.re.SubexpIndex("skip") >= 0 {
return PathMatch{Index: i, Skip: true}
}
fields := map[string]string{}
for gi, name := range p.re.SubexpNames() {
if name == "" {
continue
}
if v := strings.TrimSpace(m[gi]); v != "" {
fields[name] = v
}
}
return PathMatch{Index: i, Fields: fields}
}
return PathMatch{Index: -1}
}
+177
View File
@@ -0,0 +1,177 @@
// Package library — хранилища, сканер файлов и каталог (docs/library-and-storage.md).
package library
import (
"context"
"encoding/json"
"fmt"
"math"
"os/exec"
"path"
"strconv"
"strings"
"time"
"pipu/api/internal/library/meta"
)
// AudioExtensions — расширения, которые сканер считает аудио.
var AudioExtensions = map[string]bool{
".flac": true, ".mp3": true, ".m4a": true, ".aac": true, ".ogg": true, ".opus": true,
".wav": true, ".aiff": true, ".aif": true, ".wv": true, ".ape": true,
}
// IsAudio сообщает, похоже ли имя файла на аудио.
func IsAudio(name string) bool { return AudioExtensions[strings.ToLower(path.Ext(name))] }
// Probe — технические данные и теги файла.
type Probe struct {
Format string // по расширению: flac, mp3, ...
Codec string
Bitrate int // бит/с
SampleRate int
BitDepth int
Channels int
DurationMs int
Tags meta.Fields
}
// Lossless сообщает, без потерь ли кодек.
func (p Probe) Lossless() bool {
switch p.Codec {
case "flac", "alac", "wavpack", "ape", "tta":
return true
}
return strings.HasPrefix(p.Codec, "pcm_")
}
// Prober читает данные аудиофайла.
type Prober interface {
Probe(ctx context.Context, absPath string) (Probe, error)
}
// FFprobe — Prober на основе ffprobe (тот же бинарник нужен для транскодинга).
type FFprobe struct {
Bin string
Timeout time.Duration
}
type ffprobeOutput struct {
Format struct {
Duration string `json:"duration"`
BitRate string `json:"bit_rate"`
Tags map[string]string `json:"tags"`
} `json:"format"`
Streams []struct {
CodecType string `json:"codec_type"`
CodecName string `json:"codec_name"`
SampleRate string `json:"sample_rate"`
Channels int `json:"channels"`
BitsPerRawSample string `json:"bits_per_raw_sample"`
BitsPerSample int `json:"bits_per_sample"`
BitRate string `json:"bit_rate"`
Duration string `json:"duration"`
Tags map[string]string `json:"tags"`
} `json:"streams"`
}
// Probe реализует Prober. Путь передаётся отдельным аргументом, без shell.
func (f FFprobe) Probe(ctx context.Context, absPath string) (Probe, error) {
bin := f.Bin
if bin == "" {
bin = "ffprobe"
}
if f.Timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, f.Timeout)
defer cancel()
}
out, err := exec.CommandContext(ctx, bin, "-v", "error", "-print_format", "json",
"-show_format", "-show_streams", "--", absPath).Output()
if err != nil {
if ee, ok := err.(*exec.ExitError); ok && len(ee.Stderr) > 0 {
return Probe{}, fmt.Errorf("ffprobe: %s", strings.TrimSpace(string(ee.Stderr)))
}
return Probe{}, fmt.Errorf("ffprobe: %w", err)
}
return parseFFprobe(out, absPath)
}
func parseFFprobe(raw []byte, name string) (Probe, error) {
var o ffprobeOutput
if err := json.Unmarshal(raw, &o); err != nil {
return Probe{}, fmt.Errorf("ffprobe: decode: %w", err)
}
p := Probe{Format: strings.TrimPrefix(strings.ToLower(path.Ext(name)), ".")}
tags := map[string]string{}
addTags := func(m map[string]string) {
for k, v := range m {
k = strings.ToLower(k)
if _, ok := tags[k]; !ok && strings.TrimSpace(v) != "" {
tags[k] = v
}
}
}
addTags(o.Format.Tags)
found := false
for _, s := range o.Streams {
if s.CodecType != "audio" || found {
continue
}
found = true
p.Codec = s.CodecName
p.SampleRate = atoi(s.SampleRate)
p.Channels = s.Channels
p.BitDepth = atoi(s.BitsPerRawSample)
if p.BitDepth == 0 {
p.BitDepth = s.BitsPerSample
}
p.Bitrate = atoi(s.BitRate)
if p.DurationMs == 0 {
p.DurationMs = seconds(s.Duration)
}
// Ogg / Opus хранят теги в потоке, а не в контейнере.
addTags(s.Tags)
}
if !found {
return Probe{}, fmt.Errorf("ffprobe: no audio stream")
}
if d := seconds(o.Format.Duration); d > 0 {
p.DurationMs = d
}
if p.Bitrate == 0 {
p.Bitrate = atoi(o.Format.BitRate)
}
first := func(keys ...string) string {
for _, k := range keys {
if v := tags[k]; v != "" {
return v
}
}
return ""
}
p.Tags = meta.Fields{
Title: first("title"),
Artist: first("artist"),
AlbumArtist: first("album_artist", "albumartist", "album artist", "band"),
Album: first("album"),
Genre: first("genre"),
Year: first("date", "year", "originaldate", "tdrc"),
Track: first("track", "tracknumber"),
Disc: first("disc", "discnumber"),
}
return p, nil
}
func atoi(s string) int {
n, _ := strconv.Atoi(strings.TrimSpace(s))
return n
}
func seconds(s string) int {
f, err := strconv.ParseFloat(strings.TrimSpace(s), 64)
if err != nil || f <= 0 || math.IsInf(f, 0) {
return 0
}
return int(math.Round(f * 1000))
}
+504
View File
@@ -0,0 +1,504 @@
package library
import (
"context"
"encoding/binary"
"errors"
"fmt"
"io"
"io/fs"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/cespare/xxhash/v2"
"github.com/google/uuid"
"pipu/api/internal/library/meta"
)
// UnknownAlbum — альбом файла без тегов в корне источника.
const UnknownAlbum = "Unknown album"
// MissingRetention — сколько пропавший файл хранится, прежде чем трек удалится.
const MissingRetention = 30 * 24 * time.Hour
// Progress — состояние скана одного источника (для админки).
type Progress struct {
SourceID uuid.UUID
Running bool
Full bool
StartedAt time.Time
FinishedAt *time.Time
Found int // аудиофайлов найдено обходом
ToProcess int // требуют чтения
Processed int
Added int
Updated int
Moved int
Missing int
Errors int
Error string // фатальная ошибка скана
}
// ProgressBoard хранит прогресс сканов в памяти процесса.
type ProgressBoard struct {
mu sync.Mutex
m map[uuid.UUID]*Progress
}
// NewProgressBoard создаёт пустую доску.
func NewProgressBoard() *ProgressBoard { return &ProgressBoard{m: map[uuid.UUID]*Progress{}} }
// Get — копия прогресса источника.
func (b *ProgressBoard) Get(id uuid.UUID) (Progress, bool) {
b.mu.Lock()
defer b.mu.Unlock()
p, ok := b.m[id]
if !ok {
return Progress{}, false
}
return *p, true
}
// All — копии прогресса всех источников.
func (b *ProgressBoard) All() []Progress {
b.mu.Lock()
defer b.mu.Unlock()
out := make([]Progress, 0, len(b.m))
for _, p := range b.m {
out = append(out, *p)
}
return out
}
// start регистрирует новый скан; false — скан этого источника уже идёт.
func (b *ProgressBoard) start(id uuid.UUID, full bool, now time.Time) bool {
b.mu.Lock()
defer b.mu.Unlock()
if p, ok := b.m[id]; ok && p.Running {
return false
}
b.m[id] = &Progress{SourceID: id, Running: true, Full: full, StartedAt: now}
return true
}
func (b *ProgressBoard) update(id uuid.UUID, fn func(*Progress)) {
b.mu.Lock()
defer b.mu.Unlock()
if p, ok := b.m[id]; ok {
fn(p)
}
}
// Scanner сканирует источники и обновляет каталог.
type Scanner struct {
store *Store
prober Prober
log *slog.Logger
board *ProgressBoard
workers int
now func() time.Time
// AfterScan вызывается после каждого успешного скана (переиндексация поиска).
AfterScan func(ctx context.Context)
}
// NewScanner создаёт сканер с 8 воркерами чтения.
func NewScanner(store *Store, prober Prober, board *ProgressBoard, log *slog.Logger) *Scanner {
return &Scanner{store: store, prober: prober, log: log, board: board, workers: 8, now: func() time.Time { return time.Now().UTC() }}
}
// walked — аудиофайл, найденный обходом.
type walked struct {
rel string
size int64
mtime time.Time
hash []byte
}
// job — файл, который нужно прочитать.
type job struct {
walked
fileID uuid.UUID // Nil — новый
oldTrack *uuid.UUID // трек, к которому файл был привязан
}
type probed struct {
job
probe Probe
err error
}
// ErrScanRunning — скан источника уже идёт.
var ErrScanRunning = errors.New("library: scan already running")
// Start запускает скан в фоне. root — каталог источника в файловой системе api.
func (sc *Scanner) Start(src Source, root string, full bool) error {
if !sc.board.start(src.ID, full, sc.now()) {
return ErrScanRunning
}
go func() {
ctx := context.Background()
err := sc.run(ctx, src, root, full)
finished := sc.now()
sc.board.update(src.ID, func(p *Progress) {
p.Running, p.FinishedAt = false, &finished
if err != nil {
p.Error = err.Error()
}
})
if err != nil {
sc.log.Error("scan failed", "source_id", src.ID, "err", err)
return
}
if sc.AfterScan != nil {
sc.AfterScan(ctx)
}
}()
return nil
}
func (sc *Scanner) run(ctx context.Context, src Source, root string, full bool) error {
start := sc.now()
rows, err := sc.store.Patterns(ctx, src.ID)
if err != nil {
return err
}
var exprs []string
for _, r := range rows {
if r.IsEnabled {
exprs = append(exprs, r.Regex)
}
}
patterns, err := meta.CompilePatterns(exprs)
if err != nil {
return fmt.Errorf("library: patterns: %w", err)
}
existing, err := sc.store.Files(ctx, src.ID)
if err != nil {
return err
}
found, err := walk(root, func(rel string) bool { return meta.MatchPath(patterns, rel).Skip })
if err != nil {
return err
}
sc.board.update(src.ID, func(p *Progress) { p.Found = len(found) })
var (
jobs []job
newFiles []walked
seen = make(map[string]bool, len(found))
)
for _, w := range found {
seen[w.rel] = true
f, ok := existing[w.rel]
switch {
case !ok:
newFiles = append(newFiles, w)
case !full && f.Size == w.size && f.Mtime.Equal(w.mtime):
if f.Missing {
if err := sc.store.SetPresent(ctx, f.ID); err != nil {
return err
}
}
default:
jobs = append(jobs, job{walked: w, fileID: f.ID, oldTrack: f.TrackID})
}
}
var gone []FileRow
for rel, f := range existing {
if !seen[rel] {
gone = append(gone, f)
}
}
// Переименования: новый путь с тем же содержимым, что у пропавшего файла.
if len(gone) > 0 {
for i := range newFiles {
if h, err := contentHash(filepath.Join(root, filepath.FromSlash(newFiles[i].rel)), newFiles[i].size); err == nil {
newFiles[i].hash = h
}
}
}
moves, rest, stillGone := matchRenames(newFiles, gone)
for _, mv := range moves {
if err := sc.store.MoveFile(ctx, mv.from.ID, mv.to.rel, mv.to.size, mv.to.mtime); err != nil {
return err
}
}
for _, w := range rest {
jobs = append(jobs, job{walked: w})
}
sc.board.update(src.ID, func(p *Progress) { p.ToProcess, p.Moved = len(jobs), len(moves) })
if err := sc.process(ctx, src, root, patterns, jobs); err != nil {
return err
}
var goneIDs []uuid.UUID
affected := map[uuid.UUID]bool{}
for _, f := range stillGone {
goneIDs = append(goneIDs, f.ID)
if f.TrackID != nil {
affected[*f.TrackID] = true
}
}
if err := sc.store.MarkMissing(ctx, goneIDs, start); err != nil {
return err
}
if _, err := sc.store.PurgeMissing(ctx, src.ID, start.Add(-MissingRetention)); err != nil {
return err
}
for id := range affected {
if err := sc.store.RefreshPrimaryFile(ctx, id); err != nil {
return err
}
}
sc.board.update(src.ID, func(p *Progress) { p.Missing = len(goneIDs) })
if err := sc.store.CleanupOrphans(ctx); err != nil {
return err
}
return sc.store.SetLastScan(ctx, src.ID, sc.now())
}
// process читает файлы параллельно, а каталог обновляет один писатель —
// так дубли и новые исполнители не создаются дважды в гонке.
func (sc *Scanner) process(ctx context.Context, src Source, root string, patterns []meta.Pattern, jobs []job) error {
in := make(chan job)
out := make(chan probed)
var wg sync.WaitGroup
for range max(1, sc.workers) {
wg.Add(1)
go func() {
defer wg.Done()
for j := range in {
abs := filepath.Join(root, filepath.FromSlash(j.rel))
p, err := sc.prober.Probe(ctx, abs)
if j.hash == nil {
j.hash, _ = contentHash(abs, j.size)
}
out <- probed{job: j, probe: p, err: err}
}
}()
}
go func() {
for _, j := range jobs {
in <- j
}
close(in)
wg.Wait()
close(out)
}()
var firstErr error
for r := range out {
if firstErr != nil {
continue // дочитываем канал, чтобы воркеры завершились
}
added, err := sc.apply(ctx, src, patterns, r)
if err != nil {
firstErr = err
continue
}
sc.board.update(src.ID, func(p *Progress) {
p.Processed++
switch {
case r.err != nil:
p.Errors++
case added:
p.Added++
default:
p.Updated++
}
})
}
return firstErr
}
// apply записывает прочитанный файл в каталог. added — файл новый.
func (sc *Scanner) apply(ctx context.Context, src Source, patterns []meta.Pattern, r probed) (bool, error) {
data := FileData{RelPath: r.rel, Size: r.size, Mtime: r.mtime, ContentHash: r.hash, Probe: r.probe}
added := r.fileID == uuid.Nil
if r.err != nil {
data.Probe = Probe{Format: strings.TrimPrefix(strings.ToLower(filepath.Ext(r.rel)), ".")}
data.ScanError = r.err.Error()
if _, err := sc.store.SaveFile(ctx, src.ID, r.fileID, data, nil); err != nil {
return false, err
}
if r.oldTrack != nil {
return added, sc.store.RefreshPrimaryFile(ctx, *r.oldTrack)
}
return added, nil
}
pm := meta.MatchPath(patterns, r.rel)
m := meta.Merge(r.probe.Tags, meta.FromPath(pm.Fields), src.TagsFirst, r.rel)
if m.Album == "" {
m.Album = UnknownAlbum
}
albumArtist, err := sc.store.EnsureArtist(ctx, m.AlbumArtist)
if err != nil {
return false, err
}
albumID, err := sc.store.EnsureAlbum(ctx, albumArtist, m.Album, m.Year)
if err != nil {
return false, err
}
in := TrackInput{
AlbumID: albumID, Title: m.Title, Disc: m.Disc, Track: m.Track, Year: m.Year, DurationMs: r.probe.DurationMs,
}
trackID, dup, err := sc.store.FindTrack(ctx, in)
if err != nil {
return false, err
}
if !dup {
// Перечитанный файл сохраняет свой трек (лайки, история); новый — получает новый.
if r.oldTrack != nil {
trackID = *r.oldTrack
}
if trackID, err = sc.store.SaveTrack(ctx, trackID, in); err != nil {
return false, err
}
credits := meta.SplitArtists(m.Artist, func(k string) bool { return sc.store.ArtistKeyExists(ctx, k) })
ids := make([]uuid.UUID, 0, len(credits))
roles := make([]string, 0, len(credits))
for _, c := range credits {
id, err := sc.store.EnsureArtist(ctx, c.Name)
if err != nil {
return false, err
}
ids, roles = append(ids, id), append(roles, c.Role)
}
if err := sc.store.SetTrackArtists(ctx, trackID, ids, roles); err != nil {
return false, err
}
if err := sc.store.SetTrackGenres(ctx, trackID, m.Genres); err != nil {
return false, err
}
}
if _, err := sc.store.SaveFile(ctx, src.ID, r.fileID, data, &trackID); err != nil {
return false, err
}
if err := sc.store.RefreshPrimaryFile(ctx, trackID); err != nil {
return false, err
}
if r.oldTrack != nil && *r.oldTrack != trackID {
if err := sc.store.RefreshPrimaryFile(ctx, *r.oldTrack); err != nil {
return false, err
}
}
return added, nil
}
// walk обходит дерево источника и возвращает аудиофайлы. Скрытые файлы и каталоги пропускаются;
// skip — пути, которые шаблоны велят игнорировать. Ошибки отдельных каталогов не прерывают обход.
func walk(root string, skip func(rel string) bool) ([]walked, error) {
if _, err := os.Stat(root); err != nil {
return nil, fmt.Errorf("library: source root: %w", err)
}
var out []walked
err := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
if err != nil {
if p == root {
return err
}
return nil
}
name := d.Name()
if p != root && strings.HasPrefix(name, ".") {
if d.IsDir() {
return filepath.SkipDir
}
return nil
}
if d.IsDir() || !IsAudio(name) {
return nil
}
rel, err := filepath.Rel(root, p)
if err != nil {
return nil
}
rel = filepath.ToSlash(rel)
if skip(rel) {
return nil
}
info, err := d.Info()
if err != nil {
return nil
}
// PostgreSQL хранит время с точностью до микросекунд.
out = append(out, walked{rel: rel, size: info.Size(), mtime: info.ModTime().UTC().Truncate(time.Microsecond)})
return nil
})
if err != nil {
return nil, fmt.Errorf("library: walk: %w", err)
}
return out, nil
}
const hashChunk = 1 << 20
// contentHash — xxhash64 от размера, первого и последнего мегабайта: быстро даже по NFS
// и не меняется при переименовании.
func contentHash(abs string, size int64) ([]byte, error) {
f, err := os.Open(abs)
if err != nil {
return nil, err
}
defer func() { _ = f.Close() }()
h := xxhash.New()
var sz [8]byte
binary.LittleEndian.PutUint64(sz[:], uint64(size))
_, _ = h.Write(sz[:])
if _, err := io.CopyN(h, f, hashChunk); err != nil && !errors.Is(err, io.EOF) {
return nil, err
}
if size > 2*hashChunk {
if _, err := f.Seek(-hashChunk, io.SeekEnd); err != nil {
return nil, err
}
if _, err := io.CopyN(h, f, hashChunk); err != nil && !errors.Is(err, io.EOF) {
return nil, err
}
}
return h.Sum(nil), nil
}
type move struct {
from FileRow
to walked
}
// matchRenames сопоставляет новые пути с пропавшими файлами по содержимому (хэш и размер).
// Каждый пропавший файл используется не больше одного раза.
func matchRenames(newFiles []walked, gone []FileRow) (moves []move, rest []walked, stillGone []FileRow) {
byHash := map[string][]int{}
for i, f := range gone {
if len(f.ContentHash) > 0 {
k := fmt.Sprintf("%x:%d", f.ContentHash, f.Size)
byHash[k] = append(byHash[k], i)
}
}
used := make([]bool, len(gone))
for _, w := range newFiles {
k := fmt.Sprintf("%x:%d", w.hash, w.size)
if len(w.hash) > 0 && len(byHash[k]) > 0 {
i := byHash[k][0]
byHash[k] = byHash[k][1:]
used[i] = true
moves = append(moves, move{from: gone[i], to: w})
continue
}
rest = append(rest, w)
}
for i, f := range gone {
if !used[i] {
stillGone = append(stillGone, f)
}
}
return moves, rest, stillGone
}
@@ -0,0 +1,99 @@
package library
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestMatchRenames(t *testing.T) {
a := FileRow{ID: uuid.New(), RelPath: "old/a.flac", Size: 10, ContentHash: []byte{1}}
b := FileRow{ID: uuid.New(), RelPath: "old/b.flac", Size: 10, ContentHash: []byte{2}}
noHash := FileRow{ID: uuid.New(), RelPath: "old/c.flac", Size: 10}
newFiles := []walked{
{rel: "new/a.flac", size: 10, hash: []byte{1}},
{rel: "new/a-copy.flac", size: 10, hash: []byte{1}}, // второй файл с тем же содержимым — новый
{rel: "new/x.flac", size: 11, hash: []byte{2}}, // другой размер — не переименование
{rel: "new/y.flac", size: 10},
}
moves, rest, gone := matchRenames(newFiles, []FileRow{a, b, noHash})
require.Len(t, moves, 1)
assert.Equal(t, a.ID, moves[0].from.ID)
assert.Equal(t, "new/a.flac", moves[0].to.rel)
assert.Equal(t, []string{"new/a-copy.flac", "new/x.flac", "new/y.flac"}, rels(rest))
assert.Equal(t, []uuid.UUID{b.ID, noHash.ID}, []uuid.UUID{gone[0].ID, gone[1].ID})
}
func rels(ws []walked) []string {
var out []string
for _, w := range ws {
out = append(out, w.rel)
}
return out
}
func TestWalkAndHash(t *testing.T) {
root := t.TempDir()
write := func(rel string, size int) {
p := filepath.Join(root, filepath.FromSlash(rel))
require.NoError(t, os.MkdirAll(filepath.Dir(p), 0o755))
data := make([]byte, size)
for i := range data {
data[i] = byte(i)
}
require.NoError(t, os.WriteFile(p, data, 0o644))
}
write("Artist/Album/01 - Song.FLAC", 100)
write("Artist/Album/cover.jpg", 10)
write("Artist/.hidden/02.mp3", 10)
write("_incoming/new.mp3", 10)
write("big.mp3", 3<<20)
found, err := walk(root, func(rel string) bool { return rel == "_incoming/new.mp3" })
require.NoError(t, err)
assert.ElementsMatch(t, []string{"Artist/Album/01 - Song.FLAC", "big.mp3"}, rels(found))
for _, w := range found {
assert.Equal(t, w.mtime, w.mtime.Truncate(time.Microsecond))
}
h1, err := contentHash(filepath.Join(root, "big.mp3"), 3<<20)
require.NoError(t, err)
require.NoError(t, os.Rename(filepath.Join(root, "big.mp3"), filepath.Join(root, "moved.mp3")))
h2, err := contentHash(filepath.Join(root, "moved.mp3"), 3<<20)
require.NoError(t, err)
assert.Equal(t, h1, h2, "hash survives a rename")
_, err = walk(filepath.Join(root, "nope"), func(string) bool { return false })
assert.Error(t, err)
}
func TestParseFFprobe(t *testing.T) {
raw := []byte(`{
"streams": [
{"codec_type": "video", "codec_name": "mjpeg"},
{"codec_type": "audio", "codec_name": "opus", "sample_rate": "48000", "channels": 2,
"tags": {"TITLE": "Song", "ARTIST": "Band feat. Guest", "ALBUM": "Record", "DATE": "2020", "track": "3/10"}}
],
"format": {"duration": "215.004", "bit_rate": "128000", "tags": {"album_artist": "Band"}}
}`)
p, err := parseFFprobe(raw, "/x/Song.OPUS")
require.NoError(t, err)
assert.Equal(t, "opus", p.Format)
assert.Equal(t, "opus", p.Codec)
assert.Equal(t, 215004, p.DurationMs)
assert.Equal(t, 128000, p.Bitrate)
assert.Equal(t, 48000, p.SampleRate)
assert.Equal(t, "Song", p.Tags.Title)
assert.Equal(t, "Band", p.Tags.AlbumArtist)
assert.Equal(t, "3/10", p.Tags.Track)
assert.False(t, p.Lossless())
_, err = parseFFprobe([]byte(`{"streams": [], "format": {}}`), "x.mp3")
assert.Error(t, err)
}
+698
View File
@@ -0,0 +1,698 @@
package library
import (
"context"
"errors"
"fmt"
"log/slog"
"math/rand/v2"
"net/http"
"os"
"path"
"path/filepath"
"slices"
"strings"
"sync"
"time"
"unicode/utf8"
"github.com/google/uuid"
"pipu/api/internal/auth"
"pipu/api/internal/library/meta"
"pipu/api/internal/platform/apperr"
"pipu/api/internal/storageagent"
)
// Коды ошибок библиотеки.
const (
CodeSourceNotFound = "library.source_not_found"
CodeScanRunning = "library.scan_running"
CodeSourceOffline = "library.source_offline"
CodePatternInvalid = "validation.pattern_invalid"
CodeNFSInvalid = "validation.nfs_invalid"
CodeLocalPathInvalid = "validation.local_path_invalid"
CodeNameInvalid = "validation.name_invalid"
CodeIntervalInvalid = "validation.scan_interval_invalid"
CodeNotFound = "catalog.not_found"
)
// Config — пути к хранилищам в файловой системе api.
type Config struct {
LibraryRoot string // точки монтирования NFS: <root>/<source_id>
HostRoot string // LIBRARY_HOST внутри контейнера, для local-источников
}
// Service управляет хранилищами и сканами.
type Service struct {
store *Store
scanner *Scanner
board *ProgressBoard
agent Agent
prober Prober
cfg Config
log *slog.Logger
mu sync.Mutex // сериализует переподключение одного и того же источника
}
// NewService создаёт сервис.
func NewService(store *Store, scanner *Scanner, board *ProgressBoard, agent Agent, prober Prober, cfg Config, log *slog.Logger) *Service {
return &Service{store: store, scanner: scanner, board: board, agent: agent, prober: prober, cfg: cfg, log: log}
}
func forbidden() error { return apperr.Forbidden(auth.CodeForbidden, "not allowed") }
func requirePerm(ctx context.Context, perm auth.Permission) error {
p, ok := auth.PrincipalFrom(ctx)
if !ok {
return apperr.Unauthorized(auth.CodeUnauthorized, "not authenticated")
}
if !auth.Can(p.Role, perm) {
return forbidden()
}
return nil
}
func notFound(err error) error {
if errors.Is(err, ErrNotFound) {
return apperr.NotFound(CodeSourceNotFound, "source not found")
}
return err
}
// Root — каталог источника в файловой системе api.
func (s *Service) Root(src Source) (string, error) {
switch src.Type {
case SourceLocal:
rel := ""
if src.LocalPath != nil {
rel = *src.LocalPath
}
return s.hostPath(rel)
case SourceNFS:
return filepath.Join(s.cfg.LibraryRoot, src.ID.String()), nil
}
return "", fmt.Errorf("library: unknown source type %q", src.Type)
}
// hostPath — путь внутри LIBRARY_HOST; выйти за его пределы нельзя.
func (s *Service) hostPath(rel string) (string, error) {
rel = strings.Trim(strings.ReplaceAll(rel, "\\", "/"), "/")
clean := path.Clean("/" + rel)
if clean != "/"+rel && rel != "" || strings.Contains(rel, "..") {
return "", apperr.Validation("local_path", CodeLocalPathInvalid, "path must be relative without ..", nil)
}
return filepath.Join(s.cfg.HostRoot, filepath.FromSlash(strings.TrimPrefix(clean, "/"))), nil
}
// ListSources — хранилища с прогрессом сканов.
func (s *Service) ListSources(ctx context.Context) ([]Source, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return nil, err
}
return s.store.ListSources(ctx)
}
// GetSource — хранилище по id.
func (s *Service) GetSource(ctx context.Context, id uuid.UUID) (Source, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return Source{}, err
}
src, err := s.store.GetSource(ctx, id)
return src, notFound(err)
}
// Progress — прогресс скана источника, если он был.
func (s *Service) Progress(id uuid.UUID) (Progress, bool) { return s.board.Get(id) }
// SourceInput — поля создания / изменения хранилища. nil — не задано.
type SourceInput struct {
Name *string
Type SourceType
NFSServer *string
NFSExport *string
NFSOptions *string
LocalPath *string
TagsFirst *bool
ScanIntervalMinutes *int // 0 — выключить
Enabled *bool
}
func (s *Service) apply(src *Source, in SourceInput) error {
if in.Name != nil {
name := strings.TrimSpace(*in.Name)
if name == "" || utf8.RuneCountInString(name) > 100 {
return apperr.Validation("name", CodeNameInvalid, "name must be 1-100 characters", map[string]any{"max": 100})
}
src.Name = name
}
if in.TagsFirst != nil {
src.TagsFirst = *in.TagsFirst
}
if in.ScanIntervalMinutes != nil {
switch m := *in.ScanIntervalMinutes; {
case m == 0:
src.ScanInterval = nil
case m < 5 || m > 60*24*30:
return apperr.Validation("scan_interval_minutes", CodeIntervalInvalid, "interval must be 5 minutes .. 30 days",
map[string]any{"min": 5})
default:
d := time.Duration(m) * time.Minute
src.ScanInterval = &d
}
}
switch src.Type {
case SourceLocal:
if in.LocalPath != nil {
rel := strings.Trim(strings.ReplaceAll(*in.LocalPath, "\\", "/"), "/")
if _, err := s.hostPath(rel); err != nil {
return err
}
src.LocalPath = &rel
}
if src.LocalPath == nil {
empty := ""
src.LocalPath = &empty
}
case SourceNFS:
if in.NFSServer != nil {
src.NFSServer = in.NFSServer
}
if in.NFSExport != nil {
src.NFSExport = in.NFSExport
}
if in.NFSOptions != nil {
src.NFSOptions = *in.NFSOptions
}
spec, err := nfsSpec(*src).Normalize(false)
if err != nil {
var ve *storageagent.ValidationError
if errors.As(err, &ve) {
return apperr.Validation(ve.Field, CodeNFSInvalid, ve.Reason, nil)
}
return err
}
src.NFSServer, src.NFSExport, src.NFSOptions = &spec.Server, &spec.Export, spec.Options
default:
return apperr.Validation("type", "validation.source_type_invalid", "unknown source type", nil)
}
return nil
}
func nfsSpec(src Source) storageagent.NFSSpec {
var spec storageagent.NFSSpec
if src.NFSServer != nil {
spec.Server = *src.NFSServer
}
if src.NFSExport != nil {
spec.Export = *src.NFSExport
}
spec.Options = src.NFSOptions
return spec
}
// CreateSource сохраняет хранилище и сразу пытается подключить его.
func (s *Service) CreateSource(ctx context.Context, in SourceInput) (Source, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return Source{}, err
}
src := Source{ID: newID(), Type: in.Type, TagsFirst: true, Status: "pending", NFSOptions: "ro,nfsvers=4.1,soft,timeo=150,retrans=3"}
if in.Name == nil {
in.Name = new(string)
}
if err := s.apply(&src, in); err != nil {
return Source{}, err
}
src, err := s.store.SaveSource(ctx, src)
if err != nil {
return Source{}, err
}
return s.connect(ctx, src)
}
// UpdateSource меняет хранилище; смена адреса или папки переподключает его.
func (s *Service) UpdateSource(ctx context.Context, id uuid.UUID, in SourceInput) (Source, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return Source{}, err
}
src, err := s.store.GetSource(ctx, id)
if err != nil {
return Source{}, notFound(err)
}
before := src
in.Type = src.Type
if err := s.apply(&src, in); err != nil {
return Source{}, err
}
if src, err = s.store.SaveSource(ctx, src); err != nil {
return Source{}, err
}
location := func(x Source) string {
return fmt.Sprint(x.LocalPath != nil && *x.LocalPath != "", deref(x.LocalPath), deref(x.NFSServer), deref(x.NFSExport), x.NFSOptions)
}
switch {
case in.Enabled != nil && !*in.Enabled:
if src.Type == SourceNFS {
_ = s.agent.Unmount(ctx, src.ID)
}
if err := s.store.SetSourceStatus(ctx, src.ID, "disabled", nil); err != nil {
return Source{}, err
}
return s.store.GetSource(ctx, src.ID)
case in.Enabled != nil && *in.Enabled, location(before) != location(src), src.Status == "error":
return s.connect(ctx, src)
}
return src, nil
}
func deref(p *string) string {
if p == nil {
return ""
}
return *p
}
// connect подключает хранилище и записывает статус: local — проверка каталога, nfs — монтирование агентом.
func (s *Service) connect(ctx context.Context, src Source) (Source, error) {
s.mu.Lock()
defer s.mu.Unlock()
status, message := s.check(ctx, src, true)
if err := s.store.SetSourceStatus(ctx, src.ID, status, message); err != nil {
return Source{}, err
}
return s.store.GetSource(ctx, src.ID)
}
// check проверяет доступность; mount — смонтировать NFS (иначе только проверить точку).
func (s *Service) check(ctx context.Context, src Source, mount bool) (string, *string) {
msg := func(m string) *string { return &m }
switch src.Type {
case SourceLocal:
root, err := s.Root(src)
if err != nil {
return "error", msg(err.Error())
}
info, err := os.Stat(root)
if err != nil || !info.IsDir() {
return "error", msg("folder not found: " + deref(src.LocalPath))
}
return "mounted", nil
case SourceNFS:
if !mount {
root, _ := s.Root(src)
if _, err := os.Stat(root); err != nil {
return "error", msg(err.Error())
}
return "mounted", nil
}
res, err := s.agent.Mount(ctx, src.ID, nfsSpec(src))
if err != nil {
return "error", msg(err.Error())
}
if res.Status != "mounted" {
return "error", msg(res.Message)
}
return "mounted", nil
}
return "error", msg("unknown type")
}
// DeleteSource отключает хранилище и удаляет его файлы из каталога.
func (s *Service) DeleteSource(ctx context.Context, id uuid.UUID) error {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return err
}
src, err := s.store.GetSource(ctx, id)
if err != nil {
return notFound(err)
}
if p, ok := s.board.Get(id); ok && p.Running {
return apperr.Conflict(CodeScanRunning, "scan is running")
}
if src.Type == SourceNFS {
if err := s.agent.Unmount(ctx, id); err != nil {
s.log.WarnContext(ctx, "unmount failed", "source_id", id, "err", err)
}
}
if err := s.store.DeleteSource(ctx, id); err != nil {
return notFound(err)
}
return s.store.CleanupOrphans(ctx)
}
// TestResult — итог проверки подключения.
type TestResult struct {
OK bool
Message string
AudioFiles int
}
// TestSource проверяет доступ к хранилищу и считает аудиофайлы (до 1000).
func (s *Service) TestSource(ctx context.Context, id uuid.UUID) (TestResult, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return TestResult{}, err
}
src, err := s.store.GetSource(ctx, id)
if err != nil {
return TestResult{}, notFound(err)
}
if src.Type == SourceNFS {
res, err := s.agent.Test(ctx, nfsSpec(src))
if err != nil {
return TestResult{Message: err.Error()}, nil
}
return TestResult{OK: res.Status == "mounted", Message: res.Message, AudioFiles: res.AudioFiles}, nil
}
root, err := s.Root(src)
if err != nil {
return TestResult{Message: err.Error()}, nil
}
n := 0
err = filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if !d.IsDir() && IsAudio(d.Name()) {
n++
if n >= 1000 {
return filepath.SkipAll
}
}
return nil
})
if err != nil {
return TestResult{Message: err.Error(), AudioFiles: n}, nil
}
return TestResult{OK: true, Message: fmt.Sprintf("found %d audio files", n), AudioFiles: n}, nil
}
// Patterns — шаблоны путей источника.
func (s *Service) Patterns(ctx context.Context, id uuid.UUID) ([]PatternRow, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return nil, err
}
if _, err := s.store.GetSource(ctx, id); err != nil {
return nil, notFound(err)
}
return s.store.Patterns(ctx, id)
}
func compile(patterns []PatternRow) ([]meta.Pattern, error) {
var exprs []string
idx := []int{}
for i, p := range patterns {
if p.IsEnabled {
exprs = append(exprs, p.Regex)
idx = append(idx, i)
}
}
compiled, err := meta.CompilePatterns(exprs)
var pe *meta.PatternError
if errors.As(err, &pe) {
at := idx[pe.Index]
return nil, &apperr.Error{
Status: http.StatusUnprocessableEntity, Code: CodePatternInvalid, Detail: pe.Reason,
Params: map[string]any{"index": at, "reason": pe.Reason},
Fields: []apperr.FieldError{{Field: fmt.Sprintf("patterns[%d].regex", at), Code: CodePatternInvalid}},
}
}
return compiled, err
}
// PutPatterns проверяет и сохраняет шаблоны.
func (s *Service) PutPatterns(ctx context.Context, id uuid.UUID, patterns []PatternRow) ([]PatternRow, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return nil, err
}
if _, err := s.store.GetSource(ctx, id); err != nil {
return nil, notFound(err)
}
if _, err := compile(patterns); err != nil {
return nil, err
}
if err := s.store.ReplacePatterns(ctx, id, patterns); err != nil {
return nil, err
}
return s.store.Patterns(ctx, id)
}
// PreviewItem — разбор одного файла.
type PreviewItem struct {
RelPath string
PatternIndex *int
Skipped bool
PathFields map[string]string
Tags map[string]string
Result meta.Meta
Error string
}
// Preview применяет шаблоны к выборке файлов без сохранения: из каталога, а до первого скана — из обхода.
func (s *Service) Preview(ctx context.Context, id uuid.UUID, patterns []PatternRow, sample int) ([]PreviewItem, int, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return nil, 0, err
}
src, err := s.store.GetSource(ctx, id)
if err != nil {
return nil, 0, notFound(err)
}
compiled, err := compile(patterns)
if err != nil {
return nil, 0, err
}
sample = min(max(sample, 1), 200)
root, err := s.Root(src)
if err != nil {
return nil, 0, err
}
paths, err := s.store.SampleFiles(ctx, id, sample)
if err != nil {
return nil, 0, err
}
if len(paths) == 0 {
found, err := walk(root, func(string) bool { return false })
if err != nil {
return nil, 0, apperr.New(http.StatusConflict, CodeSourceOffline, err.Error())
}
rand.Shuffle(len(found), func(i, j int) { found[i], found[j] = found[j], found[i] })
for _, w := range found[:min(sample, len(found))] {
paths = append(paths, w.rel)
}
}
slices.Sort(paths)
items := make([]PreviewItem, len(paths))
var wg sync.WaitGroup
sem := make(chan struct{}, 8)
for i, rel := range paths {
wg.Add(1)
go func() {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
items[i] = s.previewOne(ctx, root, rel, compiled, src.TagsFirst)
}()
}
wg.Wait()
unmatched := 0
for _, it := range items {
if it.PatternIndex == nil {
unmatched++
}
}
return items, unmatched, nil
}
func (s *Service) previewOne(ctx context.Context, root, rel string, patterns []meta.Pattern, tagsFirst bool) PreviewItem {
it := PreviewItem{RelPath: rel, PathFields: map[string]string{}, Tags: map[string]string{}}
m := meta.MatchPath(patterns, rel)
if m.Index >= 0 {
idx := m.Index
it.PatternIndex = &idx
}
it.Skipped = m.Skip
for k, v := range m.Fields {
it.PathFields[k] = v
}
p, err := s.prober.Probe(ctx, filepath.Join(root, filepath.FromSlash(rel)))
if err != nil {
it.Error = err.Error()
}
for k, v := range map[string]string{
"artist": p.Tags.Artist, "albumartist": p.Tags.AlbumArtist, "album": p.Tags.Album, "title": p.Tags.Title,
"genre": p.Tags.Genre, "year": p.Tags.Year, "track": p.Tags.Track, "disc": p.Tags.Disc,
} {
if v != "" {
it.Tags[k] = v
}
}
it.Result = meta.Merge(p.Tags, meta.FromPath(m.Fields), tagsFirst, rel)
if it.Result.Album == "" {
it.Result.Album = UnknownAlbum
}
return it
}
// StartScan запускает скан источника в фоне.
func (s *Service) StartScan(ctx context.Context, id uuid.UUID, full bool) (Progress, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return Progress{}, err
}
src, err := s.store.GetSource(ctx, id)
if err != nil {
return Progress{}, notFound(err)
}
if err := s.startScan(src, full); err != nil {
return Progress{}, err
}
p, _ := s.board.Get(id)
return p, nil
}
func (s *Service) startScan(src Source, full bool) error {
if src.Status != "mounted" {
return apperr.New(http.StatusConflict, CodeSourceOffline, "source is not connected")
}
root, err := s.Root(src)
if err != nil {
return err
}
if err := s.scanner.Start(src, root, full); errors.Is(err, ErrScanRunning) {
return apperr.Conflict(CodeScanRunning, "scan is already running")
} else if err != nil {
return err
}
return nil
}
// Status — состояние библиотеки (admin, moderator).
type Status struct {
Counts Counts
Sources []Source
}
// Status собирает счётчики и состояние источников.
func (s *Service) Status(ctx context.Context) (Status, error) {
if err := requirePerm(ctx, auth.PermViewLibraryStatus); err != nil {
return Status{}, err
}
c, err := s.store.Counts(ctx)
if err != nil {
return Status{}, err
}
sources, err := s.store.ListSources(ctx)
if err != nil {
return Status{}, err
}
return Status{Counts: c, Sources: sources}, nil
}
// FileErrors — файлы с ошибками чтения.
func (s *Service) FileErrors(ctx context.Context, after string, limit int) ([]FileError, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return nil, err
}
return s.store.FileErrors(ctx, after, limit)
}
// HostDirs — подпапки каталога LIBRARY_HOST для выбора local-хранилища.
func (s *Service) HostDirs(ctx context.Context, rel string) (string, []string, error) {
if err := requirePerm(ctx, auth.PermManageLibrary); err != nil {
return "", nil, err
}
dir, err := s.hostPath(rel)
if err != nil {
return "", nil, err
}
entries, err := os.ReadDir(dir)
if err != nil {
return "", nil, apperr.Validation("path", CodeLocalPathInvalid, "folder not found", nil)
}
dirs := []string{}
for _, e := range entries {
if e.IsDir() && !strings.HasPrefix(e.Name(), ".") {
dirs = append(dirs, e.Name())
}
}
slices.Sort(dirs)
return strings.Trim(strings.ReplaceAll(rel, "\\", "/"), "/"), dirs, nil
}
// Restore при старте api переподключает NFS-хранилища и проверяет local (после перезапуска агента).
func (s *Service) Restore(ctx context.Context) {
sources, err := s.store.ListSources(ctx)
if err != nil {
s.log.Error("restore sources", "err", err)
return
}
for _, src := range sources {
if src.Status == "disabled" {
continue
}
if _, err := s.connect(ctx, src); err != nil {
s.log.Error("restore source", "source_id", src.ID, "err", err)
}
}
}
// RunSchedule раз в минуту проверяет доступность хранилищ и запускает сканы по расписанию.
func (s *Service) RunSchedule(ctx context.Context) {
t := time.NewTicker(time.Minute)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
}
sources, err := s.store.ListSources(ctx)
if err != nil {
s.log.Warn("schedule: list sources", "err", err)
continue
}
for _, src := range sources {
if src.Status == "disabled" {
continue
}
s.healthCheck(ctx, src)
if src.ScanInterval == nil || src.Status != "mounted" {
continue
}
if src.LastScanAt == nil || time.Since(*src.LastScanAt) >= *src.ScanInterval {
if err := s.startScan(src, false); err != nil && !errors.Is(err, apperr.Conflict(CodeScanRunning, "")) {
s.log.Warn("scheduled scan", "source_id", src.ID, "err", err)
}
}
}
}
}
// healthCheck: недоступное хранилище — status=error (треки серые), вернулось — снова mounted.
func (s *Service) healthCheck(ctx context.Context, src Source) {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
done := make(chan struct{})
var status string
var message *string
go func() {
status, message = s.check(ctx, src, false)
close(done)
}()
select {
case <-done:
case <-ctx.Done():
m := "storage does not respond"
status, message = "error", &m
}
if status == "error" && src.Type == SourceNFS && src.Status == "error" {
// Попробовать перемонтировать, как при старте.
if _, err := s.connect(ctx, src); err == nil {
return
}
}
if status != src.Status || deref(message) != deref(src.StatusMessage) {
if err := s.store.SetSourceStatus(ctx, src.ID, status, message); err != nil {
s.log.Warn("health check", "source_id", src.ID, "err", err)
}
}
}
+515
View File
@@ -0,0 +1,515 @@
package library
import (
"context"
"errors"
"fmt"
"time"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"pipu/api/internal/library/meta"
)
// ErrNotFound — запись не найдена.
var ErrNotFound = errors.New("library: not found")
// SourceType — тип хранилища.
type SourceType string
// Типы хранилищ.
const (
SourceNFS SourceType = "nfs"
SourceLocal SourceType = "local"
)
// Source — хранилище (library_sources).
type Source struct {
ID uuid.UUID
Name string
Type SourceType
NFSServer *string
NFSExport *string
NFSOptions string
LocalPath *string
TagsFirst bool
ScanInterval *time.Duration
Status string // pending | mounted | error | disabled
StatusMessage *string
LastScanAt *time.Time
CreatedAt time.Time
AcceptsUploads bool
}
// PatternRow — шаблон пути источника.
type PatternRow struct {
Regex string
IsEnabled bool
}
// FileRow — то, что сканеру нужно знать о ранее найденном файле.
type FileRow struct {
ID uuid.UUID
RelPath string
Size int64
Mtime time.Time
ContentHash []byte
TrackID *uuid.UUID
Missing bool
}
// FileData — результат чтения файла для записи в media_files.
type FileData struct {
RelPath string
Size int64
Mtime time.Time
ContentHash []byte
Probe Probe
ScanError string
}
// Store — доступ к таблицам библиотеки и каталога.
type Store struct {
pool *pgxpool.Pool
}
// NewStore создаёт Store.
func NewStore(pool *pgxpool.Pool) *Store { return &Store{pool: pool} }
func newID() uuid.UUID { return uuid.Must(uuid.NewV7()) }
const sourceColumns = `id, name, type, nfs_server, nfs_export, nfs_options, local_path, tag_priority = 'tags_first',
extract(epoch from scan_interval)::bigint, status, status_message, last_scan_at, created_at, accepts_uploads`
func scanSource(row pgx.Row) (Source, error) {
var (
s Source
typ string
interval *int64
)
err := row.Scan(&s.ID, &s.Name, &typ, &s.NFSServer, &s.NFSExport, &s.NFSOptions, &s.LocalPath, &s.TagsFirst,
&interval, &s.Status, &s.StatusMessage, &s.LastScanAt, &s.CreatedAt, &s.AcceptsUploads)
if errors.Is(err, pgx.ErrNoRows) {
return Source{}, ErrNotFound
}
if err != nil {
return Source{}, fmt.Errorf("library: scan source: %w", err)
}
s.Type = SourceType(typ)
if interval != nil {
d := time.Duration(*interval) * time.Second
s.ScanInterval = &d
}
return s, nil
}
// ListSources — все хранилища по дате создания.
func (st *Store) ListSources(ctx context.Context) ([]Source, error) {
rows, err := st.pool.Query(ctx, `SELECT `+sourceColumns+` FROM library_sources ORDER BY created_at`)
if err != nil {
return nil, fmt.Errorf("library: list sources: %w", err)
}
defer rows.Close()
var out []Source
for rows.Next() {
s, err := scanSource(rows)
if err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
// GetSource — хранилище по id.
func (st *Store) GetSource(ctx context.Context, id uuid.UUID) (Source, error) {
return scanSource(st.pool.QueryRow(ctx, `SELECT `+sourceColumns+` FROM library_sources WHERE id = $1`, id))
}
func intervalSeconds(d *time.Duration) *int64 {
if d == nil {
return nil
}
sec := int64(*d / time.Second)
return &sec
}
func tagPriority(tagsFirst bool) string {
if tagsFirst {
return "tags_first"
}
return "path_first"
}
// SaveSource создаёт или полностью обновляет хранилище (кроме статуса и времени скана).
func (st *Store) SaveSource(ctx context.Context, s Source) (Source, error) {
return scanSource(st.pool.QueryRow(ctx, `
INSERT INTO library_sources (id, name, type, nfs_server, nfs_export, nfs_options, local_path, tag_priority,
scan_interval, status, status_message, created_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, make_interval(secs => $9::bigint), $10, $11, now())
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, nfs_server = EXCLUDED.nfs_server,
nfs_export = EXCLUDED.nfs_export, nfs_options = EXCLUDED.nfs_options, local_path = EXCLUDED.local_path,
tag_priority = EXCLUDED.tag_priority, scan_interval = EXCLUDED.scan_interval
RETURNING `+sourceColumns,
s.ID, s.Name, string(s.Type), s.NFSServer, s.NFSExport, s.NFSOptions, s.LocalPath, tagPriority(s.TagsFirst),
intervalSeconds(s.ScanInterval), s.Status, s.StatusMessage))
}
// SetSourceStatus обновляет состояние подключения.
func (st *Store) SetSourceStatus(ctx context.Context, id uuid.UUID, status string, message *string) error {
_, err := st.pool.Exec(ctx, `UPDATE library_sources SET status = $2, status_message = $3 WHERE id = $1`,
id, status, message)
if err != nil {
return fmt.Errorf("library: set source status: %w", err)
}
return nil
}
// SetLastScan отмечает время завершения скана.
func (st *Store) SetLastScan(ctx context.Context, id uuid.UUID, at time.Time) error {
_, err := st.pool.Exec(ctx, `UPDATE library_sources SET last_scan_at = $2 WHERE id = $1`, id, at)
if err != nil {
return fmt.Errorf("library: set last scan: %w", err)
}
return nil
}
// DeleteSource удаляет хранилище; файлы удаляются каскадно, каталог чистится отдельно (CleanupOrphans).
func (st *Store) DeleteSource(ctx context.Context, id uuid.UUID) error {
tag, err := st.pool.Exec(ctx, `DELETE FROM library_sources WHERE id = $1`, id)
if err != nil {
return fmt.Errorf("library: delete source: %w", err)
}
if tag.RowsAffected() == 0 {
return ErrNotFound
}
return nil
}
// Patterns — шаблоны источника по порядку.
func (st *Store) Patterns(ctx context.Context, sourceID uuid.UUID) ([]PatternRow, error) {
rows, err := st.pool.Query(ctx,
`SELECT regex, is_enabled FROM path_patterns WHERE source_id = $1 ORDER BY position`, sourceID)
if err != nil {
return nil, fmt.Errorf("library: patterns: %w", err)
}
return pgx.CollectRows(rows, func(r pgx.CollectableRow) (PatternRow, error) {
var p PatternRow
return p, r.Scan(&p.Regex, &p.IsEnabled)
})
}
// ReplacePatterns заменяет список шаблонов источника.
func (st *Store) ReplacePatterns(ctx context.Context, sourceID uuid.UUID, patterns []PatternRow) error {
return pgx.BeginFunc(ctx, st.pool, func(tx pgx.Tx) error {
if _, err := tx.Exec(ctx, `DELETE FROM path_patterns WHERE source_id = $1`, sourceID); err != nil {
return fmt.Errorf("library: replace patterns: %w", err)
}
for i, p := range patterns {
if _, err := tx.Exec(ctx, `
INSERT INTO path_patterns (id, source_id, position, regex, is_enabled) VALUES ($1, $2, $3, $4, $5)`,
newID(), sourceID, i, p.Regex, p.IsEnabled); err != nil {
return fmt.Errorf("library: replace patterns: %w", err)
}
}
return nil
})
}
// Files — все известные файлы источника по относительному пути.
func (st *Store) Files(ctx context.Context, sourceID uuid.UUID) (map[string]FileRow, error) {
rows, err := st.pool.Query(ctx, `
SELECT id, rel_path, size_bytes, mtime, content_hash, track_id, missing_since IS NOT NULL
FROM media_files WHERE source_id = $1`, sourceID)
if err != nil {
return nil, fmt.Errorf("library: files: %w", err)
}
defer rows.Close()
out := map[string]FileRow{}
for rows.Next() {
var f FileRow
if err := rows.Scan(&f.ID, &f.RelPath, &f.Size, &f.Mtime, &f.ContentHash, &f.TrackID, &f.Missing); err != nil {
return nil, fmt.Errorf("library: files: %w", err)
}
out[f.RelPath] = f
}
return out, rows.Err()
}
// SampleFiles — случайная выборка путей источника (для предпросмотра шаблонов).
func (st *Store) SampleFiles(ctx context.Context, sourceID uuid.UUID, n int) ([]string, error) {
rows, err := st.pool.Query(ctx, `
SELECT rel_path FROM media_files WHERE source_id = $1 AND missing_since IS NULL ORDER BY random() LIMIT $2`,
sourceID, n)
if err != nil {
return nil, fmt.Errorf("library: sample files: %w", err)
}
return pgx.CollectRows(rows, pgx.RowTo[string])
}
// SaveFile записывает файл (новый — id = uuid.Nil) и возвращает его id.
func (st *Store) SaveFile(ctx context.Context, sourceID, id uuid.UUID, d FileData, trackID *uuid.UUID) (uuid.UUID, error) {
if id == uuid.Nil {
id = newID()
}
var scanErr *string
if d.ScanError != "" {
scanErr = &d.ScanError
}
p := d.Probe
if p.Format == "" {
p.Format = "unknown"
}
_, err := st.pool.Exec(ctx, `
INSERT INTO media_files (id, source_id, rel_path, size_bytes, mtime, content_hash, format, codec, bitrate,
sample_rate, bit_depth, channels, duration_ms, track_id, scan_error)
VALUES ($1, $2, $3, $4, $5, $6, $7, NULLIF($8, ''), NULLIF($9, 0), NULLIF($10, 0), NULLIF($11, 0),
NULLIF($12, 0), $13, $14, $15)
ON CONFLICT (id) DO UPDATE SET rel_path = EXCLUDED.rel_path, size_bytes = EXCLUDED.size_bytes,
mtime = EXCLUDED.mtime, content_hash = EXCLUDED.content_hash, format = EXCLUDED.format,
codec = EXCLUDED.codec, bitrate = EXCLUDED.bitrate, sample_rate = EXCLUDED.sample_rate,
bit_depth = EXCLUDED.bit_depth, channels = EXCLUDED.channels, duration_ms = EXCLUDED.duration_ms,
track_id = EXCLUDED.track_id, scan_error = EXCLUDED.scan_error, missing_since = NULL`,
id, sourceID, d.RelPath, d.Size, d.Mtime, d.ContentHash, p.Format, p.Codec, p.Bitrate, p.SampleRate,
p.BitDepth, p.Channels, p.DurationMs, trackID, scanErr)
if err != nil {
return uuid.Nil, fmt.Errorf("library: save file %s: %w", d.RelPath, err)
}
return id, nil
}
// MoveFile — переименование / перенос: путь меняется, трек (лайки, история) сохраняется.
func (st *Store) MoveFile(ctx context.Context, id uuid.UUID, relPath string, size int64, mtime time.Time) error {
_, err := st.pool.Exec(ctx, `
UPDATE media_files SET rel_path = $2, size_bytes = $3, mtime = $4, missing_since = NULL WHERE id = $1`,
id, relPath, size, mtime)
if err != nil {
return fmt.Errorf("library: move file: %w", err)
}
return nil
}
// SetPresent снимает отметку «пропал» с файла, который снова на месте.
func (st *Store) SetPresent(ctx context.Context, id uuid.UUID) error {
_, err := st.pool.Exec(ctx, `UPDATE media_files SET missing_since = NULL WHERE id = $1`, id)
if err != nil {
return fmt.Errorf("library: set present: %w", err)
}
return nil
}
// MarkMissing отмечает пропавшие файлы (если ещё не отмечены).
func (st *Store) MarkMissing(ctx context.Context, ids []uuid.UUID, at time.Time) error {
if len(ids) == 0 {
return nil
}
_, err := st.pool.Exec(ctx, `
UPDATE media_files SET missing_since = $2 WHERE id = ANY($1) AND missing_since IS NULL`, ids, at)
if err != nil {
return fmt.Errorf("library: mark missing: %w", err)
}
return nil
}
// PurgeMissing удаляет файлы, пропавшие раньше before.
func (st *Store) PurgeMissing(ctx context.Context, sourceID uuid.UUID, before time.Time) (int64, error) {
tag, err := st.pool.Exec(ctx, `DELETE FROM media_files WHERE source_id = $1 AND missing_since < $2`,
sourceID, before)
if err != nil {
return 0, fmt.Errorf("library: purge missing: %w", err)
}
return tag.RowsAffected(), nil
}
// ArtistKeyExists — есть ли исполнитель с таким ключом (для разбора «A & B»).
func (st *Store) ArtistKeyExists(ctx context.Context, key string) bool {
var ok bool
_ = st.pool.QueryRow(ctx, `SELECT EXISTS (SELECT 1 FROM artists WHERE name_norm = $1)`, key).Scan(&ok)
return ok
}
// EnsureArtist находит или создаёт исполнителя по нормализованному имени.
func (st *Store) EnsureArtist(ctx context.Context, name string) (uuid.UUID, error) {
var id uuid.UUID
err := st.pool.QueryRow(ctx, `
INSERT INTO artists (id, name, sort_name, name_norm) VALUES ($1, $2, $3, $4)
ON CONFLICT (name_norm) DO UPDATE SET name_norm = EXCLUDED.name_norm
RETURNING id`, newID(), name, meta.SortName(name), meta.Key(name)).Scan(&id)
if err != nil {
return uuid.Nil, fmt.Errorf("library: ensure artist %q: %w", name, err)
}
return id, nil
}
// EnsureAlbum находит или создаёт альбом исполнителя; год заполняется, если был пуст.
func (st *Store) EnsureAlbum(ctx context.Context, artistID uuid.UUID, title string, year int) (uuid.UUID, error) {
var id uuid.UUID
err := st.pool.QueryRow(ctx, `
INSERT INTO albums (id, title, title_norm, album_artist_id, year) VALUES ($1, $2, $3, $4, NULLIF($5, 0))
ON CONFLICT (album_artist_id, title_norm) DO UPDATE SET year = COALESCE(albums.year, EXCLUDED.year)
RETURNING id`, newID(), title, meta.Key(title), artistID, year).Scan(&id)
if err != nil {
return uuid.Nil, fmt.Errorf("library: ensure album %q: %w", title, err)
}
return id, nil
}
// TrackInput — данные трека из метаданных файла.
type TrackInput struct {
AlbumID uuid.UUID
Title string
Disc int
Track int
Year int
DurationMs int
}
// FindTrack ищет дубль: тот же альбом и название, длительность ±2 с.
func (st *Store) FindTrack(ctx context.Context, in TrackInput) (uuid.UUID, bool, error) {
var id uuid.UUID
err := st.pool.QueryRow(ctx, `
SELECT id FROM tracks WHERE album_id = $1 AND title_norm = $2 AND abs(duration_ms - $3) <= 2000
ORDER BY abs(duration_ms - $3) LIMIT 1`, in.AlbumID, meta.Key(in.Title), in.DurationMs).Scan(&id)
if errors.Is(err, pgx.ErrNoRows) {
return uuid.Nil, false, nil
}
if err != nil {
return uuid.Nil, false, fmt.Errorf("library: find track: %w", err)
}
return id, true, nil
}
// SaveTrack создаёт (id = uuid.Nil) или обновляет трек.
func (st *Store) SaveTrack(ctx context.Context, id uuid.UUID, in TrackInput) (uuid.UUID, error) {
if id == uuid.Nil {
id = newID()
}
_, err := st.pool.Exec(ctx, `
INSERT INTO tracks (id, title, title_norm, album_id, disc_no, track_no, year, duration_ms)
VALUES ($1, $2, $3, $4, NULLIF($5, 0), NULLIF($6, 0), NULLIF($7, 0), $8)
ON CONFLICT (id) DO UPDATE SET title = EXCLUDED.title, title_norm = EXCLUDED.title_norm,
album_id = EXCLUDED.album_id, disc_no = EXCLUDED.disc_no, track_no = EXCLUDED.track_no,
year = EXCLUDED.year, duration_ms = EXCLUDED.duration_ms, updated_at = now()`,
id, in.Title, meta.Key(in.Title), in.AlbumID, in.Disc, in.Track, in.Year, in.DurationMs)
if err != nil {
return uuid.Nil, fmt.Errorf("library: save track: %w", err)
}
return id, nil
}
// SetTrackArtists заменяет исполнителей трека.
func (st *Store) SetTrackArtists(ctx context.Context, trackID uuid.UUID, ids []uuid.UUID, roles []string) error {
return pgx.BeginFunc(ctx, st.pool, func(tx pgx.Tx) error {
if _, err := tx.Exec(ctx, `DELETE FROM track_artists WHERE track_id = $1`, trackID); err != nil {
return fmt.Errorf("library: track artists: %w", err)
}
for i, id := range ids {
if _, err := tx.Exec(ctx, `
INSERT INTO track_artists (track_id, artist_id, role, position) VALUES ($1, $2, $3, $4)
ON CONFLICT DO NOTHING`, trackID, id, roles[i], i); err != nil {
return fmt.Errorf("library: track artists: %w", err)
}
}
return nil
})
}
// SetTrackGenres заменяет жанры трека из тегов.
func (st *Store) SetTrackGenres(ctx context.Context, trackID uuid.UUID, genres []string) error {
return pgx.BeginFunc(ctx, st.pool, func(tx pgx.Tx) error {
if _, err := tx.Exec(ctx, `DELETE FROM track_genres WHERE track_id = $1 AND source = 'tag'`, trackID); err != nil {
return fmt.Errorf("library: track genres: %w", err)
}
for _, g := range genres {
var gid int
if err := tx.QueryRow(ctx, `
INSERT INTO genres (name, name_norm) VALUES ($1, $2)
ON CONFLICT (name_norm) DO UPDATE SET name_norm = EXCLUDED.name_norm RETURNING id`,
g, meta.Key(g)).Scan(&gid); err != nil {
return fmt.Errorf("library: genre %q: %w", g, err)
}
if _, err := tx.Exec(ctx, `
INSERT INTO track_genres (track_id, genre_id, source) VALUES ($1, $2, 'tag') ON CONFLICT DO NOTHING`,
trackID, gid); err != nil {
return fmt.Errorf("library: track genres: %w", err)
}
}
return nil
})
}
// RefreshPrimaryFile выбирает лучший доступный файл трека: lossless, затем битрейт.
func (st *Store) RefreshPrimaryFile(ctx context.Context, trackID uuid.UUID) error {
_, err := st.pool.Exec(ctx, `
UPDATE tracks SET primary_file_id = (
SELECT id FROM media_files
WHERE track_id = $1 AND missing_since IS NULL AND scan_error IS NULL
ORDER BY (codec IN ('flac', 'alac', 'wavpack', 'ape', 'tta') OR codec LIKE 'pcm_%') DESC,
bitrate DESC NULLS LAST, size_bytes DESC
LIMIT 1)
WHERE id = $1`, trackID)
if err != nil {
return fmt.Errorf("library: refresh primary file: %w", err)
}
return nil
}
// CleanupOrphans удаляет треки без файлов, затем пустые альбомы и исполнителей.
func (st *Store) CleanupOrphans(ctx context.Context) error {
for _, q := range []string{
`DELETE FROM tracks t WHERE NOT EXISTS (SELECT 1 FROM media_files f WHERE f.track_id = t.id)`,
`DELETE FROM albums a WHERE NOT EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id)`,
`DELETE FROM artists a WHERE NOT EXISTS (SELECT 1 FROM albums al WHERE al.album_artist_id = a.id)
AND NOT EXISTS (SELECT 1 FROM track_artists ta WHERE ta.artist_id = a.id)`,
`DELETE FROM genres g WHERE NOT EXISTS (SELECT 1 FROM track_genres tg WHERE tg.genre_id = g.id)
AND NOT EXISTS (SELECT 1 FROM genres c WHERE c.parent_id = g.id)`,
} {
if _, err := st.pool.Exec(ctx, q); err != nil {
return fmt.Errorf("library: cleanup: %w", err)
}
}
return nil
}
// Counts — счётчики для статуса библиотеки.
type Counts struct {
Artists, Albums, Tracks, Files, Missing, Errors int64
}
// Counts считает объём библиотеки.
func (st *Store) Counts(ctx context.Context) (Counts, error) {
var c Counts
err := st.pool.QueryRow(ctx, `
SELECT (SELECT count(*) FROM artists), (SELECT count(*) FROM albums), (SELECT count(*) FROM tracks),
(SELECT count(*) FROM media_files), (SELECT count(*) FROM media_files WHERE missing_since IS NOT NULL),
(SELECT count(*) FROM media_files WHERE scan_error IS NOT NULL)`).
Scan(&c.Artists, &c.Albums, &c.Tracks, &c.Files, &c.Missing, &c.Errors)
if err != nil {
return Counts{}, fmt.Errorf("library: counts: %w", err)
}
return c, nil
}
// FileError — файл, который не удалось прочитать.
type FileError struct {
ID uuid.UUID
SourceID uuid.UUID
RelPath string
ScanError string
}
// FileErrors — страница файлов с ошибками чтения (по пути, после after).
func (st *Store) FileErrors(ctx context.Context, after string, limit int) ([]FileError, error) {
rows, err := st.pool.Query(ctx, `
SELECT id, source_id, rel_path, scan_error FROM media_files
WHERE scan_error IS NOT NULL AND rel_path > $1 ORDER BY rel_path LIMIT $2`, after, limit)
if err != nil {
return nil, fmt.Errorf("library: file errors: %w", err)
}
return pgx.CollectRows(rows, func(r pgx.CollectableRow) (FileError, error) {
var e FileError
return e, r.Scan(&e.ID, &e.SourceID, &e.RelPath, &e.ScanError)
})
}
@@ -31,6 +31,14 @@ type API struct {
BootstrapAdminUsername string `env:"BOOTSTRAP_ADMIN_USERNAME" envDefault:"admin"`
BootstrapAdminPassword string `env:"BOOTSTRAP_ADMIN_PASSWORD"`
// LibraryRoot — точки монтирования NFS-хранилищ (общий каталог с storage-agent).
LibraryRoot string `env:"LIBRARY_ROOT" envDefault:"/library"`
// LibraryHostRoot — LIBRARY_HOST внутри контейнера: local-хранилища — его подпапки.
LibraryHostRoot string `env:"LIBRARY_HOST_ROOT" envDefault:"/library-host"`
StorageAgentURL string `env:"STORAGE_AGENT_URL" envDefault:"http://storage-agent:8090"`
InternalToken string `env:"INTERNAL_TOKEN"`
FFprobePath string `env:"FFPROBE_PATH" envDefault:"ffprobe"`
}
// StorageAgent — конфигурация внутреннего сервиса storage-agent.
+6 -1
View File
@@ -10,6 +10,7 @@ import (
"pipu/api/internal/api/gen"
"pipu/api/internal/auth"
"pipu/api/internal/library"
"pipu/api/internal/platform/httpx"
"pipu/api/internal/system"
)
@@ -19,6 +20,8 @@ type Deps struct {
Log *slog.Logger
System *system.Handler
Auth *auth.Handler
// Library — хранилища, сканы и каталог.
Library *library.Handler
// AuthService проверяет access-токены в middleware. nil — только для тестов служебных эндпоинтов.
AuthService *auth.Service
}
@@ -29,11 +32,13 @@ type Deps struct {
type (
systemHandler = system.Handler
authHandler = auth.Handler
libraryHandler = library.Handler
)
type handlers struct {
*systemHandler
*authHandler
*libraryHandler
}
var _ gen.StrictServerInterface = handlers{}
@@ -53,7 +58,7 @@ func NewRouter(d Deps) http.Handler {
r.NotFound(httpx.NotFound)
r.MethodNotAllowed(httpx.MethodNotAllowed)
strict := gen.NewStrictHandlerWithOptions(handlers{systemHandler: d.System, authHandler: d.Auth}, nil, gen.StrictHTTPServerOptions{
strict := gen.NewStrictHandlerWithOptions(handlers{systemHandler: d.System, authHandler: d.Auth, libraryHandler: d.Library}, nil, gen.StrictHTTPServerOptions{
RequestErrorHandlerFunc: httpx.RequestError,
ResponseErrorHandlerFunc: httpx.ResponseError(log),
})
+175
View File
@@ -0,0 +1,175 @@
package storageagent
import (
"context"
"errors"
"fmt"
"io/fs"
"net"
"os"
"os/exec"
"path"
"path/filepath"
"regexp"
"strconv"
"strings"
"time"
)
// NFSSpec — параметры подключения NFS-хранилища.
type NFSSpec struct {
Server string `json:"server"`
Export string `json:"export"`
Options string `json:"options"`
}
// Опции mount, которые можно передать (docs/library-and-storage.md). Значение — допустимы ли аргументы.
var allowedOptions = map[string]*regexp.Regexp{
"ro": nil, "rw": nil, "soft": nil, "hard": nil, "noatime": nil, "nolock": nil,
"nfsvers": regexp.MustCompile(`^(3|4|4\.0|4\.1|4\.2)$`),
"vers": regexp.MustCompile(`^(3|4|4\.0|4\.1|4\.2)$`),
"port": regexp.MustCompile(`^\d{1,5}$`),
"timeo": regexp.MustCompile(`^\d{1,6}$`),
"retrans": regexp.MustCompile(`^\d{1,3}$`),
"rsize": regexp.MustCompile(`^\d{4,8}$`),
"wsize": regexp.MustCompile(`^\d{4,8}$`),
"sec": regexp.MustCompile(`^(sys|krb5|krb5i|krb5p|none)$`),
}
var hostnameRe = regexp.MustCompile(`^[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$`)
// ValidationError — недопустимое поле NFS-подключения.
type ValidationError struct {
Field string `json:"field"`
Reason string `json:"reason"`
}
func (e *ValidationError) Error() string { return e.Field + ": " + e.Reason }
// Normalize проверяет параметры и возвращает опции в каноническом виде.
// writable = false принудительно добавляет ro (только чтение) и убирает rw.
// Ничего из ввода не попадает в shell: значения — отдельные аргументы exec.Command.
func (s NFSSpec) Normalize(writable bool) (NFSSpec, error) {
server := strings.TrimSpace(s.Server)
if ip := net.ParseIP(strings.Trim(server, "[]")); ip != nil {
server = ip.String()
if ip.To4() == nil {
server = "[" + server + "]"
}
} else if len(server) > 253 || !hostnameRe.MatchString(server) {
return NFSSpec{}, &ValidationError{Field: "nfs_server", Reason: "not a hostname or IP address"}
}
export := strings.TrimSpace(s.Export)
if !strings.HasPrefix(export, "/") || len(export) > 1024 || strings.ContainsAny(export, "\x00\n\r,") {
return NFSSpec{}, &ValidationError{Field: "nfs_export", Reason: "must be an absolute path"}
}
for _, part := range strings.Split(export, "/") {
if part == ".." {
return NFSSpec{}, &ValidationError{Field: "nfs_export", Reason: "must not contain .."}
}
}
export = path.Clean(export)
opts := []string{}
seen := map[string]bool{}
for _, raw := range strings.Split(s.Options, ",") {
raw = strings.TrimSpace(raw)
if raw == "" {
continue
}
name, value, hasValue := strings.Cut(raw, "=")
re, ok := allowedOptions[name]
if !ok || (re == nil) == hasValue || (re != nil && !re.MatchString(value)) {
return NFSSpec{}, &ValidationError{Field: "nfs_options", Reason: fmt.Sprintf("option %q is not allowed", raw)}
}
if name == "ro" || name == "rw" || seen[name] {
continue
}
seen[name] = true
opts = append(opts, raw)
}
mode := "ro"
if writable {
mode = "rw"
}
opts = append([]string{mode}, opts...)
return NFSSpec{Server: server, Export: export, Options: strings.Join(opts, ",")}, nil
}
// Mounter выполняет mount / umount. Подменяется в тестах.
type Mounter interface {
Mount(ctx context.Context, spec NFSSpec, target string) error
Unmount(ctx context.Context, target string) error
}
// ExecMounter вызывает системные mount / umount.
type ExecMounter struct{}
// Mount реализует Mounter.
func (ExecMounter) Mount(ctx context.Context, spec NFSSpec, target string) error {
if err := os.MkdirAll(target, 0o755); err != nil {
return err
}
out, err := exec.CommandContext(ctx, "mount", "-t", "nfs", "-o", spec.Options,
spec.Server+":"+spec.Export, target).CombinedOutput()
if err != nil {
return fmt.Errorf("mount: %s", strings.TrimSpace(string(out)))
}
return nil
}
// Unmount реализует Mounter. Ленивое отключение: зависший NFS-сервер не блокирует агента.
func (ExecMounter) Unmount(ctx context.Context, target string) error {
out, err := exec.CommandContext(ctx, "umount", "-l", target).CombinedOutput()
if err != nil && !strings.Contains(string(out), "not mounted") {
return fmt.Errorf("umount: %s", strings.TrimSpace(string(out)))
}
return nil
}
var sourceIDRe = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
// mountPoint — каталог источника в корне библиотеки; id проверяется, чтобы путь не ушёл из корня.
func mountPoint(root, sourceID string) (string, error) {
if !sourceIDRe.MatchString(sourceID) {
return "", errors.New("invalid source id")
}
return filepath.Join(root, sourceID), nil
}
var audioExt = map[string]bool{
".flac": true, ".mp3": true, ".m4a": true, ".aac": true, ".ogg": true, ".opus": true,
".wav": true, ".aiff": true, ".aif": true, ".wv": true, ".ape": true,
}
// countAudio считает аудиофайлы в дереве, останавливаясь после limit файлов или по таймауту.
func countAudio(ctx context.Context, root string, limit int) (int, error) {
n := 0
stop := errors.New("stop")
err := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
if ctx.Err() != nil || n >= limit {
return stop
}
if err != nil {
if p == root {
return err
}
return nil
}
if !d.IsDir() && audioExt[strings.ToLower(filepath.Ext(p))] {
n++
}
return nil
})
if err != nil && !errors.Is(err, stop) {
return n, err
}
return n, nil
}
// probeTimeout — сколько ждать листинга при проверке подключения.
const probeTimeout = 20 * time.Second
// itoa — для сообщений.
func itoa(n int) string { return strconv.Itoa(n) }
@@ -0,0 +1,105 @@
package storageagent
import (
"bytes"
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNFSNormalize(t *testing.T) {
ok := []struct {
in NFSSpec
writable bool
want NFSSpec
}{
{NFSSpec{"nas.lan", "/export/music/", "rw,nfsvers=4.1,soft"}, false, NFSSpec{"nas.lan", "/export/music", "ro,nfsvers=4.1,soft"}},
{NFSSpec{" 192.168.1.10 ", "/m", ""}, true, NFSSpec{"192.168.1.10", "/m", "rw"}},
{NFSSpec{"fd00::1", "/m", "timeo=150,timeo=10"}, false, NFSSpec{"[fd00::1]", "/m", "ro,timeo=150"}},
}
for _, tc := range ok {
got, err := tc.in.Normalize(tc.writable)
require.NoError(t, err, tc.in)
assert.Equal(t, tc.want, got)
}
bad := []struct {
in NFSSpec
field string
}{
{NFSSpec{"nas;rm -rf /", "/m", ""}, "nfs_server"},
{NFSSpec{"-oexec", "/m", ""}, "nfs_server"},
{NFSSpec{"nas", "relative", ""}, "nfs_export"},
{NFSSpec{"nas", "/a/../etc", ""}, "nfs_export"},
{NFSSpec{"nas", "/m,exec", ""}, "nfs_export"},
{NFSSpec{"nas", "/m", "exec"}, "nfs_options"},
{NFSSpec{"nas", "/m", "nfsvers=9"}, "nfs_options"},
{NFSSpec{"nas", "/m", "soft=1"}, "nfs_options"},
{NFSSpec{"nas", "/m", "port"}, "nfs_options"},
}
for _, tc := range bad {
_, err := tc.in.Normalize(false)
var ve *ValidationError
require.ErrorAs(t, err, &ve, tc.in)
assert.Equal(t, tc.field, ve.Field, tc.in)
}
}
type fakeMounter struct {
mounted map[string]NFSSpec
files int
}
func (f *fakeMounter) Mount(_ context.Context, spec NFSSpec, target string) error {
if err := os.MkdirAll(target, 0o755); err != nil {
return err
}
for i := range f.files {
_ = os.WriteFile(filepath.Join(target, "t"+itoa(i)+".flac"), nil, 0o644)
}
f.mounted[target] = spec
return nil
}
func (f *fakeMounter) Unmount(_ context.Context, target string) error {
delete(f.mounted, target)
return nil
}
func TestMountEndpoint(t *testing.T) {
root := t.TempDir()
m := &fakeMounter{mounted: map[string]NFSSpec{}, files: 3}
h := NewRouterWithMounter(nil, "secret", root, m)
post := func(path string, body any) *httptest.ResponseRecorder {
raw, _ := json.Marshal(body)
req := httptest.NewRequest(http.MethodPost, path, bytes.NewReader(raw))
req.Header.Set("Authorization", "Bearer secret")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec
}
id := "0190a1b2-c3d4-7e5f-8a9b-0c1d2e3f4a5b"
rec := post("/internal/mounts", map[string]any{"source_id": id, "server": "nas", "export": "/music", "options": "soft"})
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var res MountResult
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &res))
assert.Equal(t, MountResult{Status: "mounted", AudioFiles: 3, Message: "found 3 audio files"}, res)
assert.Equal(t, "ro,soft", m.mounted[filepath.Join(root, id)].Options)
assert.Equal(t, http.StatusUnprocessableEntity,
post("/internal/mounts", map[string]any{"source_id": "../etc", "server": "nas", "export": "/m"}).Code)
assert.Equal(t, http.StatusUnprocessableEntity,
post("/internal/mounts/test", map[string]any{"server": "nas", "export": "/m", "options": "exec"}).Code)
rec = post("/internal/mounts/test", map[string]any{"server": "nas", "export": "/m"})
require.Equal(t, http.StatusOK, rec.Code)
assert.Len(t, m.mounted, 1, "test mount is removed")
}
+124 -2
View File
@@ -1,10 +1,13 @@
// Package storageagent — внутренний сервис монтирования хранилищ (см. docs/library-and-storage.md).
// В M0 — каркас: проверка здоровья, аутентификация межсервисных запросов, список точек монтирования.
package storageagent
import (
"context"
"crypto/rand"
"crypto/subtle"
"encoding/hex"
"encoding/json"
"errors"
"log/slog"
"net/http"
"os"
@@ -25,6 +28,11 @@ type Mount struct {
// NewRouter возвращает роутер агента. token — общий секрет INTERNAL_TOKEN;
// при пустом token все запросы к /internal отклоняются.
func NewRouter(log *slog.Logger, token, libraryRoot string) http.Handler {
return NewRouterWithMounter(log, token, libraryRoot, ExecMounter{})
}
// NewRouterWithMounter — NewRouter с заданным способом монтирования (для тестов).
func NewRouterWithMounter(log *slog.Logger, token, libraryRoot string, mounter Mounter) http.Handler {
if log == nil {
log = slog.New(slog.DiscardHandler)
}
@@ -44,10 +52,124 @@ func NewRouter(log *slog.Logger, token, libraryRoot string) http.Handler {
}
writeJSON(w, http.StatusOK, map[string]any{"mounts": mounts})
})
a := &agent{log: log, root: libraryRoot, mounter: mounter}
r.Post("/mounts", a.mount)
r.Post("/mounts/test", a.test)
r.Delete("/mounts/{sourceID}", a.unmount)
})
return r
}
type agent struct {
log *slog.Logger
root string
mounter Mounter
}
// MountRequest — тело POST /internal/mounts.
type MountRequest struct {
SourceID string `json:"source_id"`
NFSSpec
Writable bool `json:"writable"`
}
// MountResult — итог монтирования или проверки.
type MountResult struct {
Status string `json:"status"` // mounted | error
Message string `json:"message,omitempty"`
AudioFiles int `json:"audio_files"`
}
func (a *agent) decode(w http.ResponseWriter, r *http.Request, v any) bool {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<16)).Decode(v); err != nil {
httpx.WriteProblem(w, http.StatusBadRequest, "common.bad_request", "invalid json")
return false
}
return true
}
func writeValidation(w http.ResponseWriter, err error) {
var ve *ValidationError
if errors.As(err, &ve) {
writeJSON(w, http.StatusUnprocessableEntity, map[string]any{
"status": http.StatusUnprocessableEntity, "code": "validation.nfs_invalid", "detail": ve.Reason,
"errors": []map[string]string{{"field": ve.Field, "code": "validation.nfs_invalid"}},
})
return
}
httpx.WriteProblem(w, http.StatusBadRequest, "common.bad_request", err.Error())
}
// mount (пере)монтирует источник в <root>/<source_id> и считает аудиофайлы для проверки.
func (a *agent) mount(w http.ResponseWriter, r *http.Request) {
var req MountRequest
if !a.decode(w, r, &req) {
return
}
target, err := mountPoint(a.root, req.SourceID)
if err != nil {
writeValidation(w, &ValidationError{Field: "source_id", Reason: err.Error()})
return
}
spec, err := req.NFSSpec.Normalize(req.Writable)
if err != nil {
writeValidation(w, err)
return
}
ctx, cancel := context.WithTimeout(r.Context(), probeTimeout)
defer cancel()
_ = a.mounter.Unmount(ctx, target)
writeJSON(w, http.StatusOK, a.mountAndCount(ctx, spec, target))
}
// test монтирует во временную точку, считает файлы и отключает.
func (a *agent) test(w http.ResponseWriter, r *http.Request) {
var spec NFSSpec
if !a.decode(w, r, &spec) {
return
}
spec, err := spec.Normalize(false)
if err != nil {
writeValidation(w, err)
return
}
var rnd [6]byte
_, _ = rand.Read(rnd[:])
target := filepath.Join(a.root, ".test-"+hex.EncodeToString(rnd[:]))
ctx, cancel := context.WithTimeout(r.Context(), probeTimeout)
defer cancel()
res := a.mountAndCount(ctx, spec, target)
_ = a.mounter.Unmount(context.Background(), target)
_ = os.Remove(target)
writeJSON(w, http.StatusOK, res)
}
func (a *agent) mountAndCount(ctx context.Context, spec NFSSpec, target string) MountResult {
if err := a.mounter.Mount(ctx, spec, target); err != nil {
a.log.WarnContext(ctx, "nfs mount failed", "target", target, "err", err)
return MountResult{Status: "error", Message: err.Error()}
}
n, err := countAudio(ctx, target, 1000)
if err != nil {
return MountResult{Status: "error", Message: "mounted, but listing failed: " + err.Error()}
}
return MountResult{Status: "mounted", AudioFiles: n, Message: "found " + itoa(n) + " audio files"}
}
func (a *agent) unmount(w http.ResponseWriter, r *http.Request) {
target, err := mountPoint(a.root, chi.URLParam(r, "sourceID"))
if err != nil {
writeValidation(w, &ValidationError{Field: "source_id", Reason: err.Error()})
return
}
if err := a.mounter.Unmount(r.Context(), target); err != nil {
httpx.WriteProblem(w, http.StatusInternalServerError, "storage.unmount_failed", err.Error())
return
}
_ = os.Remove(target)
w.WriteHeader(http.StatusNoContent)
}
func requireToken(token string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -72,7 +194,7 @@ func listMounts(root string) ([]Mount, error) {
}
mounts := make([]Mount, 0, len(entries))
for _, e := range entries {
if e.IsDir() {
if e.IsDir() && !strings.HasPrefix(e.Name(), ".") {
mounts = append(mounts, Mount{SourceID: e.Name(), Path: filepath.Join(root, e.Name())})
}
}
+135
View File
@@ -0,0 +1,135 @@
-- +goose Up
CREATE TYPE source_type AS ENUM ('nfs', 'local');
CREATE TYPE source_status AS ENUM ('pending', 'mounted', 'error', 'disabled');
CREATE TYPE tag_priority AS ENUM ('tags_first', 'path_first');
CREATE TABLE library_sources (
id uuid PRIMARY KEY,
name text NOT NULL,
type source_type NOT NULL,
nfs_server text,
nfs_export text,
nfs_options text NOT NULL DEFAULT 'ro,nfsvers=4.1,soft,timeo=150,retrans=3',
local_path text,
tag_priority tag_priority NOT NULL DEFAULT 'tags_first',
accepts_uploads boolean NOT NULL DEFAULT false,
write_template text,
scan_interval interval,
status source_status NOT NULL DEFAULT 'pending',
status_message text,
last_scan_at timestamptz,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE path_patterns (
id uuid PRIMARY KEY,
source_id uuid NOT NULL REFERENCES library_sources(id) ON DELETE CASCADE,
position int NOT NULL,
regex text NOT NULL,
is_enabled boolean NOT NULL DEFAULT true,
UNIQUE (source_id, position)
);
CREATE TABLE artists (
id uuid PRIMARY KEY,
name text NOT NULL,
sort_name text NOT NULL,
name_norm text NOT NULL UNIQUE,
image_path text,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE albums (
id uuid PRIMARY KEY,
title text NOT NULL,
title_norm text NOT NULL,
album_artist_id uuid NOT NULL REFERENCES artists(id),
year int,
cover_path text,
cover_colors text[],
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (album_artist_id, title_norm)
);
CREATE TABLE media_files (
id uuid PRIMARY KEY,
source_id uuid NOT NULL REFERENCES library_sources(id) ON DELETE CASCADE,
rel_path text NOT NULL,
size_bytes bigint NOT NULL,
mtime timestamptz NOT NULL,
content_hash bytea,
sha256 bytea,
upload_file_id uuid,
format text NOT NULL,
codec text,
bitrate int,
sample_rate int,
bit_depth int,
channels int,
duration_ms int NOT NULL,
track_id uuid,
missing_since timestamptz,
scan_error text,
UNIQUE (source_id, rel_path)
);
CREATE INDEX media_files_content_hash_idx ON media_files (content_hash);
CREATE INDEX media_files_sha256_idx ON media_files (sha256);
CREATE INDEX media_files_track_idx ON media_files (track_id);
CREATE TABLE tracks (
id uuid PRIMARY KEY,
title text NOT NULL,
title_norm text NOT NULL,
album_id uuid NOT NULL REFERENCES albums(id),
disc_no int,
track_no int,
year int,
duration_ms int NOT NULL,
primary_file_id uuid REFERENCES media_files(id) ON DELETE SET NULL,
explicit boolean,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX tracks_album_idx ON tracks (album_id, disc_no, track_no);
-- Дубли: тот же альбом, название и длительность ±2 с ищутся по (album_id, title_norm).
CREATE INDEX tracks_dedup_idx ON tracks (album_id, title_norm);
ALTER TABLE media_files ADD FOREIGN KEY (track_id) REFERENCES tracks(id) ON DELETE SET NULL;
CREATE TABLE track_artists (
track_id uuid REFERENCES tracks(id) ON DELETE CASCADE,
artist_id uuid REFERENCES artists(id) ON DELETE CASCADE,
role text NOT NULL DEFAULT 'main',
position int NOT NULL,
PRIMARY KEY (track_id, artist_id, role)
);
CREATE INDEX track_artists_artist_idx ON track_artists (artist_id);
CREATE TABLE genres (
id serial PRIMARY KEY,
name text NOT NULL,
name_norm text NOT NULL UNIQUE,
parent_id int REFERENCES genres(id)
);
CREATE TABLE track_genres (
track_id uuid REFERENCES tracks(id) ON DELETE CASCADE,
genre_id int REFERENCES genres(id),
source text NOT NULL,
weight real NOT NULL DEFAULT 1,
PRIMARY KEY (track_id, genre_id, source)
);
-- +goose Down
DROP TABLE track_genres;
DROP TABLE genres;
DROP TABLE track_artists;
ALTER TABLE media_files DROP CONSTRAINT media_files_track_id_fkey;
DROP TABLE tracks;
DROP TABLE media_files;
DROP TABLE albums;
DROP TABLE artists;
DROP TABLE path_patterns;
DROP TABLE library_sources;
DROP TYPE tag_priority;
DROP TYPE source_status;
DROP TYPE source_type;