homelab/scripts/cli/ha-sync.md

ha-sync

Overview

ha-sync performs bidirectional file synchronisation between the Dell node (192.168.2.100) and the HP node (192.168.2.193) over NFS-mounted volumes. Each sync run is recorded to MySQL, and runs are scheduled as Kubernetes CronJobs in the infrastructure namespace.

A distributed Kubernetes Lease (ha-sync-<jobname>) prevents concurrent runs of the same job, and an optional dry-run mode lets you preview what would happen without touching any files.

CronJobs are now managed by ha-sync-ctl and the DB — static YAML manifests have been archived. Do not apply cron-*.yaml manually.

---

Tools

Binary	Purpose
ha-sync	Sync runner — invoked by K8s CronJobs
ha-sync-ctl	Management CLI — create/manage jobs, inspect runs and operations

---

ha-sync (runner)

The runner is normally invoked by a Kubernetes CronJob. It is not intended for direct use.

CLI Flags

Flag	Required	Default	Description
--src	✅	—	Source directory path inside the pod
--dest	✅	—	Destination directory path inside the pod
--pair	✅	—	Logical sync-pair name (e.g. media)
--direction		fwd	Direction label (e.g. dell-to-hp)
--src-host		""	Source host identifier (e.g. dell) — recorded in DB
--dest-host		""	Destination host identifier (e.g. hp) — recorded in DB
--db-dsn		$HA_SYNC_DB_DSN	MySQL DSN
--lock-ttl		3600	Lease TTL in seconds
--log-dir		/var/log/ha-sync	Operation log directory
--log-retain-days		10	Purge logs older than N days
--mtime-threshold		2s	mtime equality tolerance
--delete-missing		false	Delete files at dest absent from src
--workers		4	Concurrent file-copy workers
--dry-run		false	Preview only — no files touched
--verbose		false	Verbose log output

Environment Variables

Variable	Description
HA_SYNC_DB_DSN	MySQL DSN when --db-dsn is not set

---

ha-sync-ctl (management CLI)

The management CLI controls jobs, inspects runs, and drills into per-file operations.

Environment Variables

Variable	Description
HA_SYNC_DB_DSN	MySQL DSN (required for all DB commands)
HA_SYNC_NAMESPACE	K8s namespace (default: infrastructure)
KUBECONFIG	Path to kubeconfig (falls back to in-cluster)

Global Flags

Flag	Description
--output=json	Output results as JSON instead of table

---

Job Commands

ha-sync-ctl jobs list

List all sync jobs with their last run status and file counts.

NAME              PAIR       DIRECTION    ENABLED  SCHEDULE       LAST RUN        STATUS
media-d2h         media      dell→hp      yes      0 2 * * *      2h ago          ok
media-h2d         media      hp→dell      yes      0 3 * * *      2h ago          ok
games-d2h         games      dell→hp      no       0 4 * * *      5d ago          disabled

ha-sync-ctl jobs create

Create a new sync job and apply its CronJob to K8s.

ha-sync-ctl jobs create \
  --name=media-dell-to-hp \
  --pair=media \
  --direction=dell-to-hp \
  --src=/mnt/dell/media \
  --dest=/mnt/hp/media \
  --src-host=dell \
  --dest-host=hp \
  --schedule="0 2 * * *" \
  --workers=4 \
  --lock-ttl=86400

Flag	Default	Description
--name	required	Unique job name
--pair	required	One of: media, photos, owncloud, games, infra, ai
--direction	required	dell-to-hp or hp-to-dell
--src	required	Source path
--dest	required	Destination path
--src-host	""	Source host label
--dest-host	""	Destination host label
--schedule	0 2 * * *	Cron schedule
--lock-ttl	3600	Lock TTL in seconds
--workers	4	Parallel workers
--dry-run	false	Start in dry-run mode
--delete-missing	false	Delete files at dest not in src
--excludes	""	Comma-separated glob patterns to exclude

ha-sync-ctl jobs edit <name-or-id>

Update one or more fields of an existing job. Only flags you pass are updated.

ha-sync-ctl jobs edit media-dell-to-hp --schedule="0 3 * * *" --workers=8

ha-sync-ctl jobs delete <name-or-id>

Delete a job from the DB and remove its CronJob from K8s.

ha-sync-ctl jobs delete media-dell-to-hp

ha-sync-ctl jobs enable <name-or-id>

Enable a job (unsuspends the K8s CronJob).

ha-sync-ctl jobs enable games-dell-to-hp

ha-sync-ctl jobs disable <name-or-id>

Disable a job (suspends the K8s CronJob — no new runs scheduled).

ha-sync-ctl jobs disable games-dell-to-hp

ha-sync-ctl jobs trigger <name-or-id>

Trigger an immediate run by creating a K8s Job from the CronJob.

ha-sync-ctl jobs trigger media-dell-to-hp

ha-sync-ctl jobs lock-status <name-or-id>

Show whether the job is currently running (locked via K8s Lease).

ha-sync-ctl jobs lock-status media-dell-to-hp
# locked=true  holder=ha-sync-media-d2h-abc  expires_at=2026-04-09T03:15:00Z

ha-sync-ctl jobs apply-all

Apply (create or update) K8s CronJobs for all enabled jobs in the DB. Safe to run multiple times (idempotent).

ha-sync-ctl jobs apply-all
# Applied 10 CronJobs

ha-sync-ctl jobs import-k8s

Import existing K8s CronJobs (labelled app=ha-sync) into the DB. Skips jobs already present (idempotent).

ha-sync-ctl jobs import-k8s
# Imported: 10  Already existed: 0

---

Run Commands

ha-sync-ctl runs list [--job=<name>] [--limit=N]

List recent sync runs.

RUN ID    JOB           STARTED              DURATION   COPIED  DELETED  SKIPPED  ERRORS
abc123    media-d2h     2026-04-09 02:00     4m32s      142     0        891      0
def456    photos-d2h    2026-04-09 01:00     1m15s      0       0        3421     0

ha-sync-ctl runs show <run-id>

Show detail for a single run including per-action counts.

---

Operation Commands

ha-sync-ctl ops list <run-id> [--limit=N] [--action=copy|delete|skip|error]

List per-file operations for a run.

ACTION   PATH                               FROM→TO    SIZE     OWNER
copy     /media/movies/film.mkv             dell→hp    4.2 GB   dan
copy     /media/music/album/track.flac      dell→hp    32 MB    dan
skip     /media/photos/img001.jpg           dell→hp    3.1 MB   dan
delete   /media/old/deleted.mp4             dell→hp    800 MB   dan

---

Migration: Static CronJobs → DB-managed

The old cron-*.yaml manifests are archived in deployment/ha-sync/archive/. Run this once after deploying the new image:

# 1. Import existing K8s CronJobs into DB
ha-sync-ctl jobs import-k8s

# 2. Delete old static CronJobs from the cluster
kubectl delete cronjobs -n infrastructure \
  ha-sync-ai-dell-to-hp ha-sync-ai-hp-to-dell \
  ha-sync-games-dell-to-hp ha-sync-games-hp-to-dell \
  ha-sync-infra-dell-to-hp ha-sync-infra-hp-to-dell \
  ha-sync-media-dell-to-hp ha-sync-media-hp-to-dell \
  ha-sync-owncloud-dell-to-hp ha-sync-owncloud-hp-to-dell \
  ha-sync-photos-dell-to-hp ha-sync-photos-hp-to-dell

# 3. Apply DB-managed CronJobs
ha-sync-ctl jobs apply-all

---

Delete Safety Warning

⚠️ --delete-missing defaults to false.

Enabling --delete-missing on both directions simultaneously risks data loss if one node is temporarily unavailable or partially mounted.

Recommended practice:

1. Complete an initial full sync without --delete-missing and verify parity via the UI dashboard.
2. Enable --delete-missing only on the primary direction (e.g. dell-to-hp) once parity is confirmed.
3. Leave the reverse direction without --delete-missing unless you understand the implications.

---

UI Dashboard

https://ha-sync.vandachevici.ro

3-view SPA showing:

• Jobs grid — all jobs at a glance: status, last run, file counts, lock indicator, enable/disable toggle, Run Now button
• Job detail — configuration + full run history; click a run to drill in
• Operations drawer — per-file breakdown: action, path, from→to host, size, owner; filterable by action type

Auto-refreshes every 30 seconds.