homelab/execution-plans/ha-sync.md

# HA Sync — Execution Plan

## Problem Statement

Two servers (Dell OptiPlex 7070 at `192.168.2.100` and HP ProLiant at `192.168.2.193`) each export the same folder set over NFS. A Kubernetes-native tool must keep each folder pair in bidirectional sync: newest file wins, mtime is preserved on copy, delete propagation is strict (one-way per CronJob), and every operation is logged in the MySQL instance in the `infrastructure` namespace.

---

## Architecture Decisions (Agreed)

| Decision | Choice | Rationale |
|---|---|---|
| Language | **Go** | Single static binary, excellent async I/O, no runtime overhead |
| Sync direction | **Bidirectional via two one-way CronJobs** | Each folder pair gets `a→b` and `b→a` jobs; newest-mtime wins |
| Loop prevention | **Preserve mtime on copy + `--delete-missing` flag** | Mtime equality → skip; no extra DB state needed |
| Lock | **Kubernetes `Lease` object (coordination.k8s.io/v1)** | Native K8s TTL; survives MySQL outage; sync blocked only if K8s API is down (already required for CronJob) |
| Change detection | **mtime + size first; MD5 only on mtime/size mismatch** | Efficient for large datasets |
| Delete propagation | **Strict mirror — configurable per job via `--delete-missing`** | See ⚠️ note below |
| Volume access | **NFS mounts (both servers already export NFS)** | No HostPath or node-affinity needed |

| Audit logging | **Write to opslog file during run; flush to MySQL on completion** | MySQL outage does not block sync; unprocessed opslogs are retried on next run |
| Opslog storage | **Persistent NFS-backed PVC at `/var/log/ha-sync/`** | `/tmp` is ephemeral (lost on pod exit); NFS PVC persists across CronJob runs for 10-day retention |

### Locking: Kubernetes Lease

Each sync pair uses a `coordination.k8s.io/v1` Lease object named `ha-sync-<pair>` in the `infrastructure` namespace.

- `spec.holderIdentity` = `<pod-name>/<iteration-id>`
- `spec.leaseDurationSeconds` = `--lock-ttl` (default 3600)
- A background goroutine renews (`spec.renewTime`) every `leaseDurationSeconds / 3` seconds
- On normal exit or SIGTERM: Lease is deleted (released)
- Stale leases (holder crashed without release): expire automatically after `leaseDurationSeconds`
- Requires RBAC: `ServiceAccount` with `create/get/update/delete` on `leases` in `infrastructure`

### Audit Logging: Opslog + MySQL Flush

1. On sync start: open `/var/log/ha-sync/opslog-<pair>-<direction>-<RFC3339>.jsonl`
2. Each file operation: append one JSON line (all `sync_operations` fields)
3. On sync end: attempt flush to MySQL (`sync_iterations` + `sync_operations` batch INSERT)
4. On successful flush: delete the opslog file
5. On MySQL failure: leave the opslog; on next run, scan `/var/log/ha-sync/` for unprocessed opslogs and retry flush before starting new sync
6. Cleanup: after each run, delete opslogs older than 10 days (`os.Stat` mtime check)

### ⚠️ Delete Propagation Warning

With two one-way jobs per pair, ordering matters for deletes. If `dell→hp` runs before `hp→dell` and `--delete-missing` is ON for both, files that only exist on HP will be deleted before they're copied to Dell.

**Safe default**: `--delete-missing=false` for all jobs. Enable `--delete-missing=true` only on the **primary direction** (e.g., `dell→hp` for each pair) once the initial full sync has completed and both sides are known-equal.

---

## NFS Sync Pairs

| Pair name | Dell NFS (192.168.2.100) | HP NFS (192.168.2.193) |
|---|---|---|
| `media` | `/data/media` | `/data/media` |
| `photos` | `/data/photos` | `/data/photos` |
| `owncloud` | `/data/owncloud` | `/data/owncloud` |
| `games` | `/data/games` | `/data/games` |
| `infra` | `/data/infra` | `/data/infra` |
| `ai` | `/data/ai` | `/data/ai` |

Each pair produces **two CronJobs** in the `infrastructure` namespace.

---

## CLI Interface (`ha-sync`)

```
ha-sync [flags]

Required:
  --src <path>            Source directory (absolute path inside pod)
  --dest <path>           Destination directory (absolute path inside pod)
  --pair <name>           Logical pair name (e.g. "media"); used as Lease name ha-sync-<pair>

Optional:
  --direction <str>       Label for logging, e.g. "dell-to-hp" (default: "fwd")
  --db-dsn <dsn>          MySQL DSN (default: from env HA_SYNC_DB_DSN)
  --lock-ttl <seconds>    Lease TTL before considered stale (default: 3600)
  --log-dir <path>        Directory for opslog files (default: /var/log/ha-sync)
  --log-retain-days <n>   Delete opslogs older than N days (default: 10)
  --mtime-threshold <s>   Seconds of tolerance for mtime equality (default: 2)
  --delete-missing        Delete dest files not present in src (default: false)
  --workers <n>           Concurrent file workers (default: 4)
  --dry-run               Compute what would sync, save to DB as dry_run rows, print plan; do not copy/delete (default: false)
  --verbose               Verbose output
  --help
```

---

## MySQL Schema (database: `general_db`)

```sql
-- One row per CronJob execution
CREATE TABLE IF NOT EXISTS sync_iterations (
  id                     BIGINT AUTO_INCREMENT PRIMARY KEY,
  sync_pair              VARCHAR(255)  NOT NULL,
  direction              VARCHAR(64)   NOT NULL,
  src                    VARCHAR(512)  NOT NULL,
  dest                   VARCHAR(512)  NOT NULL,
  started_at             DATETIME(3)   NOT NULL,
  ended_at               DATETIME(3),
  status                 ENUM('running','success','partial_failure','failed') NOT NULL DEFAULT 'running',
  dry_run                TINYINT(1)    NOT NULL DEFAULT 0,
  files_created          INT           DEFAULT 0,
  files_updated          INT           DEFAULT 0,
  files_deleted          INT           DEFAULT 0,
  files_skipped          INT           DEFAULT 0,
  files_failed           INT           DEFAULT 0,
  total_bytes_transferred BIGINT       DEFAULT 0,
  error_message          TEXT,
  INDEX idx_pair    (sync_pair),
  INDEX idx_started (started_at),
  INDEX idx_dry_run (dry_run)
);

-- One row per individual file operation (flushed from opslog on sync completion)
CREATE TABLE IF NOT EXISTS sync_operations (
  id            BIGINT AUTO_INCREMENT PRIMARY KEY,
  iteration_id  BIGINT        NOT NULL,
  dry_run       TINYINT(1)    NOT NULL DEFAULT 0,
  operation     ENUM('create','update','delete') NOT NULL,
  filepath      VARCHAR(4096) NOT NULL,
  size_before   BIGINT,
  size_after    BIGINT,
  md5_before    VARCHAR(32),
  md5_after     VARCHAR(32),
  started_at    DATETIME(3)   NOT NULL,
  ended_at      DATETIME(3),
  status        ENUM('success','fail') NOT NULL,
  error_message VARCHAR(4096),
  INDEX idx_iteration (iteration_id),
  CONSTRAINT fk_iteration FOREIGN KEY (iteration_id) REFERENCES sync_iterations(id)
);
```

> No `sync_locks` table — locking is handled by Kubernetes Lease objects.

### Dry-run Idempotency Rules

1. **`--dry-run` mode**: walk source and dest, compute the full set of would-be operations (create/update/delete), save to DB with `dry_run = 1`, print the plan. **No files are copied or deleted.**
2. **Idempotency check**: before running a dry-run, query for the last successful dry-run iteration for `(pair, direction)`:
   ```sql
   SELECT id, started_at FROM sync_iterations
   WHERE sync_pair = ? AND direction = ? AND dry_run = 1 AND status = 'success'
   ORDER BY started_at DESC LIMIT 1;
   ```
   Then re-walk the source and dest and compute the would-be operation set. Compare it against the `sync_operations` rows from that previous dry-run iteration (same set of `filepath + operation + size_before`). If **identical** → print `"Dry-run already current as of <started_at>. Nothing has changed."` and exit without writing new rows.
3. **Production run (`--dry-run` not set)**: all queries for previous iterations use `WHERE dry_run = 0`. Dry-run rows are **never considered** for skip logic, idempotency, or status reporting in production runs.
4. **Lease is still acquired** during dry-run (prevents two dry-runs from racing each other).

---

## Project Structure

```
services/ha-sync/
  cmd/ha-sync/
    main.go              # Sync CLI entry point
  cmd/ha-sync-ui/
    main.go              # Dashboard HTTP server entry point (serves ha-sync.vandachevici.ro)
  internal/
    config/
      config.go          # Config struct, defaults, validation (shared by both binaries)
    db/
      db.go              # MySQL connect, auto-migrate schema
      logging.go         # StartIteration, FinishIteration, BulkInsertOperations, LastDryRunOps
    lease/
      lease.go           # Acquire/release/heartbeat Kubernetes Lease object
    opslog/
      writer.go          # Append JSON lines to /var/log/ha-sync/opslog-<pair>-<direction>-<RFC3339>.jsonl
      flusher.go         # Scan for unprocessed opslogs, batch INSERT; cleanup logs >10 days
    sync/
      engine.go          # Main sync loop: walk, compare, dispatch; dryRun flag skips writes
      walker.go          # Recursive directory walk
      compare.go         # mtime+size comparison; conditional MD5
      copy.go            # File copy with os.Chtimes() mtime preservation
      delete.go          # Safe delete with pre-check
    ui/
      handler.go         # HTTP handlers: index, /api/iterations, /api/operations, /api/pairs
      templates/
        index.html       # Dashboard HTML; auto-refreshes every 10s via fetch(); vanilla JS only
  go.mod
  go.sum
  Dockerfile             # Multi-stage: golang:1.22-alpine builder (builds ha-sync + ha-sync-ui) → alpine:3.20
  Makefile               # build, docker-build IMAGE=<registry>/ha-sync:latest, docker-push targets

deployment/ha-sync/
  serviceaccount.yaml    # ServiceAccount: ha-sync, namespace: infrastructure
  rbac.yaml              # Role + RoleBinding: leases (coordination.k8s.io) create/get/update/delete
  secret.yaml            # NOTE: create manually — see Phase 3C instructions
  pv-logs.yaml           # PersistentVolume: NFS 192.168.2.193:/data/infra/ha-sync-logs, 10Gi, RWX
  pvc-logs.yaml          # PVC bound to pv-logs; all CronJobs mount at /var/log/ha-sync
  pv-dell-<pair>.yaml    # PersistentVolume: NFS 192.168.2.100:/data/<pair>  (one per pair × 6)
  pv-hp-<pair>.yaml      # PersistentVolume: NFS 192.168.2.193:/data/<pair>  (one per pair × 6)
  pvc-dell-<pair>.yaml   # PVC → pv-dell-<pair>   (one per pair × 6)
  pvc-hp-<pair>.yaml     # PVC → pv-hp-<pair>     (one per pair × 6)
  cron-<pair>-dell-to-hp.yaml   # --dry-run is DEFAULT; remove flag to enable production sync
  cron-<pair>-hp-to-dell.yaml   # same
  ui-deployment.yaml     # Deployment: ha-sync-ui, 1 replica, image: <registry>/ha-sync:latest, cmd: ha-sync-ui
  ui-service.yaml        # ClusterIP Service: port 8080 → ha-sync-ui pod
  ui-ingress.yaml        # Ingress: ha-sync.vandachevici.ro → ui-service:8080; cert-manager TLS
  kustomization.yaml     # Kustomize root listing all resources

scripts/cli/
  ha-sync.md             # CLI reference doc
```

### UI Dashboard (`ha-sync.vandachevici.ro`)

- **Binary**: `ha-sync-ui` — Go HTTP server, port 8080
- **Routes**:
  - `GET /` — HTML dashboard; auto-refreshes via `setInterval` + `fetch`
  - `GET /api/pairs` — JSON: per-pair last iteration summary (dry_run=0 and dry_run=1 separately)
  - `GET /api/iterations?pair=&limit=20` — JSON: recent iterations
  - `GET /api/operations?iteration_id=` — JSON: operations for one iteration
- **Dashboard shows**: per-pair status cards (last real sync, last dry-run, files created/updated/deleted/failed), recent activity table, errors highlighted in red
- **Env vars**: `HA_SYNC_DB_DSN` (same secret as CronJobs)
- **K8s**: Deployment in `infrastructure` namespace, 1 replica, same ServiceAccount as CronJobs (read-only DB access only)

---

## Tasks

> **Parallelism key**: Tasks marked `[P]` can be executed in parallel by separate agents. Tasks marked `[SEQ]` must follow the listed dependency chain.

---

### Phase 0 — Scaffolding `[SEQ]`

Must complete before any code is written; all subsequent tasks depend on this.

| # | Task | Command / Notes |
|---|---|---|
| 0.1 | Create Go module | `cd services/ha-sync && go mod init github.com/vandachevici/homelab/ha-sync` |
| 0.2 | Create directory tree | `mkdir -p cmd/ha-sync internal/{config,db,lease,opslog,sync}` |
| 0.3 | Create Dockerfile | Multi-stage: `FROM golang:1.22-alpine AS build` → `FROM alpine:3.20`; copy binary; `ENTRYPOINT ["/ha-sync"]` |
| 0.4 | Create Makefile | Targets: `build`, `docker-build IMAGE=<registry>/ha-sync:latest`, `docker-push IMAGE=...` |

---

### Phase 1 — Core Go packages `[P after Phase 0]`

Sub-tasks 1A, 1B, 1C, 1E are **fully independent** — assign to separate agents simultaneously. 1D depends on all of them.

#### 1A — `internal/config` `[P]`
| # | Task | Notes |
|---|---|---|
| 1A.1 | Write `config.go` | Define `Config` struct with all CLI flags; use `flag` stdlib or `cobra`; set defaults from CLI Interface section above |

#### 1B — `internal/db` `[P]`
| # | Task | Notes |
|---|---|---|
| 1B.1 | Write `db.go` | `Connect(dsn string) (*sql.DB, error)`; run `CREATE TABLE IF NOT EXISTS` for both tables (include `dry_run TINYINT(1) NOT NULL DEFAULT 0` column in both) on startup |
| 1B.2 | Write `logging.go` | `StartIteration(dryRun bool, ...) (id int64)` → INSERT with `dry_run` set; `FinishIteration(id, status, counts)` → UPDATE; `BulkInsertOperations(iterID int64, dryRun bool, []OpRecord)` → batch INSERT; `LastDryRunOps(db, pair, direction string) ([]OpRecord, error)` → fetch ops for last successful `dry_run=1` iteration for idempotency check |

#### 1C — `internal/lease` `[P]`
| # | Task | Notes |
|---|---|---|
| 1C.1 | Write `lease.go` | Use `k8s.io/client-go` in-cluster config; `Acquire(ctx, client, namespace, leaseName, holderID, ttlSec)` — create or update Lease if expired; `Release(ctx, client, namespace, leaseName, holderID)` — delete Lease; `Heartbeat(ctx, ...)` — goroutine that calls `Update` on `spec.renewTime` every `ttlSec/3` seconds |

#### 1D — `internal/opslog` `[P]`
| # | Task | Notes |
|---|---|---|
| 1D.1 | Write `writer.go` | `Open(logDir, pair, direction string) (*Writer, error)` — creates `/var/log/ha-sync/opslog-<pair>-<direction>-<RFC3339>.jsonl`; `Append(op OpRecord) error` — JSON-encode one line |
| 1D.2 | Write `flusher.go` | `FlushAll(logDir string, db *sql.DB) error` — scan dir for `*.jsonl`, for each: decode lines → call `BulkInsertOperations`, delete file on success; `CleanOld(logDir string, retainDays int)` — delete files with mtime older than N days |

#### 1E — `internal/sync` `[P]`
| # | Task | Notes |
|---|---|---|
| 1E.1 | Write `walker.go` | `Walk(root string) ([]FileInfo, error)` — returns slice of `{RelPath, AbsPath, Size, ModTime, IsDir}`; use `filepath.WalkDir` |
| 1E.2 | Write `compare.go` | `NeedsSync(src, dest FileInfo, threshold time.Duration) bool` — mtime+size check; `MD5File(path string) (string, error)` — streaming MD5; `MD5Changed(srcPath, destPath string) bool` |
| 1E.3 | Write `copy.go` | `CopyFile(src, dest string, srcModTime time.Time) error` — copy bytes, then `os.Chtimes(dest, srcModTime, srcModTime)` to preserve mtime |
| 1E.4 | Write `delete.go` | `DeleteFile(path string) error` — `os.Remove`; `DeleteDir(path string) error` — `os.RemoveAll` only if dir is empty after child removal |
| 1E.5 | Write `engine.go` | Walk src+dest, compare, dispatch create/update/delete via worker pool (`sync.WaitGroup` + buffered channel of `--workers` size); if `dryRun=true`, build op list but **do not call copy/delete** — return ops for caller to log; write each op to opslog.Writer (tagged with dry_run flag); return summary counts |

#### 1F — `cmd/ha-sync/main.go` `[SEQ, depends on 1A+1B+1C+1D+1E]`
| # | Task | Notes |
|---|---|---|
| 1F.1 | Write `main.go` | Parse flags → build config → connect DB → flush old opslogs → acquire Lease → **if `--dry-run`: call `LastDryRunOps`, walk src+dest, compute would-be ops, compare; if identical → print "already current" + exit; else run engine(dryRun=true)** → open opslog writer (tagged dry_run) → start iteration row (`dry_run` = true/false) → run engine → finish iteration → flush opslog to DB → release Lease; trap SIGTERM to release Lease before exit; **production queries always filter `dry_run = 0`** |

---

### Phase 2 — Build & Docker Image `[SEQ after Phase 1]`

| # | Task | Command |
|---|---|---|
| 2.1 | Fetch Go deps | `cd services/ha-sync && go mod tidy` |
| 2.2 | Build binary | `cd services/ha-sync && make build` |
| 2.3 | Build Docker image | `make docker-build IMAGE=192.168.2.100:5000/ha-sync:latest` *(replace registry if different)* |
| 2.4 | Push Docker image | `make docker-push IMAGE=192.168.2.100:5000/ha-sync:latest` |

---

### Phase 3 — Kubernetes Manifests `[P, can start during Phase 1]`

All manifest sub-tasks are **independent** and can be parallelized.

#### 3A — RBAC + Shared Resources `[P]`
| # | Task | Notes |
|---|---|---|
| 3A.1 | Create `serviceaccount.yaml` | `name: ha-sync`, `namespace: infrastructure` |
| 3A.2 | Create `rbac.yaml` | `Role` with rules: `apiGroups: [coordination.k8s.io]`, `resources: [leases]`, `verbs: [create, get, update, delete]`; `RoleBinding` binding `ha-sync` SA to the Role |
| 3A.3 | Create `pv-logs.yaml` + `pvc-logs.yaml` | PV: `nfs.server: 192.168.2.193`, `nfs.path: /data/infra/ha-sync-logs`, capacity `10Gi`, `accessModes: [ReadWriteMany]`; PVC: `storageClassName: ""`, `volumeName: pv-ha-sync-logs`, namespace `infrastructure` |

#### 3B — PVs and PVCs per pair `[P]`
| # | Task | Notes |
|---|---|---|
| 3B.1 | Create `pv-dell-<pair>.yaml` for each of 6 pairs | `spec.nfs.server: 192.168.2.100`, `spec.nfs.path: /data/<pair>`; capacity per pair: `media: 2Ti`, `photos: 500Gi`, `games: 500Gi`, `owncloud: 500Gi`, `infra: 100Gi`, `ai: 500Gi`; `accessModes: [ReadWriteMany]` |
| 3B.2 | Create `pv-hp-<pair>.yaml` for each of 6 pairs | Same structure; `spec.nfs.server: 192.168.2.193` |
| 3B.3 | Create `pvc-dell-<pair>.yaml` + `pvc-hp-<pair>.yaml` | `namespace: infrastructure`; `accessModes: [ReadWriteMany]`; `storageClassName: ""` (manual bind); `volumeName: pv-dell-<pair>` / `pv-hp-<pair>` |

#### 3C — CronJobs `[P, depends on 3A+3B for volume/SA names]`
| # | Task | Notes |
|---|---|---|
| 3C.1 | Create `cron-<pair>-dell-to-hp.yaml` for each pair | `namespace: infrastructure`; `serviceAccountName: ha-sync`; `schedule: "*/15 * * * *"`; image: `<registry>/ha-sync:latest`; args: `["--src=/mnt/dell/<pair>","--dest=/mnt/hp/<pair>","--pair=<pair>","--direction=dell-to-hp","--db-dsn=$(HA_SYNC_DB_DSN)","--log-dir=/var/log/ha-sync"]`; volumeMounts: `pvc-dell-<pair>` → `/mnt/dell/<pair>`, `pvc-hp-<pair>` → `/mnt/hp/<pair>`, `pvc-ha-sync-logs` → `/var/log/ha-sync`; envFrom: `ha-sync-db-secret` |
| 3C.2 | Create `cron-<pair>-hp-to-dell.yaml` for each pair | Same but src/dest swapped, `direction=hp-to-dell`; offset schedule by 7 min: `"7,22,37,52 * * * *"` |
| 3C.3 | Create `secret.yaml` | Comment-only file; actual secret created manually: `kubectl create secret generic ha-sync-db-secret --from-literal=HA_SYNC_DB_DSN='<user>:<pass>@tcp(general-purpose-db.infrastructure.svc.cluster.local:3306)/general_db' -n infrastructure` |
| 3C.4 | Create `kustomization.yaml` | Resources in order: `serviceaccount.yaml`, `rbac.yaml`, `pv-logs.yaml`, `pvc-logs.yaml`, all `pv-*.yaml`, all `pvc-*.yaml`, all `cron-*.yaml` |

---

### Phase 4 — CLI Documentation `[P, independent]`

| # | Task | Notes |
|---|---|---|
| 4.1 | Create `scripts/cli/ha-sync.md` | Document all flags, defaults, example invocations, env vars (`HA_SYNC_DB_DSN`); note `--dry-run` for safe first-run; note `--delete-missing` rollout guidance |

---

### Phase 5 — Deploy & Verify `[SEQ after Phase 2+3]`

| # | Task | Command |
|---|---|---|
| 5.1 | Create DB secret | `kubectl create secret generic ha-sync-db-secret --from-literal=HA_SYNC_DB_DSN='<user>:<pass>@tcp(general-purpose-db.infrastructure.svc.cluster.local:3306)/general_db' -n infrastructure` |
| 5.2 | Apply manifests | `kubectl apply -k deployment/ha-sync/` |
| 5.3 | Dry-run smoke test | `kubectl create job ha-sync-test --from=cronjob/ha-sync-media-dell-to-hp -n infrastructure` then: `kubectl logs -l job-name=ha-sync-test -n infrastructure -f` |
| 5.4 | Verify Lease is created | `kubectl get lease ha-sync-media -n infrastructure -o yaml` |
| 5.5 | Verify DB rows | `kubectl exec -it <general-purpose-db-pod> -n infrastructure -- mysql -u<user> -p general_db -e "SELECT * FROM sync_iterations ORDER BY id DESC LIMIT 5;"` |
| 5.6 | Verify opslog flush | Check `/var/log/ha-sync/` on the logs PVC — no `.jsonl` files should remain after a successful run |
| 5.7 | Trigger real first run | Delete the test job; let CronJob run on schedule; observe `sync_operations` table |

---

## Open Questions / Future Work

- **MySQL HA**: `general-purpose-db` is a single-replica StatefulSet — no HA. Since locking is now handled by K8s Lease and MySQL is only used for audit logging (with opslog fallback), a MySQL outage won't block sync. If full MySQL HA is later desired, **MariaDB Galera Cluster (3 replicas)** is the recommended path for this homelab.
- **Conflict resolution**: Currently "newest mtime wins". If clocks drift between nodes, a file could ping-pong. Consider NTP enforcement across all nodes or use `--mtime-threshold` >= observed clock skew.
- **Delete safety**: `--delete-missing` defaults to `false`. Staged rollout: run one full cycle disabled first → confirm parity → enable on primary direction only.
- **Alerting**: Add a Prometheus/Grafana alert on `sync_iterations.status = 'failed'` (query general_db directly or expose a future `/metrics` endpoint).
- **DB retention**: `sync_operations` will grow large. Add a cleanup step: `DELETE FROM sync_operations WHERE started_at < NOW() - INTERVAL 30 DAY` as a weekly CronJob.
- **Registry**: Dockerfile assumes local registry at `192.168.2.100:5000`. Confirm registry address before Phase 2.