Add pve-backup-watch 0.0.1

A Go CLI that waits for a Proxmox VE vzdump backup to finish and
then restarts the guest.

- Take one or more VMIDs and poll the node's vzdump tasks via pvesh
- Handle QEMU VMs (qm) and LXC containers (pct): unlock, graceful
  shutdown with forced stop after 120s, then start
- Restart after any result by default; -only-failed restarts only when
  the backup did not finish OK, and -warn-ok treats warnings as success
- Ignore backups that finished before the tool started
- Add -dry-run, -timeout, -interval and -node options
- Exit with 0, 1 (failure or timeout) or 2 (usage error)
- Embed version, commit and build date via ldflags (VERSION file, Makefile)
- Add README and CHANGELOG

CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+Versioning follows [Semantic Versioning](https://semver.org). The current
+version lives in the `VERSION` file and is embedded into the binary at build time.
+
+## 1.0.0
+- Initial release.
+- Waits for the vzdump task of the given VMID(s) to finish, then restarts the guest (QEMU or LXC).
+- Flags: `-dry-run`, `-timeout`, `-interval`, `-only-failed`, `-warn-ok`, `-node`, `-version`.

Makefile

@@ -0,0 +1,12 @@
+BINARY  := proxmox-backup-watch
+VERSION := $(shell cat VERSION)
+COMMIT  := $(shell git rev-parse --short HEAD 2>/dev/null || echo none)
+DATE    := $(shell date -u +%Y-%m-%dT%H:%M:%SZ)
+LDFLAGS := -s -w -X main.version=$(VERSION) -X main.commit=$(COMMIT) -X main.date=$(DATE)
+
+.PHONY: build clean
+build:
+	GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "$(LDFLAGS)" -o $(BINARY) .
+
+clean:
+	rm -f $(BINARY)

README.md

@@ -1,3 +1,150 @@
-# pve-backup-watcher
+# proxmox-backup-watch
 
-A small Go CLI for Proxmox VE that waits for a VM or container’s vzdump backup to finish, then restarts the guest.
+Wait for a Proxmox VE backup to finish, then restart the guest.
+
+`proxmox-backup-watch` is a small, dependency-free Go CLI that runs on a Proxmox VE node. You give it one or more VMIDs. It watches the node's task list until the vzdump backup for each guest has finished, then restarts that guest. You don't need to know when the backup job will run or how long it will take.
+
+## Contents
+
+- [Features](#features)
+- [Requirements](#requirements)
+- [Install](#install)
+- [Usage](#usage)
+- [Options](#options)
+- [How it works](#how-it-works)
+- [Exit codes](#exit-codes)
+- [Examples](#examples)
+- [Things to know](#things-to-know)
+- [Versioning](#versioning)
+
+## Features
+
+- Acts only on the VMIDs you name, never on the whole node
+- Works with both QEMU VMs (`qm`) and LXC containers (`pct`)
+- Restarts after any result by default, or only after a failed backup with `-only-failed`
+- Ignores old results, so a failure from last week never triggers a restart
+- Dry-run mode to preview exactly what would happen
+- Configurable poll interval and timeout
+- Stops cleanly on Ctrl-C or SIGTERM
+- Meaningful exit codes for use in scripts and cron
+- Single static binary, standard library only
+
+## Requirements
+
+- A Proxmox VE node (run the tool on the node that hosts the guest)
+- Root, or a user allowed to run `pvesh`, `qm` and `pct`
+- `pvesh`, `qm` and `pct` on the `PATH` (standard on Proxmox VE)
+- Go 1.21 or newer, only to build from source
+
+## Install
+
+Build from source and copy the binary to the node:
+
+```bash
+make
+scp proxmox-backup-watch root@your-node:/usr/local/bin/
+```
+
+`make` builds a static `linux/amd64` binary and embeds the version, git commit and build date. If you build on the node itself, or want another architecture, run `go build` directly:
+
+```bash
+go build -o proxmox-backup-watch .
+```
+
+A plain `go build` reports the version as `dev`.
+
+## Usage
+
+```
+proxmox-backup-watch [options] VMID [VMID...]
+```
+
+Options can appear before, between or after the VMIDs. With several VMIDs, they are handled one after another in the order given.
+
+## Options
+
+| Option | Default | Description |
+|---|---|---|
+| `-n`, `-dry-run` | off | Print the restart commands instead of running them. It still waits for the backup. |
+| `-t`, `-timeout` | `6h` | Maximum time to wait per VMID. `0` waits forever. On timeout, that guest is not restarted. |
+| `-i`, `-interval` | `30s` | How often to poll the task list. |
+| `-f`, `-only-failed` | off | Restart only if the backup did not finish with status `OK`. |
+| `-w`, `-warn-ok` | off | With `-only-failed`, treat `WARNINGS: n` as success. |
+| `-node` | short hostname | Proxmox node name, if it differs from the hostname. |
+| `-v`, `-version` | | Print version, commit and build date, then exit. |
+
+Durations use Go syntax: `30s`, `15m`, `6h`.
+
+## How it works
+
+1. Reads the node's recent vzdump tasks with `pvesh get /nodes/<node>/tasks --typefilter vzdump`.
+2. Picks the newest task whose `id` matches the VMID and that is either still running or finished after the tool was started.
+3. Polls until that task has an end time, then reads its `status` (`OK`, `WARNINGS: n`, or an error message).
+4. Decides whether to restart (always by default, or only on failure with `-only-failed`).
+5. Restarts the guest:
+   1. `qm` or `pct` `unlock`, to clear a leftover backup lock (errors are ignored)
+   2. `shutdown --timeout 120 --forceStop 1`, a graceful shutdown with a forced stop after 120 seconds
+   3. `start`
+
+Temporary errors from `pvesh` or JSON parsing don't abort the wait. A warning is printed and the next poll tries again.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | Every VMID was handled without error |
+| `1` | At least one VMID failed: timeout, guest not found, or a failed shutdown or start |
+| `2` | Invalid usage, such as no VMID given or a non-numeric VMID |
+
+## Examples
+
+Preview what would happen for VM 123:
+
+```bash
+proxmox-backup-watch -n 123
+```
+
+Restart VM 123 as soon as its backup finishes, whatever the result:
+
+```bash
+proxmox-backup-watch 123
+```
+
+Restart only if the backup failed, treating warnings as success:
+
+```bash
+proxmox-backup-watch -f -w 123
+```
+
+Watch several guests, checking every 15 seconds and giving up after 2 hours each:
+
+```bash
+proxmox-backup-watch -i 15s -t 2h 123 124 200
+```
+
+Run in the background and keep a log:
+
+```bash
+nohup proxmox-backup-watch 123 &> /var/log/proxmox-backup-watch-123.log &
+```
+
+## Things to know
+
+- **Start it before or during the backup.** It only reacts to a backup that is running or starts after the tool launches. If you start it after the backup has already finished, it waits for the next one (or times out).
+- **vzdump `stop` mode restarts the guest itself.** In that mode, this tool causes a second restart. It is most useful with `snapshot` or `suspend` mode, or when you want a restart after the backup regardless.
+- **A restart will not fix a broken backup.** If backups fail because of storage or network problems, using `-only-failed` on a schedule reboots the guest every time without solving the cause.
+- **Local node only.** It looks at tasks on the node it runs on, and the guest must be on that node.
+- **Recent tasks only.** It reads the latest 500 vzdump tasks on the node, which is plenty for a backup that is running now.
+- **Timeouts never restart.** If the backup doesn't finish in time, the guest is left alone and the tool exits with code `1`.
+
+## Versioning
+
+This project follows [Semantic Versioning](https://semver.org). The current version is in the `VERSION` file and is embedded in the binary at build time. Check it with:
+
+```bash
+proxmox-backup-watch -version
+```
+
+The tool also prints its version at the start of every run, so log files show which version produced them. See `CHANGELOG.md` for the history.
+
+To release a change, update `VERSION`, add an entry to `CHANGELOG.md`, and rebuild with `make`.

VERSION

@@ -0,0 +1 @@
+0.0.1

go.mod

@@ -0,0 +1,3 @@
+module proxmox-backup-watch
+
+go 1.21

main.go

@@ -0,0 +1,276 @@
+// proxmox-backup-watch
+//
+// Waits for the vzdump task of the given VMID(s) to finish, then restarts the
+// guest (QEMU VM or LXC container). By default it restarts regardless of the
+// backup result; use -only-failed to restart only when the backup did not
+// finish OK.
+//
+// Usage: proxmox-backup-watch [options] VMID [VMID...]
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"flag"
+	"fmt"
+	"os"
+	"os/exec"
+	"os/signal"
+	"strings"
+	"syscall"
+	"time"
+)
+
+// Set at build time with -ldflags (see Makefile).
+var (
+	version = "dev"
+	commit  = "none"
+	date    = "unknown"
+)
+
+type task struct {
+	ID        string `json:"id"`
+	StartTime int64  `json:"starttime"`
+	EndTime   *int64 `json:"endtime"`
+	Status    string `json:"status"`
+	UPID      string `json:"upid"`
+}
+
+type config struct {
+	node       string
+	dryRun     bool
+	timeout    time.Duration
+	interval   time.Duration
+	onlyFailed bool
+	warnOK     bool
+	startedAt  int64
+}
+
+func main() {
+	os.Exit(run())
+}
+
+func run() int {
+	cfg := config{startedAt: time.Now().Unix()}
+
+	fs := flag.NewFlagSet("proxmox-backup-watch", flag.ContinueOnError)
+	fs.BoolVar(&cfg.dryRun, "dry-run", false, "show what would be done, don't restart anything")
+	fs.BoolVar(&cfg.dryRun, "n", false, "shorthand for -dry-run")
+	fs.DurationVar(&cfg.timeout, "timeout", 6*time.Hour, "max time to wait per VMID (0 = forever)")
+	fs.DurationVar(&cfg.timeout, "t", 6*time.Hour, "shorthand for -timeout")
+	fs.DurationVar(&cfg.interval, "interval", 30*time.Second, "poll interval")
+	fs.DurationVar(&cfg.interval, "i", 30*time.Second, "shorthand for -interval")
+	fs.BoolVar(&cfg.onlyFailed, "only-failed", false, "restart only if the backup did NOT finish OK")
+	fs.BoolVar(&cfg.onlyFailed, "f", false, "shorthand for -only-failed")
+	fs.BoolVar(&cfg.warnOK, "warn-ok", false, `with -only-failed, treat "WARNINGS: n" as success`)
+	fs.BoolVar(&cfg.warnOK, "w", false, "shorthand for -warn-ok")
+	fs.StringVar(&cfg.node, "node", "", "Proxmox node name (default: short hostname)")
+	var showVersion bool
+	fs.BoolVar(&showVersion, "version", false, "print version and exit")
+	fs.BoolVar(&showVersion, "v", false, "shorthand for -version")
+	fs.Usage = func() {
+		fmt.Fprintln(os.Stderr, "Usage: proxmox-backup-watch [options] VMID [VMID...]")
+		fs.PrintDefaults()
+	}
+
+	// Allow flags before, between and after VMIDs.
+	var vmids []string
+	args := os.Args[1:]
+	for {
+		if err := fs.Parse(args); err != nil {
+			if errors.Is(err, flag.ErrHelp) {
+				return 0
+			}
+			return 2
+		}
+		rest := fs.Args()
+		if len(rest) == 0 {
+			break
+		}
+		vmids = append(vmids, rest[0])
+		args = rest[1:]
+	}
+
+	if showVersion {
+		fmt.Printf("proxmox-backup-watch %s (commit %s, built %s)\n", version, commit, date)
+		return 0
+	}
+
+	if len(vmids) == 0 {
+		fmt.Fprintln(os.Stderr, "Error: at least one VMID is required")
+		fs.Usage()
+		return 2
+	}
+	for _, id := range vmids {
+		if !isNumeric(id) {
+			fmt.Fprintf(os.Stderr, "Error: invalid VMID %q\n", id)
+			return 2
+		}
+	}
+
+	if cfg.node == "" {
+		h, err := os.Hostname()
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Error: cannot determine hostname: %v\n", err)
+			return 1
+		}
+		cfg.node, _, _ = strings.Cut(h, ".")
+	}
+
+	fmt.Printf("proxmox-backup-watch %s\n", version)
+
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
+	defer stop()
+
+	failed := false
+	for _, vmid := range vmids {
+		if err := handle(ctx, &cfg, vmid); err != nil {
+			fmt.Fprintf(os.Stderr, "VM %s: %v\n", vmid, err)
+			failed = true
+			if ctx.Err() != nil {
+				break
+			}
+		}
+	}
+	if failed {
+		return 1
+	}
+	return 0
+}
+
+func handle(ctx context.Context, cfg *config, vmid string) error {
+	fmt.Printf("VM %s: waiting for backup task to finish...\n", vmid)
+
+	t, err := waitForTask(ctx, cfg, vmid)
+	if err != nil {
+		return err
+	}
+
+	if cfg.onlyFailed {
+		ok := t.Status == "OK" || (cfg.warnOK && strings.HasPrefix(t.Status, "WARNINGS"))
+		if ok {
+			fmt.Printf("VM %s: backup finished (%s), no restart needed\n", vmid, t.Status)
+			return nil
+		}
+	}
+
+	fmt.Printf("VM %s: backup finished (%s) -> restarting\n", vmid, t.Status)
+	return restartGuest(ctx, cfg, vmid)
+}
+
+// waitForTask polls until the newest vzdump task for vmid (running now, or
+// finished after this program started) has an end time.
+func waitForTask(ctx context.Context, cfg *config, vmid string) (*task, error) {
+	var deadline time.Time
+	if cfg.timeout > 0 {
+		deadline = time.Now().Add(cfg.timeout)
+	}
+
+	for {
+		t, err := latestTask(ctx, cfg, vmid)
+		if err != nil {
+			// Transient pvesh/JSON errors shouldn't abort a long wait.
+			fmt.Fprintf(os.Stderr, "VM %s: warning: %v\n", vmid, err)
+		} else if t != nil && t.EndTime != nil {
+			return t, nil
+		}
+
+		if !deadline.IsZero() && time.Now().After(deadline) {
+			return nil, fmt.Errorf("timed out after %s, no restart", cfg.timeout)
+		}
+
+		select {
+		case <-ctx.Done():
+			return nil, ctx.Err()
+		case <-time.After(cfg.interval):
+		}
+	}
+}
+
+func latestTask(ctx context.Context, cfg *config, vmid string) (*task, error) {
+	out, err := output(ctx, "pvesh", "get", "/nodes/"+cfg.node+"/tasks",
+		"--typefilter", "vzdump", "--limit", "500", "--output-format", "json")
+	if err != nil {
+		return nil, err
+	}
+
+	var tasks []task
+	if err := json.Unmarshal(out, &tasks); err != nil {
+		return nil, fmt.Errorf("parsing pvesh output: %w", err)
+	}
+
+	var latest *task
+	for i := range tasks {
+		t := &tasks[i]
+		if t.ID != vmid {
+			continue
+		}
+		// Running (no endtime) or finished after we started.
+		if t.EndTime != nil && *t.EndTime < cfg.startedAt {
+			continue
+		}
+		if latest == nil || t.StartTime > latest.StartTime {
+			latest = t
+		}
+	}
+	return latest, nil
+}
+
+func restartGuest(ctx context.Context, cfg *config, vmid string) error {
+	var tool string
+	switch {
+	case exec.CommandContext(ctx, "qm", "status", vmid).Run() == nil:
+		tool = "qm"
+	case exec.CommandContext(ctx, "pct", "status", vmid).Run() == nil:
+		tool = "pct"
+	default:
+		return errors.New("not found on this node")
+	}
+
+	// Clear a leftover 'backup' lock; failure here is not fatal.
+	_ = runCmd(ctx, cfg, tool, "unlock", vmid)
+
+	if err := runCmd(ctx, cfg, tool, "shutdown", vmid, "--timeout", "120", "--forceStop", "1"); err != nil {
+		return fmt.Errorf("shutdown failed: %w", err)
+	}
+	if err := runCmd(ctx, cfg, tool, "start", vmid); err != nil {
+		return fmt.Errorf("start failed: %w", err)
+	}
+	return nil
+}
+
+func runCmd(ctx context.Context, cfg *config, name string, args ...string) error {
+	if cfg.dryRun {
+		fmt.Printf("[dry-run] %s %s\n", name, strings.Join(args, " "))
+		return nil
+	}
+	cmd := exec.CommandContext(ctx, name, args...)
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	return cmd.Run()
+}
+
+func output(ctx context.Context, name string, args ...string) ([]byte, error) {
+	var stdout, stderr bytes.Buffer
+	cmd := exec.CommandContext(ctx, name, args...)
+	cmd.Stdout = &stdout
+	cmd.Stderr = &stderr
+	if err := cmd.Run(); err != nil {
+		return nil, fmt.Errorf("%s: %w: %s", name, err, strings.TrimSpace(stderr.String()))
+	}
+	return stdout.Bytes(), nil
+}
+
+func isNumeric(s string) bool {
+	if s == "" {
+		return false
+	}
+	for _, r := range s {
+		if r < '0' || r > '9' {
+			return false
+		}
+	}
+	return true
+}