ochenstarik/server-monitor-manager
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases 17 Packages Wiki Activity Actions 2

fix(verification): verify releases the way an operator does

Browse source 
PR #40 correctly publishes the keyless signing certificate and makes
verify_archive require manifest, signature and certificate together. Three
things around it were inconsistent.

The release-verification scripts fetched assets with `gh release download`.
The workspace-isolation step deliberately removes `.git` so that the install
comes from the release rather than the source tree, and `gh` resolves the
repository from that git context — the tool fought the isolation it runs
inside, which is what broke the first runs. An operator has neither `gh` nor a
token, so the scripts now use public `curl`, `sha256sum` and `cosign` only.
`verify-assets.sh` keeps `gh`: it runs before isolation and only lists assets.

The monitor check addressed a user and a home directory that do not exist.
The bootstrap creates `ochenstarik-monitor` with `/var/lib/ochenstarik-monitor`;
the script used `ochenstarik-smm-monitor` under the Control state directory, so
that section could never have passed. It now also asserts that the forced
command is pinned in `authorized_keys` before running it, and quotes the
command instead of splitting it on whitespace.

The expected asset list did not include the new certificate, so a correct
release would have been reported as unexpected.

Also: a negative case for an archive published without its certificate, removal
of drafting comments that quoted the task text, and documentation of the three
signature files, since `verify-release` now requires them beside the archive and
the documented download list stopped being sufficient.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Ochenstarik 2026-08-15 13:16:27 +07:00
parent 55cfdcffac
commit c1e47684e6
4 changed files with 129 additions and 81 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
docs/linux-bootstrap.md
View file

@ -10,9 +10,14 @@ Server Monitor Manager устанавливает Control (Hub) и Agent (Node)
- `ochenstarik-server-monitor-manager.sh` и `.sha256`; - `ochenstarik-server-monitor-manager.sh` и `.sha256`;
- `server-monitor-manager-linux-x64.tar.gz` или `server-monitor-manager-linux-arm64.tar.gz`; - `server-monitor-manager-linux-x64.tar.gz` или `server-monitor-manager-linux-arm64.tar.gz`;
- соответствующий `.tar.gz.sha256`. - соответствующий `.tar.gz.sha256`;
- `server-monitor-manager-manifest.json`, `server-monitor-manager-manifest.sig` и `server-monitor-manager-manifest.pem`.
Bootstrap проверяет SHA-256 до распаковки и принимает в архиве только каталоги `agent`, `control`, `deploy` и `bootstrap`. Все файлы должны лежать в одном каталоге: `verify-release` ищет manifest, подпись и сертификат рядом с архивом. Без любого из трёх проверка отказывает и предлагает явный обход `SMM_ALLOW_UNSIGNED=1` — он предназначен только для сборок, выпущенных до появления подписи, и в обычной установке не используется.
Сертификат нужен потому, что manifest подписывается keyless-режимом cosign: подпись проверяется эфемерным сертификатом, привязанным к workflow выпуска, а не постоянным ключом.
Bootstrap проверяет подпись manifest, затем SHA-256 архива по manifest, и принимает в архиве только каталоги `agent`, `control`, `deploy` и `bootstrap`.
## 1. Установка главного сервера (Hub) ## 1. Установка главного сервера (Hub)

83
tests/release-verification/run-negative-tests.sh
View file

@ -1,67 +1,94 @@
#!/bin/bash #!/bin/bash
# Every tampering scenario an operator could hit must be rejected by the release
# artefacts themselves. Downloads use public curl only: gh needs git context that
# the isolated workspace removes, and the operator has neither gh nor a token.
set -euo pipefail set -euo pipefail
TAG="${1:-}" TAG="${1:-}"
REPOSITORY="${SMM_REPOSITORY:-ochenstarik-ui/server-monitor-manager}"
if [[ -z "$TAG" ]]; then if [[ -z "$TAG" ]]; then
echo "Usage: $0 <tag>" echo "Usage: $0 <tag>" >&2
exit 1 exit 1
fi fi
BASE_URL="https://github.com/${REPOSITORY}/releases/download/${TAG}"
echo "Running negative tests against release $TAG..." echo "Running negative tests against release $TAG..."
# We will need smm-setup.sh or ochenstarik-server-monitor-manager.sh download() {
# We'll download ochenstarik-server-monitor-manager.sh directly to test verify-release local name="$1"
gh release download "$TAG" -p 'ochenstarik-server-monitor-manager.sh' curl --fail --silent --show-error --location --retry 3 \
chmod +x ochenstarik-server-monitor-manager.sh -o "$name" "${BASE_URL}/${name}" \
|| { echo "FAIL: asset is not downloadable: $name" >&2; exit 1; }
}
ARCHIVE="server-monitor-manager-linux-$(uname -m | sed -e 's/x86_64/x64/' -e 's/aarch64/arm64/').tar.gz" case "$(uname -m)" in
gh release download "$TAG" -p "$ARCHIVE" x86_64) RUNTIME="linux-x64" ;;
gh release download "$TAG" -p "server-monitor-manager-manifest.json" aarch64|arm64) RUNTIME="linux-arm64" ;;
gh release download "$TAG" -p "server-monitor-manager-manifest.sig" *) echo "FAIL: unsupported architecture $(uname -m)" >&2; exit 1 ;;
gh release download "$TAG" -p "server-monitor-manager-manifest.pem" esac
ARCHIVE="server-monitor-manager-${RUNTIME}.tar.gz"
download ochenstarik-server-monitor-manager.sh
chmod +x ochenstarik-server-monitor-manager.sh
download "$ARCHIVE"
download "$ARCHIVE.sha256"
download server-monitor-manager-manifest.json
download server-monitor-manager-manifest.sig
download server-monitor-manager-manifest.pem
echo "Test 1: Altered byte in archive" echo "Test 1: Altered byte in archive"
cp "$ARCHIVE" "corrupted-$ARCHIVE" cp "$ARCHIVE" "corrupted-$ARCHIVE"
echo "corrupted" >> "corrupted-$ARCHIVE" cp "$ARCHIVE.sha256" "corrupted-$ARCHIVE.sha256"
echo "corrupted" >>"corrupted-$ARCHIVE"
if ./ochenstarik-server-monitor-manager.sh verify-release "corrupted-$ARCHIVE" >/dev/null 2>&1; then if ./ochenstarik-server-monitor-manager.sh verify-release "corrupted-$ARCHIVE" >/dev/null 2>&1; then
echo "FAIL: Altered archive was accepted!" echo "FAIL: Altered archive was accepted!" >&2
exit 1 exit 1
fi fi
echo "PASS: Altered archive rejected." echo "PASS: Altered archive rejected."
rm "corrupted-$ARCHIVE" rm -f "corrupted-$ARCHIVE" "corrupted-$ARCHIVE.sha256"
echo "Test 2: Substituted hash in manifest without resigning" echo "Test 2: Substituted hash in manifest without resigning"
cp server-monitor-manager-manifest.json corrupted-manifest.json cp server-monitor-manager-manifest.json corrupted-manifest.json
# Replace all hashes with zeros
sed -i 's/"[a-f0-9]\{64\}"/"0000000000000000000000000000000000000000000000000000000000000000"/g' corrupted-manifest.json sed -i 's/"[a-f0-9]\{64\}"/"0000000000000000000000000000000000000000000000000000000000000000"/g' corrupted-manifest.json
# Test verify-manifest directly if ./ochenstarik-server-monitor-manager.sh verify-manifest corrupted-manifest.json \
if ./ochenstarik-server-monitor-manager.sh verify-manifest corrupted-manifest.json server-monitor-manager-manifest.sig server-monitor-manager-manifest.pem >/dev/null 2>&1; then server-monitor-manager-manifest.sig server-monitor-manager-manifest.pem >/dev/null 2>&1; then
echo "FAIL: Manifest with substituted hash accepted!" echo "FAIL: Manifest with substituted hash accepted!" >&2
exit 1 exit 1
fi fi
echo "PASS: Substituted hash rejected." echo "PASS: Substituted hash rejected."
rm corrupted-manifest.json rm -f corrupted-manifest.json
echo "Test 3: Manifest without signature" echo "Test 3: Manifest without signature"
# We just pass an empty string for the signature file argument if ./ochenstarik-server-monitor-manager.sh verify-manifest server-monitor-manager-manifest.json \
if ./ochenstarik-server-monitor-manager.sh verify-manifest server-monitor-manager-manifest.json "" server-monitor-manager-manifest.pem >/dev/null 2>&1; then "" server-monitor-manager-manifest.pem >/dev/null 2>&1; then
echo "FAIL: Manifest without signature accepted!" echo "FAIL: Manifest without signature accepted!" >&2
exit 1 exit 1
fi fi
echo "PASS: Missing signature rejected." echo "PASS: Missing signature rejected."
echo "Test 4: Signature made by another identity" echo "Test 4: Signature made by another identity"
# Generate a local keypair and sign the manifest
export COSIGN_PASSWORD="" export COSIGN_PASSWORD=""
cosign generate-key-pair cosign generate-key-pair >/dev/null
cosign sign-blob --yes --key cosign.key --output-signature fake.sig server-monitor-manager-manifest.json cosign sign-blob --yes --key cosign.key \
# Verification must fail because ochenstarik-server-monitor-manager.sh enforces keyless OIDC identity! --output-signature fake.sig server-monitor-manager-manifest.json >/dev/null
if ./ochenstarik-server-monitor-manager.sh verify-manifest server-monitor-manager-manifest.json fake.sig server-monitor-manager-manifest.pem >/dev/null 2>&1; then if ./ochenstarik-server-monitor-manager.sh verify-manifest server-monitor-manager-manifest.json \
echo "FAIL: Signature from wrong identity accepted!" fake.sig server-monitor-manager-manifest.pem >/dev/null 2>&1; then
echo "FAIL: Signature from wrong identity accepted!" >&2
exit 1 exit 1
fi fi
echo "PASS: Wrong identity signature rejected." echo "PASS: Wrong identity signature rejected."
rm cosign.key cosign.pub fake.sig rm -f cosign.key cosign.pub fake.sig
echo "Test 5: Missing certificate beside the archive"
mkdir -p no-cert && cp "$ARCHIVE" "$ARCHIVE.sha256" \
server-monitor-manager-manifest.json server-monitor-manager-manifest.sig no-cert/
if ./ochenstarik-server-monitor-manager.sh verify-release "no-cert/$ARCHIVE" >/dev/null 2>&1; then
echo "FAIL: Archive accepted without the signing certificate!" >&2
exit 1
fi
echo "PASS: Missing certificate rejected."
rm -rf no-cert
echo "All negative tests passed!" echo "All negative tests passed!"

117
tests/release-verification/run-positive-installation.sh
View file

@ -1,91 +1,106 @@
#!/bin/bash #!/bin/bash
# Installs a published release exactly the way an operator does it: public curl
# downloads, checksum verification, signature verification, then the documented
# bootstrap commands. No gh CLI and no token, because the operator has neither —
# and because gh resolves the repository from git context, which the isolated
# workspace deliberately removes.
set -euo pipefail set -euo pipefail
TAG="${1:-}" TAG="${1:-}"
REPOSITORY="${SMM_REPOSITORY:-ochenstarik-ui/server-monitor-manager}"
if [[ -z "$TAG" ]]; then if [[ -z "$TAG" ]]; then
echo "Usage: $0 <tag>" echo "Usage: $0 <tag>" >&2
exit 1 exit 1
fi fi
BASE_URL="https://github.com/${REPOSITORY}/releases/download/${TAG}"
CONTROL_PORT=17443
MONITOR_USER="ochenstarik-monitor"
MONITOR_HOME="/var/lib/ochenstarik-monitor"
METRICS_SCRIPT="/usr/local/libexec/ochenstarik-smm-metrics"
echo "Running positive installation test for $TAG..." echo "Running positive installation test for $TAG..."
# Fetch smm-setup.sh download() {
gh release download "$TAG" -p 'smm-setup.sh*' local name="$1"
curl --fail --silent --show-error --location --retry 3 \
-o "$name" "${BASE_URL}/${name}" \
|| { echo "FAIL: asset is not downloadable: $name" >&2; exit 1; }
}
# Verify checksum case "$(uname -m)" in
x86_64) RUNTIME="linux-x64" ;;
aarch64|arm64) RUNTIME="linux-arm64" ;;
*) echo "FAIL: unsupported architecture $(uname -m)" >&2; exit 1 ;;
esac
ARCHIVE="server-monitor-manager-${RUNTIME}.tar.gz"
download smm-setup.sh
download smm-setup.sh.sha256
sha256sum -c smm-setup.sh.sha256 sha256sum -c smm-setup.sh.sha256
# The archive is downloaded by verify-release or we must download it? download "$ARCHIVE"
# In smm-setup.sh, the owner manually downloads the archive? download "$ARCHIVE.sha256"
# Wait, let's look at docs: "загрузка bootstrap и архива из релиза, проверка контрольных сумм, проверка подписи manifest"
# Actually, the user does:
ARCHIVE="server-monitor-manager-linux-$(uname -m | sed -e 's/x86_64/x64/' -e 's/aarch64/arm64/').tar.gz"
gh release download "$TAG" -p "$ARCHIVE*"
gh release download "$TAG" -p "server-monitor-manager-manifest.json"
gh release download "$TAG" -p "server-monitor-manager-manifest.sig"
gh release download "$TAG" -p "server-monitor-manager-manifest.pem"
sha256sum -c "$ARCHIVE.sha256" sha256sum -c "$ARCHIVE.sha256"
# Run setup steps through smm-setup.sh # Signature material must sit beside the archive: verify_archive looks for it there.
# "preflight, verify-release, установка Control, mesh-init" download server-monitor-manager-manifest.json
sudo bash smm-setup.sh preflight download server-monitor-manager-manifest.sig
sudo bash smm-setup.sh verify-manifest server-monitor-manager-manifest.json server-monitor-manager-manifest.sig server-monitor-manager-manifest.pem download server-monitor-manager-manifest.pem
sudo bash smm-setup.sh verify-release "$ARCHIVE"
sudo bash smm-setup.sh install-control "$ARCHIVE" 127.0.0.1 17443 sudo bash smm-setup.sh --tag "$TAG" preflight
sudo bash smm-setup.sh mesh-init 127.0.0.1 51820 sudo bash smm-setup.sh --tag "$TAG" verify-manifest \
server-monitor-manager-manifest.json \
server-monitor-manager-manifest.sig \
server-monitor-manager-manifest.pem
sudo bash smm-setup.sh --tag "$TAG" verify-release "$ARCHIVE"
sudo bash smm-setup.sh --tag "$TAG" install-control "$ARCHIVE" 127.0.0.1 "$CONTROL_PORT"
sudo bash smm-setup.sh --tag "$TAG" mesh-init 127.0.0.1 51820
echo "Checking Control healthz..." echo "Checking Control healthz..."
for _ in {1..30}; do for _ in {1..30}; do
if sudo curl --fail --silent \ if sudo curl --fail --silent \
--cacert /etc/ochenstarik-server-monitor-manager/control-ca.crt \ --cacert /etc/ochenstarik-server-monitor-manager/control-ca.crt \
"https://127.0.0.1:17443/healthz" >/dev/null; then "https://127.0.0.1:${CONTROL_PORT}/healthz" >/dev/null; then
break break
fi fi
sleep 1 sleep 1
done done
sudo curl --fail --silent --show-error \ sudo curl --fail --silent --show-error \
--cacert /etc/ochenstarik-server-monitor-manager/control-ca.crt \ --cacert /etc/ochenstarik-server-monitor-manager/control-ca.crt \
"https://127.0.0.1:17443/healthz" "https://127.0.0.1:${CONTROL_PORT}/healthz"
echo "Extracting node code and installing agent..." echo "Enrolling a node..."
NODE_CODE=$(sudo bash smm-setup.sh node-code test-node) NODE_CODE="$(sudo bash smm-setup.sh --tag "$TAG" node-code test-node)"
export SMM_ENROLL_CODE="$NODE_CODE" SMM_ENROLL_CODE="$NODE_CODE" SMM_ACCEPT_CA_FINGERPRINT=1 \
export SMM_ACCEPT_CA_FINGERPRINT=1 sudo --preserve-env=SMM_ENROLL_CODE,SMM_ACCEPT_CA_FINGERPRINT \
sudo --preserve-env=SMM_ENROLL_CODE,SMM_ACCEPT_CA_FINGERPRINT bash smm-setup.sh install-node "$ARCHIVE" bash smm-setup.sh --tag "$TAG" install-node "$ARCHIVE"
sudo systemctl is-active --quiet ochenstarik-smm-agent.service sudo systemctl is-active --quiet ochenstarik-smm-agent.service
sudo systemctl is-active --quiet ochenstarik-smm-control.service sudo systemctl is-active --quiet ochenstarik-smm-control.service
# Verify install-monitor echo "Installing monitor role..."
echo "Installing monitor..." ssh-keygen -t ed25519 -N "" -f /tmp/monitor_key -q
# Generate a dummy SSH key for the test sudo bash smm-setup.sh --tag "$TAG" install-monitor "$(cat /tmp/monitor_key.pub)"
ssh-keygen -t ed25519 -N "" -f /tmp/monitor_key
MONITOR_PUB=$(cat /tmp/monitor_key.pub)
sudo bash smm-setup.sh install-monitor "$MONITOR_PUB"
echo "Verifying monitor user and forced command..." echo "Verifying monitor snapshot against the contract..."
# Run SSH locally as the monitor user (assuming ssh is configured, but actually we can just su into the user or run the forced command directly) sudo grep -Fq "command=\"${METRICS_SCRIPT}\"" "${MONITOR_HOME}/.ssh/authorized_keys" \
# The forced command is likely defined in ~smm-monitor/.ssh/authorized_keys || { echo "FAIL: forced command is not pinned in authorized_keys" >&2; exit 1; }
MONITOR_CMD=$(sudo cat /var/lib/ochenstarik-server-monitor-manager/monitor/.ssh/authorized_keys | grep -o 'command="[^"]*"' | cut -d'"' -f2)
SNAPSHOT=$(sudo -u ochenstarik-smm-monitor $MONITOR_CMD)
# Simple validation of snapshot fields (since actual values vary, we just check keys) SNAPSHOT="$(sudo -u "$MONITOR_USER" "$METRICS_SCRIPT")"
EXPECTED_KEYS=$(cat tests/contracts/monitor-snapshot-v1.txt | cut -d'=' -f1 | sort) EXPECTED_KEYS="$(cut -d'=' -f1 tests/contracts/monitor-snapshot-v1.txt | sort)"
ACTUAL_KEYS=$(echo "$SNAPSHOT" | cut -d'=' -f1 | sort) ACTUAL_KEYS="$(cut -d'=' -f1 <<<"$SNAPSHOT" | sort)"
if [[ "$EXPECTED_KEYS" == "$ACTUAL_KEYS" ]]; then if [[ "$EXPECTED_KEYS" != "$ACTUAL_KEYS" ]]; then
echo "Monitor snapshot keys match contract." echo "FAIL: monitor snapshot keys do not match the contract" >&2
else diff <(echo "$EXPECTED_KEYS") <(echo "$ACTUAL_KEYS") >&2 || true
echo "Monitor snapshot keys mismatch!"
diff <(echo "$EXPECTED_KEYS") <(echo "$ACTUAL_KEYS") || true
exit 1 exit 1
fi fi
echo "PASS: monitor snapshot matches the contract"
# Verify uninstall sudo bash smm-setup.sh --tag "$TAG" uninstall-monitor
sudo bash smm-setup.sh uninstall-monitor sudo bash smm-setup.sh --tag "$TAG" uninstall-agent --purge
sudo bash smm-setup.sh uninstall-agent --purge sudo bash smm-setup.sh --tag "$TAG" uninstall-control --confirm-destroy-control
sudo bash smm-setup.sh uninstall-control --confirm-destroy-control
echo "Positive installation test passed!" echo "Positive installation test passed!"

1
tests/release-verification/verify-assets.sh
View file

@ -30,6 +30,7 @@ smm-setup.sh
smm-setup.sh.sha256 smm-setup.sh.sha256
server-monitor-manager-manifest.json server-monitor-manager-manifest.json
server-monitor-manager-manifest.sig server-monitor-manager-manifest.sig
server-monitor-manager-manifest.pem
EOF EOF
) )