Harden Link lifecycle and Control operations
* Harden link lifecycle and control operations * Complete three-server acceptance lifecycle --------- Co-authored-by: Ochenstarik <ochenstarik@inbox.ru>
This commit is contained in:
parent
b3dd5f0a5d
commit
266900115c
21 changed files with 1303 additions and 10 deletions
5
.github/workflows/linux-control-agent.yml
vendored
5
.github/workflows/linux-control-agent.yml
vendored
|
|
@ -32,6 +32,11 @@ jobs:
|
|||
- name: Verify formatting
|
||||
run: dotnet format ServerMonitorManager.slnx --verify-no-changes --no-restore
|
||||
|
||||
- name: Verify three-server acceptance harness
|
||||
run: |
|
||||
bash -n tests/acceptance/three-server-mesh.sh
|
||||
shellcheck --severity=error tests/acceptance/three-server-mesh.sh
|
||||
|
||||
- name: Publish agent amd64
|
||||
run: dotnet publish src/ServerMonitorManager.Agent/ServerMonitorManager.Agent.csproj --configuration Release --runtime linux-x64 --self-contained true -p:PublishSingleFile=true -p:PublishTrimmed=true
|
||||
|
||||
|
|
|
|||
4
.github/workflows/windows-build.yml
vendored
4
.github/workflows/windows-build.yml
vendored
|
|
@ -29,6 +29,10 @@ jobs:
|
|||
- name: Verify formatting
|
||||
run: dotnet format src/ServerMonitorManager.Desktop/ServerMonitorManager.Desktop.csproj --verify-no-changes --no-restore
|
||||
|
||||
- name: Verify Windows desktop contracts
|
||||
shell: pwsh
|
||||
run: ./tests/windows/Test-DesktopContracts.ps1
|
||||
|
||||
- name: Build test-signed MSIX installer
|
||||
shell: pwsh
|
||||
run: |
|
||||
|
|
|
|||
|
|
@ -42,6 +42,8 @@ The control plane and data plane are separated:
|
|||
|
||||
See [architecture](docs/architecture.md), [security model](docs/security-model.md), [roadmap](docs/roadmap.md), and [installer contract](docs/installer-contract.md).
|
||||
|
||||
Operational procedures are documented in [Control backup and recovery](docs/control-backup.md) and the [three-server acceptance test](docs/three-server-acceptance.md).
|
||||
|
||||
## Repository layout
|
||||
|
||||
```text
|
||||
|
|
@ -126,7 +128,7 @@ In the application, generate or copy the monitoring SSH key, add the Hub profile
|
|||
|
||||
The current development branch implements dedicated Windows pages for Servers, Links, Sessions, and Settings; SSH monitoring; the Hub/Node WireGuard installer; directional Links; one-time enrollment; separate mTLS Agent, Operator, and source-scoped Automation identities; certificate revocation/re-enrollment; SQLite control state; audit; authenticated event streaming; Windows Control API integration; and a bounded durable Agent buffer with downsampling.
|
||||
|
||||
Reconnect reconciliation is implemented with a durable SQLite marker: after a Node returns, the Hub reapplies the latest effective disabled policies and clears the marker only after the firewall confirms success. Linux CI exercises the real Control-to-helper process boundary, repeated installation and a systemd reboot, a real WireGuard data path with directional nftables policies, and a 100-Node concurrent heartbeat and replay scenario. The Windows release pipeline produces a signed MSIX and publishes its SHA-256 checksum. Still planned: trusted public code signing and desktop/mobile clients for additional platforms.
|
||||
Reconnect reconciliation is implemented with a durable SQLite marker: after a Node returns, the Hub reapplies the latest effective disabled policies and clears the marker only after the firewall confirms success. Control also expires TTL Links through the firewall helper, prunes bounded operational data, versions its SQLite schema, and creates verified backups of SQLite state and the Control CA. Linux CI exercises the real Control-to-helper process boundary, repeated installation and a systemd reboot, a real WireGuard data path with directional nftables policies, HTTP authorization boundaries, and a 100-Node concurrent heartbeat and replay scenario. The Windows release pipeline produces a signed MSIX and publishes its SHA-256 checksum. Still planned: trusted public code signing and desktop/mobile clients for additional platforms.
|
||||
|
||||
## License and project policy
|
||||
|
||||
|
|
|
|||
42
docs/control-backup.md
Normal file
42
docs/control-backup.md
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
# Control Hub maintenance and recovery
|
||||
|
||||
Control Hub runs maintenance every 15 minutes by default. It removes expired enrollment tokens, old replay records, metrics older than seven days, and audit records older than 90 days. The intervals and retention windows are configured in the `Control` section of `appsettings.json` or with `Control__...` environment variables.
|
||||
|
||||
## Automatic backups
|
||||
|
||||
Every 24 hours Control creates a consistent SQLite online backup together with the Control CA. The default directory is:
|
||||
|
||||
```text
|
||||
/var/lib/ochenstarik-server-monitor-manager/backups
|
||||
```
|
||||
|
||||
Seven backups are retained by default. Every backup directory contains:
|
||||
|
||||
- `control.db` created with the SQLite backup API;
|
||||
- `control-ca.pfx` with owner-only permissions;
|
||||
- `manifest.json` with SHA-256 hashes and format version.
|
||||
|
||||
Create a backup immediately without interrupting the running service:
|
||||
|
||||
```bash
|
||||
sudo systemd-run --wait --pipe --quiet --collect \
|
||||
--uid=ochenstarik-smm-control \
|
||||
--gid=ochenstarik-smm-control \
|
||||
-p EnvironmentFile=/etc/ochenstarik-server-monitor-manager/control.env \
|
||||
/usr/local/lib/ochenstarik-server-monitor-manager/control/ochenstarik-smm-control backup-create
|
||||
```
|
||||
|
||||
## Restore
|
||||
|
||||
Restore is intentionally offline. Stop Control, invoke the binary as root with the same environment file, and start Control again:
|
||||
|
||||
```bash
|
||||
BACKUP=/var/lib/ochenstarik-server-monitor-manager/backups/backup-YYYYMMDDTHHMMSSZ-id
|
||||
|
||||
sudo systemctl stop ochenstarik-smm-control.service
|
||||
sudo sh -c 'set -a; . /etc/ochenstarik-server-monitor-manager/control.env; set +a; exec /usr/local/lib/ochenstarik-server-monitor-manager/control/ochenstarik-smm-control backup-restore "$1"' sh "$BACKUP"
|
||||
sudo systemctl start ochenstarik-smm-control.service
|
||||
curl --fail --silent https://127.0.0.1:7443/healthz
|
||||
```
|
||||
|
||||
Restore validates both SHA-256 hashes and runs `PRAGMA integrity_check` before replacing live state. Existing files are copied to a root-only `pre-restore-*` directory before replacement. Keep that directory until server inventory, certificates, Links, and Agent heartbeats have been verified.
|
||||
|
|
@ -78,6 +78,10 @@
|
|||
- [x] ограниченный локальный буфер и downsampling;
|
||||
- [x] idempotency key и защита от replay;
|
||||
- [x] тест нагрузки 100 Node на одном Hub (конкурентные heartbeat, inventory и replay в CI).
|
||||
- [x] фоновое истечение TTL с подтверждением удаления firewall policy;
|
||||
- [x] retention метрик, replay и audit, версия SQLite schema, backup/restore Control DB и CA;
|
||||
- [x] HTTP integration tests, Linux Agent parser tests и Windows desktop contract tests;
|
||||
- [ ] выполнить acceptance test на физическом Hub и двух Node по `docs/three-server-acceptance.md`.
|
||||
|
||||
## Этап 7 — релиз и другие платформы
|
||||
|
||||
|
|
|
|||
33
docs/three-server-acceptance.md
Normal file
33
docs/three-server-acceptance.md
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# Three-server acceptance test
|
||||
|
||||
The acceptance script verifies one public Hub, one source Node used by an AI agent, and two independent destinations (for example a home server and a second remote server). It creates a temporary mTLS Operator identity, exercises two directional Links, disables only one Link, waits for server-side TTL expiration, and creates a verified Control backup.
|
||||
|
||||
Run it from a trusted Linux administration machine that can SSH to the Hub and source Node:
|
||||
|
||||
```bash
|
||||
export HUB_SSH_HOST=203.0.113.10
|
||||
export HUB_SSH_USER=admin
|
||||
export SOURCE_SSH_HOST=198.51.100.20
|
||||
export SOURCE_SSH_USER=admin
|
||||
export SSH_IDENTITY_FILE="$HOME/.ssh/id_ed25519"
|
||||
export SOURCE_NODE_ID=ai-agent
|
||||
export HOME_NODE_ID=home
|
||||
export SECOND_NODE_ID=second
|
||||
export HOME_WG_IP=10.77.0.3
|
||||
export SECOND_WG_IP=10.77.0.4
|
||||
export TARGET_PORT=22
|
||||
|
||||
bash tests/acceptance/three-server-mesh.sh
|
||||
```
|
||||
|
||||
Set `SMM_ACCEPT_RESTORE=1` to restore the backup created by the run, restart Control, and recheck both policies. Set `SMM_ACCEPT_REBOOT=1` to reboot the Hub and source Node and verify that the first Link remains usable while the disabled Link remains blocked. Restore and reboot are deliberately opt-in because they interrupt active sessions. Use both flags for the complete release acceptance run.
|
||||
|
||||
Prerequisites:
|
||||
|
||||
- all three Agent identities are enrolled and visible in Control Hub;
|
||||
- `curl`, `jq`, `openssl`, `ssh`, `nc`, and GNU `base64` are installed on the administration machine;
|
||||
- the SSH account can run the installer lifecycle command and `systemctl` through sudo;
|
||||
- `TARGET_PORT` listens on both destination Nodes;
|
||||
- `HOME_WG_IP` and `SECOND_WG_IP` are their `smm0` addresses.
|
||||
|
||||
Success ends with `THREE_SERVER_ACCEPTANCE=PASS`. The script keeps the home Link enabled for continued testing, leaves the second Link disabled, revokes its temporary Operator certificate, and never exports the Hub CA private key.
|
||||
|
|
@ -26,21 +26,36 @@ internal static class LinuxMetrics
|
|||
|
||||
private static double ReadLoadOne()
|
||||
{
|
||||
var value = File.ReadAllText("/proc/loadavg").Split(' ', 2)[0];
|
||||
return double.Parse(value, System.Globalization.CultureInfo.InvariantCulture);
|
||||
return ParseLoadOne(File.ReadAllText("/proc/loadavg"));
|
||||
}
|
||||
|
||||
private static long ReadUptimeSeconds()
|
||||
{
|
||||
var value = File.ReadAllText("/proc/uptime").Split(' ', 2)[0];
|
||||
return (long)double.Parse(value, System.Globalization.CultureInfo.InvariantCulture);
|
||||
return ParseUptimeSeconds(File.ReadAllText("/proc/uptime"));
|
||||
}
|
||||
|
||||
private static (long Total, long Available) ReadMemory()
|
||||
{
|
||||
return ParseMemory(File.ReadLines("/proc/meminfo"));
|
||||
}
|
||||
|
||||
internal static double ParseLoadOne(string contents)
|
||||
{
|
||||
var value = contents.Split(' ', 2, StringSplitOptions.RemoveEmptyEntries)[0];
|
||||
return double.Parse(value, System.Globalization.CultureInfo.InvariantCulture);
|
||||
}
|
||||
|
||||
internal static long ParseUptimeSeconds(string contents)
|
||||
{
|
||||
var value = contents.Split(' ', 2, StringSplitOptions.RemoveEmptyEntries)[0];
|
||||
return (long)double.Parse(value, System.Globalization.CultureInfo.InvariantCulture);
|
||||
}
|
||||
|
||||
internal static (long Total, long Available) ParseMemory(IEnumerable<string> lines)
|
||||
{
|
||||
long total = 0;
|
||||
long available = 0;
|
||||
foreach (var line in File.ReadLines("/proc/meminfo"))
|
||||
foreach (var line in lines)
|
||||
{
|
||||
var parts = line.Split(' ', StringSplitOptions.RemoveEmptyEntries);
|
||||
if (parts.Length < 2)
|
||||
|
|
@ -56,14 +71,23 @@ internal static class LinuxMetrics
|
|||
available = long.Parse(parts[1]) * 1024;
|
||||
}
|
||||
}
|
||||
if (total <= 0 || available < 0 || available > total)
|
||||
{
|
||||
throw new InvalidDataException("/proc/meminfo does not contain valid memory totals.");
|
||||
}
|
||||
return (total, available);
|
||||
}
|
||||
|
||||
private static (long Receive, long Transmit) ReadNetwork()
|
||||
{
|
||||
return ParseNetwork(File.ReadLines("/proc/net/dev"));
|
||||
}
|
||||
|
||||
internal static (long Receive, long Transmit) ParseNetwork(IEnumerable<string> lines)
|
||||
{
|
||||
long receive = 0;
|
||||
long transmit = 0;
|
||||
foreach (var line in File.ReadLines("/proc/net/dev").Skip(2))
|
||||
foreach (var line in lines.Skip(2))
|
||||
{
|
||||
var parts = line.Split([':', ' '], StringSplitOptions.RemoveEmptyEntries);
|
||||
if (parts.Length < 10 || parts[0] == "lo")
|
||||
|
|
|
|||
302
src/ServerMonitorManager.Control/ControlMaintenance.cs
Normal file
302
src/ServerMonitorManager.Control/ControlMaintenance.cs
Normal file
|
|
@ -0,0 +1,302 @@
|
|||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace ServerMonitorManager.Control;
|
||||
|
||||
public sealed record ControlMaintenanceResult(
|
||||
int MetricsDeleted,
|
||||
int IdempotencyDeleted,
|
||||
int AuditDeleted,
|
||||
int TokensDeleted);
|
||||
|
||||
public sealed class LinkExpirationBackgroundService(
|
||||
LinkService links,
|
||||
IOptions<ControlOptions> options,
|
||||
TimeProvider timeProvider,
|
||||
ILogger<LinkExpirationBackgroundService> logger) : BackgroundService
|
||||
{
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(
|
||||
TimeSpan.FromSeconds(options.Value.LinkExpirationPollSeconds), timeProvider);
|
||||
do
|
||||
{
|
||||
try
|
||||
{
|
||||
var result = await links.ExpireDueLinksAsync(timeProvider.GetUtcNow(), stoppingToken);
|
||||
if (result.Disabled > 0 || result.Failed > 0)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"TTL reconciliation completed: {Disabled} disabled, {Failed} failed.",
|
||||
result.Disabled,
|
||||
result.Failed);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
break;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogError(exception, "TTL reconciliation failed.");
|
||||
}
|
||||
}
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken));
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class ControlMaintenanceBackgroundService(
|
||||
ControlStore store,
|
||||
ControlBackupService backups,
|
||||
IOptions<ControlOptions> options,
|
||||
TimeProvider timeProvider,
|
||||
ILogger<ControlMaintenanceBackgroundService> logger) : BackgroundService
|
||||
{
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(
|
||||
TimeSpan.FromMinutes(options.Value.MaintenanceIntervalMinutes), timeProvider);
|
||||
do
|
||||
{
|
||||
try
|
||||
{
|
||||
var result = await store.MaintainAsync(timeProvider.GetUtcNow(), stoppingToken);
|
||||
await backups.CreateIfDueAsync(timeProvider.GetUtcNow(), stoppingToken);
|
||||
if (result.MetricsDeleted + result.IdempotencyDeleted + result.AuditDeleted + result.TokensDeleted > 0)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Control maintenance removed {Metrics} metrics, {Idempotency} replay records, "
|
||||
+ "{Audit} audit records, and {Tokens} enrollment tokens.",
|
||||
result.MetricsDeleted,
|
||||
result.IdempotencyDeleted,
|
||||
result.AuditDeleted,
|
||||
result.TokensDeleted);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
break;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogError(exception, "Control database maintenance failed.");
|
||||
}
|
||||
}
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken));
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class ControlBackupService(
|
||||
ControlStore store,
|
||||
IOptions<ControlOptions> options,
|
||||
ILogger<ControlBackupService> logger)
|
||||
{
|
||||
private const string ManifestFileName = "manifest.json";
|
||||
private readonly ControlOptions _options = options.Value;
|
||||
|
||||
public async Task<string> CreateAsync(DateTimeOffset now, CancellationToken cancellationToken = default)
|
||||
{
|
||||
Directory.CreateDirectory(_options.BackupDirectory);
|
||||
SetDirectoryPermissions(_options.BackupDirectory);
|
||||
var name = $"backup-{now:yyyyMMddTHHmmssfffZ}-{Guid.NewGuid():N}";
|
||||
var temporary = Path.Combine(_options.BackupDirectory, $".{name}.tmp");
|
||||
var destination = Path.Combine(_options.BackupDirectory, name);
|
||||
Directory.CreateDirectory(temporary);
|
||||
try
|
||||
{
|
||||
var databaseName = "control.db";
|
||||
var authorityName = "control-ca.pfx";
|
||||
var databasePath = Path.Combine(temporary, databaseName);
|
||||
var authorityPath = Path.Combine(temporary, authorityName);
|
||||
await store.BackupDatabaseAsync(databasePath, cancellationToken);
|
||||
File.Copy(_options.CertificateAuthorityPath, authorityPath, overwrite: false);
|
||||
SetFilePermissions(databasePath);
|
||||
SetFilePermissions(authorityPath);
|
||||
var manifest = new ControlBackupManifest(
|
||||
1,
|
||||
now,
|
||||
databaseName,
|
||||
await ComputeSha256Async(databasePath, cancellationToken),
|
||||
authorityName,
|
||||
await ComputeSha256Async(authorityPath, cancellationToken));
|
||||
await File.WriteAllTextAsync(
|
||||
Path.Combine(temporary, ManifestFileName),
|
||||
JsonSerializer.Serialize(manifest, ControlMaintenanceJsonContext.Default.ControlBackupManifest),
|
||||
cancellationToken);
|
||||
SetFilePermissions(Path.Combine(temporary, ManifestFileName));
|
||||
Directory.Move(temporary, destination);
|
||||
SetDirectoryPermissions(destination);
|
||||
TrimOldBackups();
|
||||
logger.LogInformation("Control backup created at {BackupPath}.", destination);
|
||||
return destination;
|
||||
}
|
||||
catch
|
||||
{
|
||||
if (Directory.Exists(temporary))
|
||||
{
|
||||
Directory.Delete(temporary, recursive: true);
|
||||
}
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
public async Task<string?> CreateIfDueAsync(
|
||||
DateTimeOffset now,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
Directory.CreateDirectory(_options.BackupDirectory);
|
||||
var newest = Directory.EnumerateDirectories(_options.BackupDirectory, "backup-*")
|
||||
.Select(Directory.GetLastWriteTimeUtc)
|
||||
.OrderByDescending(value => value)
|
||||
.FirstOrDefault();
|
||||
return newest == default || now - newest >= TimeSpan.FromHours(_options.BackupIntervalHours)
|
||||
? await CreateAsync(now, cancellationToken)
|
||||
: null;
|
||||
}
|
||||
|
||||
public async Task RestoreAsync(string backupPath, CancellationToken cancellationToken = default)
|
||||
{
|
||||
var root = Path.GetFullPath(backupPath);
|
||||
var manifestPath = Path.Combine(root, ManifestFileName);
|
||||
var manifest = JsonSerializer.Deserialize(
|
||||
await File.ReadAllTextAsync(manifestPath, cancellationToken),
|
||||
ControlMaintenanceJsonContext.Default.ControlBackupManifest)
|
||||
?? throw new InvalidDataException("Backup manifest is empty.");
|
||||
if (manifest.Version != 1
|
||||
|| Path.GetFileName(manifest.DatabaseFile) != manifest.DatabaseFile
|
||||
|| Path.GetFileName(manifest.CertificateAuthorityFile) != manifest.CertificateAuthorityFile)
|
||||
{
|
||||
throw new InvalidDataException("Unsupported or unsafe backup manifest.");
|
||||
}
|
||||
|
||||
var databaseSource = Path.Combine(root, manifest.DatabaseFile);
|
||||
var authoritySource = Path.Combine(root, manifest.CertificateAuthorityFile);
|
||||
await VerifyHashAsync(databaseSource, manifest.DatabaseSha256, cancellationToken);
|
||||
await VerifyHashAsync(authoritySource, manifest.CertificateAuthoritySha256, cancellationToken);
|
||||
await VerifyDatabaseAsync(databaseSource, cancellationToken);
|
||||
|
||||
var safetyDirectory = Path.Combine(
|
||||
_options.BackupDirectory,
|
||||
$"pre-restore-{DateTimeOffset.UtcNow:yyyyMMddTHHmmssfffZ}-{Guid.NewGuid():N}");
|
||||
Directory.CreateDirectory(safetyDirectory);
|
||||
if (File.Exists(_options.DatabasePath))
|
||||
{
|
||||
var safetyDatabase = Path.Combine(safetyDirectory, "control.db");
|
||||
File.Copy(_options.DatabasePath, safetyDatabase);
|
||||
SetFilePermissions(safetyDatabase);
|
||||
}
|
||||
if (File.Exists(_options.CertificateAuthorityPath))
|
||||
{
|
||||
var safetyAuthority = Path.Combine(safetyDirectory, "control-ca.pfx");
|
||||
File.Copy(_options.CertificateAuthorityPath, safetyAuthority);
|
||||
SetFilePermissions(safetyAuthority);
|
||||
}
|
||||
SetDirectoryPermissions(safetyDirectory);
|
||||
|
||||
await ReplaceFileAsync(databaseSource, _options.DatabasePath, cancellationToken);
|
||||
await ReplaceFileAsync(authoritySource, _options.CertificateAuthorityPath, cancellationToken);
|
||||
File.Delete(_options.DatabasePath + "-wal");
|
||||
File.Delete(_options.DatabasePath + "-shm");
|
||||
SetFilePermissions(_options.DatabasePath);
|
||||
SetFilePermissions(_options.CertificateAuthorityPath);
|
||||
logger.LogWarning(
|
||||
"Control state restored from {BackupPath}. Previous files are in {SafetyPath}.",
|
||||
root,
|
||||
safetyDirectory);
|
||||
}
|
||||
|
||||
private void TrimOldBackups()
|
||||
{
|
||||
foreach (var directory in Directory.EnumerateDirectories(_options.BackupDirectory, "backup-*")
|
||||
.OrderByDescending(Directory.GetLastWriteTimeUtc)
|
||||
.Skip(_options.BackupRetentionCount))
|
||||
{
|
||||
Directory.Delete(directory, recursive: true);
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task ReplaceFileAsync(
|
||||
string source,
|
||||
string destination,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(destination))!);
|
||||
var temporary = destination + $".restore-{Guid.NewGuid():N}.tmp";
|
||||
await using (var input = File.OpenRead(source))
|
||||
await using (var output = new FileStream(
|
||||
temporary, FileMode.CreateNew, FileAccess.Write, FileShare.None, 81920, useAsync: true))
|
||||
{
|
||||
await input.CopyToAsync(output, cancellationToken);
|
||||
await output.FlushAsync(cancellationToken);
|
||||
}
|
||||
File.Move(temporary, destination, overwrite: true);
|
||||
}
|
||||
|
||||
private static async Task VerifyDatabaseAsync(string path, CancellationToken cancellationToken)
|
||||
{
|
||||
await using var connection = new SqliteConnection(new SqliteConnectionStringBuilder
|
||||
{
|
||||
DataSource = path,
|
||||
Mode = SqliteOpenMode.ReadOnly,
|
||||
Pooling = false
|
||||
}.ToString());
|
||||
await connection.OpenAsync(cancellationToken);
|
||||
var command = connection.CreateCommand();
|
||||
command.CommandText = "PRAGMA integrity_check;";
|
||||
var result = Convert.ToString(await command.ExecuteScalarAsync(cancellationToken));
|
||||
if (!string.Equals(result, "ok", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
throw new InvalidDataException($"Backup database integrity check failed: {result}");
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task VerifyHashAsync(
|
||||
string path,
|
||||
string expected,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var actual = await ComputeSha256Async(path, cancellationToken);
|
||||
if (!CryptographicOperations.FixedTimeEquals(
|
||||
Convert.FromHexString(actual), Convert.FromHexString(expected)))
|
||||
{
|
||||
throw new InvalidDataException($"Backup checksum mismatch for {Path.GetFileName(path)}.");
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task<string> ComputeSha256Async(string path, CancellationToken cancellationToken)
|
||||
{
|
||||
await using var stream = File.OpenRead(path);
|
||||
var hash = await SHA256.HashDataAsync(stream, cancellationToken);
|
||||
return Convert.ToHexStringLower(hash);
|
||||
}
|
||||
|
||||
private static void SetDirectoryPermissions(string path)
|
||||
{
|
||||
if (OperatingSystem.IsLinux() || OperatingSystem.IsMacOS())
|
||||
{
|
||||
File.SetUnixFileMode(path, UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute);
|
||||
}
|
||||
}
|
||||
|
||||
private static void SetFilePermissions(string path)
|
||||
{
|
||||
if (OperatingSystem.IsLinux() || OperatingSystem.IsMacOS())
|
||||
{
|
||||
File.SetUnixFileMode(path, UnixFileMode.UserRead | UnixFileMode.UserWrite);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
internal sealed record ControlBackupManifest(
|
||||
int Version,
|
||||
DateTimeOffset CreatedAt,
|
||||
string DatabaseFile,
|
||||
string DatabaseSha256,
|
||||
string CertificateAuthorityFile,
|
||||
string CertificateAuthoritySha256);
|
||||
|
||||
[JsonSerializable(typeof(ControlBackupManifest))]
|
||||
internal sealed partial class ControlMaintenanceJsonContext : JsonSerializerContext;
|
||||
|
|
@ -14,6 +14,22 @@ public sealed class ControlOptions
|
|||
|
||||
public int MaxBufferedMetricAgeHours { get; init; } = 24;
|
||||
|
||||
public int MetricRetentionHours { get; init; } = 168;
|
||||
|
||||
public int IdempotencyRetentionHours { get; init; } = 24;
|
||||
|
||||
public int AuditRetentionDays { get; init; } = 90;
|
||||
|
||||
public int MaintenanceIntervalMinutes { get; init; } = 15;
|
||||
|
||||
public int LinkExpirationPollSeconds { get; init; } = 15;
|
||||
|
||||
public string BackupDirectory { get; init; } = "/var/lib/ochenstarik-server-monitor-manager/backups";
|
||||
|
||||
public int BackupIntervalHours { get; init; } = 24;
|
||||
|
||||
public int BackupRetentionCount { get; init; } = 7;
|
||||
|
||||
public string HubHelperPath { get; init; } = "/usr/local/libexec/ochenstarik-smm-policy-apply";
|
||||
|
||||
public string PrivilegeEscalationPath { get; init; } = "/usr/bin/sudo";
|
||||
|
|
|
|||
|
|
@ -10,6 +10,8 @@ namespace ServerMonitorManager.Control;
|
|||
|
||||
public sealed class ControlStore(IOptions<ControlOptions> options)
|
||||
{
|
||||
private const int CurrentSchemaVersion = 1;
|
||||
private readonly ControlOptions _options = options.Value;
|
||||
private readonly string _connectionString = new SqliteConnectionStringBuilder
|
||||
{
|
||||
DataSource = options.Value.DatabasePath,
|
||||
|
|
@ -22,6 +24,14 @@ public sealed class ControlStore(IOptions<ControlOptions> options)
|
|||
var path = new SqliteConnectionStringBuilder(_connectionString).DataSource;
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(path))!);
|
||||
await using var connection = await OpenAsync(cancellationToken);
|
||||
var versionCommand = connection.CreateCommand();
|
||||
versionCommand.CommandText = "PRAGMA user_version;";
|
||||
var schemaVersion = Convert.ToInt32(await versionCommand.ExecuteScalarAsync(cancellationToken));
|
||||
if (schemaVersion > CurrentSchemaVersion)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"Control database schema {schemaVersion} is newer than supported schema {CurrentSchemaVersion}.");
|
||||
}
|
||||
var command = connection.CreateCommand();
|
||||
command.CommandText = """
|
||||
PRAGMA journal_mode = WAL;
|
||||
|
|
@ -114,10 +124,74 @@ public sealed class ControlStore(IOptions<ControlOptions> options)
|
|||
CREATE UNIQUE INDEX IF NOT EXISTS ux_links_active_policy
|
||||
ON links(source_node_id, target_node_id, protocol, port)
|
||||
WHERE desired_state = 'Active';
|
||||
PRAGMA user_version = 1;
|
||||
""";
|
||||
await command.ExecuteNonQueryAsync(cancellationToken);
|
||||
}
|
||||
|
||||
public async Task<ControlMaintenanceResult> MaintainAsync(
|
||||
DateTimeOffset now,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
await using var connection = await OpenAsync(cancellationToken);
|
||||
var command = connection.CreateCommand();
|
||||
command.CommandText = """
|
||||
DELETE FROM metric_samples WHERE recorded_at < $metric_cutoff;
|
||||
SELECT changes();
|
||||
DELETE FROM idempotency WHERE created_at < $idempotency_cutoff;
|
||||
SELECT changes();
|
||||
DELETE FROM audit WHERE recorded_at < $audit_cutoff;
|
||||
SELECT changes();
|
||||
DELETE FROM enrollment_tokens WHERE consumed_at IS NOT NULL OR expires_at < $now;
|
||||
SELECT changes();
|
||||
DELETE FROM device_tokens WHERE consumed_at IS NOT NULL OR expires_at < $now;
|
||||
SELECT changes();
|
||||
DELETE FROM automation_tokens WHERE consumed_at IS NOT NULL OR expires_at < $now;
|
||||
SELECT changes();
|
||||
""";
|
||||
command.Parameters.AddWithValue(
|
||||
"$metric_cutoff", now.AddHours(-_options.MetricRetentionHours).ToString("O"));
|
||||
command.Parameters.AddWithValue(
|
||||
"$idempotency_cutoff", now.AddHours(-_options.IdempotencyRetentionHours).ToString("O"));
|
||||
command.Parameters.AddWithValue(
|
||||
"$audit_cutoff", now.AddDays(-_options.AuditRetentionDays).ToString("O"));
|
||||
command.Parameters.AddWithValue("$now", now.ToString("O"));
|
||||
var deleted = new int[6];
|
||||
await using (var reader = await command.ExecuteReaderAsync(cancellationToken))
|
||||
{
|
||||
for (var index = 0; index < deleted.Length; index++)
|
||||
{
|
||||
if (await reader.ReadAsync(cancellationToken))
|
||||
{
|
||||
deleted[index] = reader.GetInt32(0);
|
||||
}
|
||||
await reader.NextResultAsync(cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
var optimize = connection.CreateCommand();
|
||||
optimize.CommandText = "PRAGMA optimize; PRAGMA wal_checkpoint(PASSIVE);";
|
||||
await optimize.ExecuteNonQueryAsync(cancellationToken);
|
||||
return new ControlMaintenanceResult(
|
||||
deleted[0], deleted[1], deleted[2], deleted[3] + deleted[4] + deleted[5]);
|
||||
}
|
||||
|
||||
public async Task BackupDatabaseAsync(string destinationPath, CancellationToken cancellationToken = default)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(destinationPath))!);
|
||||
await using var source = await OpenAsync(cancellationToken);
|
||||
await using var destination = new SqliteConnection(new SqliteConnectionStringBuilder
|
||||
{
|
||||
DataSource = destinationPath,
|
||||
Mode = SqliteOpenMode.ReadWriteCreate,
|
||||
Pooling = false
|
||||
}.ToString());
|
||||
await destination.OpenAsync(cancellationToken);
|
||||
source.BackupDatabase(destination);
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
}
|
||||
|
||||
public async Task<string> CreateEnrollmentTokenAsync(
|
||||
string nodeId,
|
||||
TimeSpan lifetime,
|
||||
|
|
@ -1081,6 +1155,30 @@ public sealed class ControlStore(IOptions<ControlOptions> options)
|
|||
return result;
|
||||
}
|
||||
|
||||
public async Task<IReadOnlyList<LinkPolicy>> ListExpiredLinksAsync(
|
||||
DateTimeOffset now,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var result = new List<LinkPolicy>();
|
||||
await using var connection = await OpenAsync(cancellationToken);
|
||||
var command = connection.CreateCommand();
|
||||
command.CommandText = """
|
||||
SELECT * FROM links
|
||||
WHERE expires_at IS NOT NULL
|
||||
AND expires_at <= $now
|
||||
AND (desired_state = 'Active'
|
||||
OR (desired_state = 'Disabled' AND actual_state IN ('Disconnecting', 'Partial')))
|
||||
ORDER BY expires_at, version;
|
||||
""";
|
||||
command.Parameters.AddWithValue("$now", now.ToString("O"));
|
||||
await using var reader = await command.ExecuteReaderAsync(cancellationToken);
|
||||
while (await reader.ReadAsync(cancellationToken))
|
||||
{
|
||||
result.Add(ReadLink(reader));
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
public async Task<IReadOnlyList<LinkPolicy>> ListEffectiveLinksForNodeAsync(
|
||||
string nodeId,
|
||||
CancellationToken cancellationToken = default)
|
||||
|
|
|
|||
|
|
@ -121,6 +121,72 @@ public sealed class LinkService(
|
|||
return new LinkReconciliationResult(reconciled, failed);
|
||||
}
|
||||
|
||||
public async Task<LinkExpirationResult> ExpireDueLinksAsync(
|
||||
DateTimeOffset now,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var disabled = 0;
|
||||
var failed = 0;
|
||||
var links = await store.ListExpiredLinksAsync(now, cancellationToken);
|
||||
foreach (var candidate in links)
|
||||
{
|
||||
var gate = _reconciliationLocks.GetOrAdd(candidate.Id, static _ => new SemaphoreSlim(1, 1));
|
||||
await gate.WaitAsync(cancellationToken);
|
||||
try
|
||||
{
|
||||
var current = (await store.ListExpiredLinksAsync(now, cancellationToken))
|
||||
.SingleOrDefault(link => link.Id == candidate.Id);
|
||||
if (current is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (current.DesiredState == "Active")
|
||||
{
|
||||
var result = await DisableAsync(
|
||||
current.Id,
|
||||
new LinkPolicyDisableRequest(Guid.NewGuid().ToString()),
|
||||
"system:ttl",
|
||||
cancellationToken);
|
||||
if (result?.ActualState == "Disabled")
|
||||
{
|
||||
disabled++;
|
||||
}
|
||||
else
|
||||
{
|
||||
failed++;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
var retrying = await store.SetLinkActualStateAsync(
|
||||
current.Id, "Disconnecting", null, "system:ttl-retry", cancellationToken) ?? current;
|
||||
Publish("link.disconnecting", retrying);
|
||||
try
|
||||
{
|
||||
await applier.ApplyDisconnectAsync(retrying, cancellationToken);
|
||||
var completed = await store.SetLinkActualStateAsync(
|
||||
retrying.Id, "Disabled", null, "system:ttl-retry", cancellationToken) ?? retrying;
|
||||
Publish("link.disabled", completed);
|
||||
disabled++;
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException)
|
||||
{
|
||||
var partial = await store.SetLinkActualStateAsync(
|
||||
retrying.Id, "Partial", CompactError(exception), "system:ttl-retry", cancellationToken)
|
||||
?? retrying;
|
||||
Publish("link.partial", partial);
|
||||
failed++;
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
gate.Release();
|
||||
}
|
||||
}
|
||||
return new LinkExpirationResult(disabled, failed);
|
||||
}
|
||||
|
||||
private void Publish(string type, LinkPolicy link)
|
||||
=> events.Publish(
|
||||
type,
|
||||
|
|
@ -131,3 +197,5 @@ public sealed class LinkService(
|
|||
=> exception.Message.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries)
|
||||
.FirstOrDefault() ?? "Policy application failed.";
|
||||
}
|
||||
|
||||
public sealed record LinkExpirationResult(int Disabled, int Failed);
|
||||
|
|
|
|||
|
|
@ -38,16 +38,28 @@ builder.Services.AddOptions<ControlOptions>()
|
|||
&& !string.IsNullOrWhiteSpace(options.CertificateAuthorityPath)
|
||||
&& !string.IsNullOrWhiteSpace(options.HubHelperPath)
|
||||
&& !string.IsNullOrWhiteSpace(options.PrivilegeEscalationPath)
|
||||
&& !string.IsNullOrWhiteSpace(options.BackupDirectory)
|
||||
&& options.HeartbeatSeconds is >= 10 and <= 300
|
||||
&& options.MaxBufferedMetricAgeHours is >= 1 and <= 168,
|
||||
"Control, helper, and privilege escalation paths are required; HeartbeatSeconds must be 10-300, and buffered metrics 1-168 hours.")
|
||||
&& options.MaxBufferedMetricAgeHours is >= 1 and <= 168
|
||||
&& options.MetricRetentionHours is >= 24 and <= 8760
|
||||
&& options.IdempotencyRetentionHours is >= 1 and <= 720
|
||||
&& options.AuditRetentionDays is >= 1 and <= 3650
|
||||
&& options.MaintenanceIntervalMinutes is >= 1 and <= 1440
|
||||
&& options.LinkExpirationPollSeconds is >= 1 and <= 300
|
||||
&& options.BackupIntervalHours is >= 1 and <= 720
|
||||
&& options.BackupRetentionCount is >= 1 and <= 100,
|
||||
"Invalid Control paths, heartbeat, retention, maintenance, expiration, or backup settings.")
|
||||
.ValidateOnStart();
|
||||
builder.Services.AddSingleton(TimeProvider.System);
|
||||
builder.Services.AddSingleton<ControlStore>();
|
||||
builder.Services.AddSingleton<CertificateAuthority>();
|
||||
builder.Services.AddSingleton<ControlEventBroker>();
|
||||
builder.Services.AddSingleton<ILinkPolicyApplier, LinkPolicyApplier>();
|
||||
builder.Services.AddSingleton<LinkService>();
|
||||
builder.Services.AddSingleton<CertificateLifecycleService>();
|
||||
builder.Services.AddSingleton<ControlBackupService>();
|
||||
builder.Services.AddHostedService<LinkExpirationBackgroundService>();
|
||||
builder.Services.AddHostedService<ControlMaintenanceBackgroundService>();
|
||||
builder.Services.AddAuthentication(CertificateAuthenticationDefaults.AuthenticationScheme)
|
||||
.AddCertificate(options =>
|
||||
{
|
||||
|
|
@ -109,8 +121,22 @@ app.UseAuthentication();
|
|||
app.UseAuthorization();
|
||||
|
||||
var store = app.Services.GetRequiredService<ControlStore>();
|
||||
var backupService = app.Services.GetRequiredService<ControlBackupService>();
|
||||
if (args is ["backup-restore", var backupPath])
|
||||
{
|
||||
await backupService.RestoreAsync(backupPath);
|
||||
Console.WriteLine("Control backup restored. Start the service and verify /healthz.");
|
||||
return 0;
|
||||
}
|
||||
await store.InitializeAsync();
|
||||
|
||||
if (args is ["backup-create"])
|
||||
{
|
||||
var createdBackupPath = await backupService.CreateAsync(DateTimeOffset.UtcNow);
|
||||
Console.WriteLine(createdBackupPath);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (args is ["token-create", var nodeId])
|
||||
{
|
||||
if (!NodeIdValidator.IsValid(nodeId))
|
||||
|
|
|
|||
|
|
@ -5,6 +5,14 @@
|
|||
"CertificateAuthorityPassword": null,
|
||||
"HeartbeatSeconds": 30,
|
||||
"MaxBufferedMetricAgeHours": 24,
|
||||
"MetricRetentionHours": 168,
|
||||
"IdempotencyRetentionHours": 24,
|
||||
"AuditRetentionDays": 90,
|
||||
"MaintenanceIntervalMinutes": 15,
|
||||
"LinkExpirationPollSeconds": 15,
|
||||
"BackupDirectory": "/var/lib/ochenstarik-server-monitor-manager/backups",
|
||||
"BackupIntervalHours": 24,
|
||||
"BackupRetentionCount": 7,
|
||||
"HubHelperPath": "/usr/local/libexec/ochenstarik-smm-policy-apply",
|
||||
"PrivilegeEscalationPath": "/usr/bin/sudo"
|
||||
},
|
||||
|
|
|
|||
|
|
@ -73,6 +73,7 @@ public sealed partial class MainPage : Page
|
|||
internal async Task ChangeLinkFromPageAsync(
|
||||
MeshNodeViewModel? source,
|
||||
MeshNodeViewModel? target,
|
||||
MeshLinkViewModel? selectedLink,
|
||||
string protocol,
|
||||
int port,
|
||||
int ttlMinutes,
|
||||
|
|
@ -83,6 +84,7 @@ public sealed partial class MainPage : Page
|
|||
LinkProtocolBox.SelectedIndex = string.Equals(protocol, "udp", StringComparison.OrdinalIgnoreCase) ? 1 : 0;
|
||||
LinkPortBox.Value = port;
|
||||
LinkTtlBox.Value = ttlMinutes;
|
||||
MeshLinksList.SelectedItem = selectedLink;
|
||||
await ChangeLinkAsync(enable);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -36,6 +36,7 @@ public sealed partial class LinksPage : Page
|
|||
await _host.ChangeLinkFromPageAsync(
|
||||
SourceNodeBox.SelectedItem as MeshNodeViewModel,
|
||||
TargetNodeBox.SelectedItem as MeshNodeViewModel,
|
||||
LinksList.SelectedItem as MeshLinkViewModel,
|
||||
protocol,
|
||||
double.IsNaN(PortBox.Value) ? 0 : checked((int)PortBox.Value),
|
||||
double.IsNaN(TtlBox.Value) ? 0 : checked((int)TtlBox.Value),
|
||||
|
|
|
|||
158
tests/ServerMonitorManager.Control.Tests/ControlApiTests.cs
Normal file
158
tests/ServerMonitorManager.Control.Tests/ControlApiTests.cs
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
extern alias controlapp;
|
||||
|
||||
using System.Net;
|
||||
using System.Net.Http.Json;
|
||||
using System.Security.Cryptography;
|
||||
using System.Security.Claims;
|
||||
using System.Security.Cryptography.X509Certificates;
|
||||
using System.Text.Encodings.Web;
|
||||
using Microsoft.AspNetCore.Authentication;
|
||||
using Microsoft.AspNetCore.Hosting;
|
||||
using Microsoft.AspNetCore.Hosting.Server;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
using Microsoft.AspNetCore.TestHost;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Xunit;
|
||||
|
||||
namespace ServerMonitorManager.Control.Tests;
|
||||
|
||||
public sealed class ControlApiTests : IAsyncDisposable
|
||||
{
|
||||
private readonly ControlApiFactory _factory = new();
|
||||
|
||||
[Fact]
|
||||
public async Task HealthEndpointIsAnonymousAndControlEndpointsRequireOperator()
|
||||
{
|
||||
using var anonymous = _factory.CreateClient();
|
||||
Assert.Equal(HttpStatusCode.OK,
|
||||
(await anonymous.GetAsync("/healthz", TestContext.Current.CancellationToken)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.Unauthorized,
|
||||
(await anonymous.GetAsync(
|
||||
"/api/v1/control/agents", TestContext.Current.CancellationToken)).StatusCode);
|
||||
|
||||
using var authenticated = _factory.CreateClient();
|
||||
authenticated.DefaultRequestHeaders.Add("X-Test-Identity", "windows-pc");
|
||||
authenticated.DefaultRequestHeaders.Add("X-Test-Role", "Operator");
|
||||
Assert.Equal(HttpStatusCode.OK,
|
||||
(await authenticated.GetAsync(
|
||||
"/api/v1/control/agents", TestContext.Current.CancellationToken)).StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AutomationIdentityCannotUseOperatorSurface()
|
||||
{
|
||||
using var client = _factory.CreateClient();
|
||||
client.DefaultRequestHeaders.Add("X-Test-Identity", "coding-agent");
|
||||
client.DefaultRequestHeaders.Add("X-Test-Role", "Automation");
|
||||
|
||||
Assert.Equal(HttpStatusCode.Forbidden,
|
||||
(await client.GetAsync(
|
||||
"/api/v1/control/links", TestContext.Current.CancellationToken)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.OK,
|
||||
(await client.GetAsync(
|
||||
"/api/v1/automation/links", TestContext.Current.CancellationToken)).StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task InvalidEnrollmentUsesProblemDetailsResponse()
|
||||
{
|
||||
using var client = _factory.CreateClient();
|
||||
using var content = JsonContent.Create(new
|
||||
{
|
||||
nodeId = "INVALID NODE",
|
||||
token = "short",
|
||||
certificateSigningRequestPem = "bad",
|
||||
idempotencyKey = "bad"
|
||||
});
|
||||
using var response = await client.PostAsync(
|
||||
"/api/v1/enroll", content, TestContext.Current.CancellationToken);
|
||||
|
||||
Assert.Equal(HttpStatusCode.BadRequest, response.StatusCode);
|
||||
Assert.Equal("application/problem+json", response.Content.Headers.ContentType?.MediaType);
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync() => await _factory.DisposeAsync();
|
||||
|
||||
private sealed class ControlApiFactory : WebApplicationFactory<controlapp::Program>
|
||||
{
|
||||
private readonly string _directory = Path.Combine(
|
||||
Path.GetTempPath(), $"smm-api-tests-{Guid.NewGuid():N}");
|
||||
|
||||
protected override void ConfigureWebHost(IWebHostBuilder builder)
|
||||
{
|
||||
Directory.CreateDirectory(_directory);
|
||||
var authorityPath = Path.Combine(_directory, "control-ca.pfx");
|
||||
if (!File.Exists(authorityPath))
|
||||
{
|
||||
using var key = ECDsa.Create(ECCurve.NamedCurves.nistP256);
|
||||
var request = new CertificateRequest("CN=SMM API Test CA", key, HashAlgorithmName.SHA256);
|
||||
request.CertificateExtensions.Add(new X509BasicConstraintsExtension(true, false, 0, true));
|
||||
request.CertificateExtensions.Add(new X509KeyUsageExtension(
|
||||
X509KeyUsageFlags.KeyCertSign | X509KeyUsageFlags.CrlSign, true));
|
||||
using var certificate = request.CreateSelfSigned(
|
||||
DateTimeOffset.UtcNow.AddMinutes(-1), DateTimeOffset.UtcNow.AddDays(1));
|
||||
File.WriteAllBytes(authorityPath, certificate.Export(X509ContentType.Pfx));
|
||||
}
|
||||
builder.ConfigureAppConfiguration((_, configuration) =>
|
||||
configuration.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["Control:DatabasePath"] = Path.Combine(_directory, "control.db"),
|
||||
["Control:CertificateAuthorityPath"] = authorityPath,
|
||||
["Control:BackupDirectory"] = Path.Combine(_directory, "backups")
|
||||
}));
|
||||
builder.ConfigureTestServices(services =>
|
||||
{
|
||||
services.RemoveAll<IHostedService>();
|
||||
services.AddAuthentication(options =>
|
||||
{
|
||||
options.DefaultAuthenticateScheme = "Test";
|
||||
options.DefaultChallengeScheme = "Test";
|
||||
})
|
||||
.AddScheme<AuthenticationSchemeOptions, TestAuthenticationHandler>("Test", _ => { });
|
||||
});
|
||||
}
|
||||
|
||||
public override async ValueTask DisposeAsync()
|
||||
{
|
||||
await base.DisposeAsync();
|
||||
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
||||
if (Directory.Exists(_directory))
|
||||
{
|
||||
Directory.Delete(_directory, recursive: true);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class TestAuthenticationHandler(
|
||||
IOptionsMonitor<AuthenticationSchemeOptions> options,
|
||||
ILoggerFactory logger,
|
||||
UrlEncoder encoder)
|
||||
: AuthenticationHandler<AuthenticationSchemeOptions>(options, logger, encoder)
|
||||
{
|
||||
protected override Task<AuthenticateResult> HandleAuthenticateAsync()
|
||||
{
|
||||
if (!Request.Headers.TryGetValue("X-Test-Identity", out var identity)
|
||||
|| !Request.Headers.TryGetValue("X-Test-Role", out var role))
|
||||
{
|
||||
return Task.FromResult(AuthenticateResult.NoResult());
|
||||
}
|
||||
var claims = new List<Claim>
|
||||
{
|
||||
new(ClaimTypes.NameIdentifier, identity.ToString()),
|
||||
new(ClaimTypes.Role, role.ToString())
|
||||
};
|
||||
if (role.ToString() == "Automation")
|
||||
{
|
||||
claims.Add(new Claim("smm:source_node_id", "ai-agent"));
|
||||
}
|
||||
var principal = new ClaimsPrincipal(new ClaimsIdentity(claims, Scheme.Name));
|
||||
return Task.FromResult(AuthenticateResult.Success(
|
||||
new AuthenticationTicket(principal, Scheme.Name)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,192 @@
|
|||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ServerMonitorManager.Control;
|
||||
using ServerMonitorManager.Core;
|
||||
using Xunit;
|
||||
|
||||
namespace ServerMonitorManager.Control.Tests;
|
||||
|
||||
public sealed class ControlMaintenanceTests : IAsyncDisposable
|
||||
{
|
||||
private readonly string _directory = Path.Combine(
|
||||
Path.GetTempPath(), $"smm-maintenance-tests-{Guid.NewGuid():N}");
|
||||
|
||||
[Fact]
|
||||
public async Task ExpiredLinkIsDisabledAndFirewallStateIsReconciled()
|
||||
{
|
||||
var cancellationToken = TestContext.Current.CancellationToken;
|
||||
var (store, _) = CreateServices();
|
||||
await store.InitializeAsync(cancellationToken);
|
||||
await EnrollAgentAsync(store, "ai-agent", "AA11", cancellationToken);
|
||||
await EnrollAgentAsync(store, "home", "BB22", cancellationToken);
|
||||
var applier = new RecordingPolicyApplier();
|
||||
var service = new LinkService(store, applier, new ControlEventBroker());
|
||||
var link = await service.CreateAsync(
|
||||
new LinkPolicyCreateRequest(
|
||||
"ai-agent", "home", "tcp", 22, 1, "ttl-test", Guid.NewGuid().ToString()),
|
||||
"windows-pc",
|
||||
cancellationToken);
|
||||
|
||||
var result = await service.ExpireDueLinksAsync(
|
||||
link.CreatedAt.AddMinutes(2), cancellationToken);
|
||||
var persisted = Assert.Single(await store.ListLinksAsync(cancellationToken));
|
||||
|
||||
Assert.Equal(new LinkExpirationResult(1, 0), result);
|
||||
Assert.Equal("Disabled", persisted.DesiredState);
|
||||
Assert.Equal("Disabled", persisted.ActualState);
|
||||
Assert.Equal(1, applier.DisconnectCalls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ExpirationRetriesPartialFirewallRemoval()
|
||||
{
|
||||
var cancellationToken = TestContext.Current.CancellationToken;
|
||||
var (store, _) = CreateServices();
|
||||
await store.InitializeAsync(cancellationToken);
|
||||
await EnrollAgentAsync(store, "ai-agent", "CC33", cancellationToken);
|
||||
await EnrollAgentAsync(store, "home", "DD44", cancellationToken);
|
||||
var applier = new RecordingPolicyApplier { FailDisconnect = true };
|
||||
var service = new LinkService(store, applier, new ControlEventBroker());
|
||||
var link = await service.CreateAsync(
|
||||
new LinkPolicyCreateRequest(
|
||||
"ai-agent", "home", "tcp", 22, 1, "retry-test", Guid.NewGuid().ToString()),
|
||||
"windows-pc",
|
||||
cancellationToken);
|
||||
var expiredAt = link.CreatedAt.AddMinutes(2);
|
||||
|
||||
Assert.Equal(new LinkExpirationResult(0, 1),
|
||||
await service.ExpireDueLinksAsync(expiredAt, cancellationToken));
|
||||
applier.FailDisconnect = false;
|
||||
Assert.Equal(new LinkExpirationResult(1, 0),
|
||||
await service.ExpireDueLinksAsync(expiredAt.AddSeconds(1), cancellationToken));
|
||||
|
||||
var persisted = Assert.Single(await store.ListLinksAsync(cancellationToken));
|
||||
Assert.Equal("Disabled", persisted.ActualState);
|
||||
Assert.Equal(2, applier.DisconnectCalls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MaintenancePrunesExpiredOperationalDataAndVersionsSchema()
|
||||
{
|
||||
var cancellationToken = TestContext.Current.CancellationToken;
|
||||
var (store, options) = CreateServices();
|
||||
await store.InitializeAsync(cancellationToken);
|
||||
await EnrollAgentAsync(store, "home", "EE55", cancellationToken);
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
await store.RecordHeartbeatAsync(
|
||||
new AgentHeartbeat(
|
||||
"home", "test", now, 0.2, 1, 2, 1, 2, 3, 4, 5, Guid.NewGuid().ToString()),
|
||||
30,
|
||||
cancellationToken);
|
||||
|
||||
await using (var connection = new SqliteConnection($"Data Source={options.DatabasePath}"))
|
||||
{
|
||||
await connection.OpenAsync(cancellationToken);
|
||||
var age = connection.CreateCommand();
|
||||
age.CommandText = """
|
||||
UPDATE metric_samples SET recorded_at = $old;
|
||||
UPDATE idempotency SET created_at = $old;
|
||||
UPDATE audit SET recorded_at = $old;
|
||||
INSERT INTO device_tokens(token_hash, device_id, expires_at, consumed_at)
|
||||
VALUES ('expired', 'old-device', $old, NULL);
|
||||
""";
|
||||
age.Parameters.AddWithValue("$old", now.AddDays(-400).ToString("O"));
|
||||
await age.ExecuteNonQueryAsync(cancellationToken);
|
||||
}
|
||||
|
||||
var result = await store.MaintainAsync(now, cancellationToken);
|
||||
Assert.True(result.MetricsDeleted >= 1);
|
||||
Assert.True(result.IdempotencyDeleted >= 1);
|
||||
Assert.True(result.AuditDeleted >= 1);
|
||||
Assert.True(result.TokensDeleted >= 1);
|
||||
|
||||
await using var verify = new SqliteConnection($"Data Source={options.DatabasePath}");
|
||||
await verify.OpenAsync(cancellationToken);
|
||||
var version = verify.CreateCommand();
|
||||
version.CommandText = "PRAGMA user_version;";
|
||||
Assert.Equal(1L, (long)(await version.ExecuteScalarAsync(cancellationToken))!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task BackupCanRestoreDatabaseAndCertificateAuthority()
|
||||
{
|
||||
var cancellationToken = TestContext.Current.CancellationToken;
|
||||
var (store, options) = CreateServices();
|
||||
await File.WriteAllBytesAsync(options.CertificateAuthorityPath, [1, 2, 3, 4], cancellationToken);
|
||||
await store.InitializeAsync(cancellationToken);
|
||||
await EnrollAgentAsync(store, "home", "FF66", cancellationToken);
|
||||
var backups = new ControlBackupService(
|
||||
store, Options.Create(options), NullLogger<ControlBackupService>.Instance);
|
||||
var backupPath = await backups.CreateAsync(DateTimeOffset.UtcNow, cancellationToken);
|
||||
|
||||
SqliteConnection.ClearAllPools();
|
||||
File.Delete(options.DatabasePath);
|
||||
await File.WriteAllBytesAsync(options.CertificateAuthorityPath, [9, 9], cancellationToken);
|
||||
await backups.RestoreAsync(backupPath, cancellationToken);
|
||||
|
||||
var restoredStore = new ControlStore(Options.Create(options));
|
||||
await restoredStore.InitializeAsync(cancellationToken);
|
||||
Assert.Single(await restoredStore.ListAgentsAsync(cancellationToken));
|
||||
Assert.Equal([1, 2, 3, 4], await File.ReadAllBytesAsync(
|
||||
options.CertificateAuthorityPath, cancellationToken));
|
||||
}
|
||||
|
||||
public ValueTask DisposeAsync()
|
||||
{
|
||||
SqliteConnection.ClearAllPools();
|
||||
if (Directory.Exists(_directory))
|
||||
{
|
||||
Directory.Delete(_directory, recursive: true);
|
||||
}
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
|
||||
private (ControlStore Store, ControlOptions Options) CreateServices()
|
||||
{
|
||||
Directory.CreateDirectory(_directory);
|
||||
var options = new ControlOptions
|
||||
{
|
||||
DatabasePath = Path.Combine(_directory, "control.db"),
|
||||
CertificateAuthorityPath = Path.Combine(_directory, "control-ca.pfx"),
|
||||
BackupDirectory = Path.Combine(_directory, "backups"),
|
||||
MetricRetentionHours = 24,
|
||||
IdempotencyRetentionHours = 1,
|
||||
AuditRetentionDays = 1
|
||||
};
|
||||
return (new ControlStore(Options.Create(options)), options);
|
||||
}
|
||||
|
||||
private static async Task EnrollAgentAsync(
|
||||
ControlStore store,
|
||||
string nodeId,
|
||||
string thumbprint,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var token = await store.CreateEnrollmentTokenAsync(
|
||||
nodeId, TimeSpan.FromMinutes(10), cancellationToken);
|
||||
var result = await store.EnrollAsync(
|
||||
new EnrollmentRequest(nodeId, token, "csr", Guid.NewGuid().ToString()),
|
||||
() => new IssuedCertificate(
|
||||
"certificate", "ca", thumbprint, DateTimeOffset.UtcNow.AddYears(1)),
|
||||
cancellationToken);
|
||||
Assert.NotNull(result);
|
||||
}
|
||||
|
||||
private sealed class RecordingPolicyApplier : ILinkPolicyApplier
|
||||
{
|
||||
public bool FailDisconnect { get; set; }
|
||||
public int DisconnectCalls { get; private set; }
|
||||
|
||||
public Task ApplyConnectAsync(LinkPolicy link, CancellationToken cancellationToken)
|
||||
=> Task.CompletedTask;
|
||||
|
||||
public Task ApplyDisconnectAsync(LinkPolicy link, CancellationToken cancellationToken)
|
||||
{
|
||||
DisconnectCalls++;
|
||||
return FailDisconnect
|
||||
? Task.FromException(new InvalidOperationException("simulated firewall failure"))
|
||||
: Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
using ServerMonitorManager.Agent;
|
||||
using Xunit;
|
||||
|
||||
namespace ServerMonitorManager.Control.Tests;
|
||||
|
||||
public sealed class LinuxMetricsTests
|
||||
{
|
||||
[Fact]
|
||||
public void ProcParsersHandleRepresentativeLinuxData()
|
||||
{
|
||||
Assert.Equal(0.42, LinuxMetrics.ParseLoadOne("0.42 0.31 0.20 1/100 123\n"));
|
||||
Assert.Equal(1234L, LinuxMetrics.ParseUptimeSeconds("1234.98 567.00\n"));
|
||||
Assert.Equal(
|
||||
(2L * 1024 * 1024, 512L * 1024),
|
||||
LinuxMetrics.ParseMemory(
|
||||
[
|
||||
"MemTotal: 2048 kB",
|
||||
"MemFree: 100 kB",
|
||||
"MemAvailable: 512 kB"
|
||||
]));
|
||||
Assert.Equal(
|
||||
(400L, 600L),
|
||||
LinuxMetrics.ParseNetwork(
|
||||
[
|
||||
"Inter-| Receive | Transmit",
|
||||
" face |bytes ...|bytes ...",
|
||||
"lo: 10 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0",
|
||||
"eth0: 100 0 0 0 0 0 0 0 200 0 0 0 0 0 0 0",
|
||||
"wg0: 300 0 0 0 0 0 0 0 400 0 0 0 0 0 0 0"
|
||||
]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void InvalidMemoryTotalsAreRejected()
|
||||
{
|
||||
Assert.Throws<InvalidDataException>(() =>
|
||||
LinuxMetrics.ParseMemory(["MemAvailable: 10 kB"]));
|
||||
}
|
||||
}
|
||||
|
|
@ -7,7 +7,8 @@
|
|||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\src\ServerMonitorManager.Agent\ServerMonitorManager.Agent.csproj" />
|
||||
<ProjectReference Include="..\..\src\ServerMonitorManager.Control\ServerMonitorManager.Control.csproj" />
|
||||
<ProjectReference Include="..\..\src\ServerMonitorManager.Control\ServerMonitorManager.Control.csproj" Aliases="global,controlapp" />
|
||||
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.10" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />
|
||||
<PackageReference Include="xunit.v3" Version="3.2.2" />
|
||||
<PackageReference Include="xunit.runner.visualstudio" Version="3.1.5" />
|
||||
|
|
|
|||
225
tests/acceptance/three-server-mesh.sh
Normal file
225
tests/acceptance/three-server-mesh.sh
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
required=(curl jq openssl ssh nc base64)
|
||||
for command_name in "${required[@]}"; do
|
||||
command -v "$command_name" >/dev/null || {
|
||||
echo "Missing required command: $command_name" >&2
|
||||
exit 2
|
||||
}
|
||||
done
|
||||
|
||||
: "${HUB_SSH_HOST:?Set HUB_SSH_HOST}"
|
||||
: "${HUB_SSH_USER:?Set HUB_SSH_USER}"
|
||||
: "${SOURCE_SSH_HOST:?Set SOURCE_SSH_HOST}"
|
||||
: "${SOURCE_SSH_USER:?Set SOURCE_SSH_USER}"
|
||||
: "${HOME_WG_IP:?Set HOME_WG_IP}"
|
||||
: "${SECOND_WG_IP:?Set SECOND_WG_IP}"
|
||||
|
||||
HUB_SSH_PORT="${HUB_SSH_PORT:-22}"
|
||||
SOURCE_SSH_PORT="${SOURCE_SSH_PORT:-22}"
|
||||
SOURCE_NODE_ID="${SOURCE_NODE_ID:-ai-agent}"
|
||||
HOME_NODE_ID="${HOME_NODE_ID:-home}"
|
||||
SECOND_NODE_ID="${SECOND_NODE_ID:-second}"
|
||||
TARGET_PORT="${TARGET_PORT:-22}"
|
||||
CONTROL_DEVICE_ID="acceptance-$(date +%s)"
|
||||
INSTALLER_COMMAND="${INSTALLER_COMMAND:-sudo /usr/local/sbin/ochenstarik-server-monitor-manager.sh}"
|
||||
WORK_DIRECTORY="$(mktemp -d)"
|
||||
LINK_HOME_ID=''
|
||||
LINK_SECOND_ID=''
|
||||
|
||||
cleanup() {
|
||||
rm -rf "$WORK_DIRECTORY"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
ssh_options=(-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new)
|
||||
if [[ -n "${SSH_IDENTITY_FILE:-}" ]]; then
|
||||
ssh_options+=(-i "$SSH_IDENTITY_FILE")
|
||||
fi
|
||||
|
||||
hub_ssh() {
|
||||
ssh "${ssh_options[@]}" -p "$HUB_SSH_PORT" \
|
||||
"$HUB_SSH_USER@$HUB_SSH_HOST" "$@"
|
||||
}
|
||||
|
||||
source_ssh() {
|
||||
ssh "${ssh_options[@]}" -p "$SOURCE_SSH_PORT" \
|
||||
"$SOURCE_SSH_USER@$SOURCE_SSH_HOST" "$@"
|
||||
}
|
||||
|
||||
expect_reachable() {
|
||||
local ip="$1"
|
||||
source_ssh "nc -z -w 5 '$ip' '$TARGET_PORT'"
|
||||
}
|
||||
|
||||
expect_blocked() {
|
||||
local ip="$1"
|
||||
if source_ssh "nc -z -w 5 '$ip' '$TARGET_PORT'"; then
|
||||
echo "Unexpected access from $SOURCE_NODE_ID to $ip:$TARGET_PORT" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
decode_base64url() {
|
||||
local value="${1//-/+}"
|
||||
value="${value//_/\/}"
|
||||
case $((${#value} % 4)) in
|
||||
2) value+='==' ;;
|
||||
3) value+='=' ;;
|
||||
1) echo 'Invalid base64url value' >&2; return 1 ;;
|
||||
esac
|
||||
printf '%s' "$value" | base64 --decode
|
||||
}
|
||||
|
||||
echo '[1/11] Checking installed services on all three nodes'
|
||||
hub_ssh "sudo systemctl is-active ochenstarik-smm-control.service >/dev/null"
|
||||
source_ssh "sudo systemctl is-active ochenstarik-smm-agent.service >/dev/null"
|
||||
|
||||
echo '[2/11] Creating an isolated operator identity for this acceptance run'
|
||||
device_code="$(hub_ssh "$INSTALLER_COMMAND control-device-code '$CONTROL_DEVICE_ID'" | tr -d '\r' | grep -o 'SMMDEV1-[A-Za-z0-9_-]*' | tail -n1)"
|
||||
[[ -n "$device_code" ]] || { echo 'Hub did not return SMMDEV1 code' >&2; exit 1; }
|
||||
decode_base64url "${device_code#SMMDEV1-}" >"$WORK_DIRECTORY/device.env"
|
||||
control_url="$(sed -n 's/^URL=//p' "$WORK_DIRECTORY/device.env")"
|
||||
token="$(sed -n 's/^TOKEN=//p' "$WORK_DIRECTORY/device.env")"
|
||||
ca_base64="$(sed -n 's/^CA=//p' "$WORK_DIRECTORY/device.env")"
|
||||
[[ "$control_url" == https://* && -n "$token" && -n "$ca_base64" ]] || {
|
||||
echo 'Invalid SMMDEV1 payload' >&2
|
||||
exit 1
|
||||
}
|
||||
printf '%s' "$ca_base64" | base64 --decode >"$WORK_DIRECTORY/ca.der"
|
||||
openssl x509 -inform DER -in "$WORK_DIRECTORY/ca.der" -out "$WORK_DIRECTORY/ca.pem"
|
||||
openssl ecparam -name prime256v1 -genkey -noout -out "$WORK_DIRECTORY/operator.key"
|
||||
openssl req -new -sha256 -key "$WORK_DIRECTORY/operator.key" \
|
||||
-subj "/CN=$CONTROL_DEVICE_ID" -out "$WORK_DIRECTORY/operator.csr"
|
||||
jq -n \
|
||||
--arg deviceId "$CONTROL_DEVICE_ID" \
|
||||
--arg token "$token" \
|
||||
--arg csr "$(cat "$WORK_DIRECTORY/operator.csr")" \
|
||||
--arg idempotencyKey "$(cat /proc/sys/kernel/random/uuid)" \
|
||||
'{deviceId:$deviceId,token:$token,certificateSigningRequestPem:$csr,idempotencyKey:$idempotencyKey}' \
|
||||
>"$WORK_DIRECTORY/enroll.json"
|
||||
curl --fail --silent --show-error --cacert "$WORK_DIRECTORY/ca.pem" \
|
||||
-H 'Content-Type: application/json' --data-binary @"$WORK_DIRECTORY/enroll.json" \
|
||||
"$control_url/api/v1/device-enroll" >"$WORK_DIRECTORY/enrollment.json"
|
||||
jq -r '.certificatePem' "$WORK_DIRECTORY/enrollment.json" >"$WORK_DIRECTORY/operator.pem"
|
||||
|
||||
api_get() {
|
||||
curl --fail --silent --show-error --cacert "$WORK_DIRECTORY/ca.pem" \
|
||||
--cert "$WORK_DIRECTORY/operator.pem" --key "$WORK_DIRECTORY/operator.key" \
|
||||
"$control_url$1"
|
||||
}
|
||||
|
||||
api_post() {
|
||||
local path="$1"
|
||||
local body="$2"
|
||||
curl --fail --silent --show-error --cacert "$WORK_DIRECTORY/ca.pem" \
|
||||
--cert "$WORK_DIRECTORY/operator.pem" --key "$WORK_DIRECTORY/operator.key" \
|
||||
-H 'Content-Type: application/json' --data-binary "$body" "$control_url$path"
|
||||
}
|
||||
|
||||
create_link() {
|
||||
local target="$1"
|
||||
local ttl="$2"
|
||||
api_post '/api/v1/control/links' "$(jq -cn \
|
||||
--arg source "$SOURCE_NODE_ID" --arg target "$target" \
|
||||
--argjson port "$TARGET_PORT" --argjson ttl "$ttl" \
|
||||
--arg key "$(cat /proc/sys/kernel/random/uuid)" \
|
||||
'{sourceNodeId:$source,targetNodeId:$target,protocol:"tcp",port:$port,ttlMinutes:$ttl,reason:"three-server acceptance",idempotencyKey:$key}')"
|
||||
}
|
||||
|
||||
disable_link() {
|
||||
local id="$1"
|
||||
api_post "/api/v1/control/links/$id/disable" \
|
||||
"$(jq -cn --arg key "$(cat /proc/sys/kernel/random/uuid)" '{idempotencyKey:$key}')"
|
||||
}
|
||||
|
||||
echo '[3/11] Confirming all expected Agent identities are online'
|
||||
agents="$(api_get '/api/v1/control/agents')"
|
||||
for node in "$SOURCE_NODE_ID" "$HOME_NODE_ID" "$SECOND_NODE_ID"; do
|
||||
jq -e --arg node "$node" '.[] | select(.nodeId == $node)' <<<"$agents" >/dev/null || {
|
||||
echo "Control Hub does not contain Agent $node" >&2
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
|
||||
echo '[4/11] Creating independent Links to home and second server'
|
||||
home_link="$(create_link "$HOME_NODE_ID" 0)"
|
||||
second_link="$(create_link "$SECOND_NODE_ID" 0)"
|
||||
LINK_HOME_ID="$(jq -r '.id' <<<"$home_link")"
|
||||
LINK_SECOND_ID="$(jq -r '.id' <<<"$second_link")"
|
||||
jq -e '.actualState == "Active"' <<<"$home_link" >/dev/null
|
||||
jq -e '.actualState == "Active"' <<<"$second_link" >/dev/null
|
||||
|
||||
echo '[5/11] Verifying routed access through both Links'
|
||||
expect_reachable "$HOME_WG_IP"
|
||||
expect_reachable "$SECOND_WG_IP"
|
||||
|
||||
echo '[6/11] Disabling only the second Link'
|
||||
disable_link "$LINK_SECOND_ID" | jq -e '.actualState == "Disabled"' >/dev/null
|
||||
expect_reachable "$HOME_WG_IP"
|
||||
expect_blocked "$SECOND_WG_IP"
|
||||
|
||||
echo '[7/11] Verifying automatic TTL expiration'
|
||||
ttl_link="$(create_link "$SECOND_NODE_ID" 1)"
|
||||
ttl_id="$(jq -r '.id' <<<"$ttl_link")"
|
||||
expect_reachable "$SECOND_WG_IP"
|
||||
deadline=$((SECONDS + 120))
|
||||
while ((SECONDS < deadline)); do
|
||||
state="$(api_get '/api/v1/control/links' | jq -r --arg id "$ttl_id" '.[] | select(.id == $id) | .actualState')"
|
||||
[[ "$state" == 'Disabled' ]] && break
|
||||
sleep 5
|
||||
done
|
||||
[[ "${state:-}" == 'Disabled' ]] || { echo 'TTL Link did not become Disabled' >&2; exit 1; }
|
||||
expect_blocked "$SECOND_WG_IP"
|
||||
expect_reachable "$HOME_WG_IP"
|
||||
|
||||
echo '[8/11] Creating and validating a Control backup'
|
||||
backup_path="$(hub_ssh "sudo systemd-run --wait --pipe --quiet --collect --uid=ochenstarik-smm-control --gid=ochenstarik-smm-control -p EnvironmentFile=/etc/ochenstarik-server-monitor-manager/control.env /usr/local/lib/ochenstarik-server-monitor-manager/control/ochenstarik-smm-control backup-create" | grep '/backup-' | tail -n1)"
|
||||
[[ "$backup_path" == */backup-* ]] || { echo 'Backup command did not return a backup path' >&2; exit 1; }
|
||||
hub_ssh "sudo test -s '$backup_path/manifest.json' && sudo test -s '$backup_path/control.db' && sudo test -s '$backup_path/control-ca.pfx'"
|
||||
|
||||
if [[ "${SMM_ACCEPT_RESTORE:-0}" == '1' ]]; then
|
||||
echo '[9/11] Restoring the verified backup and restarting Control'
|
||||
hub_ssh "sudo sh -c 'set -a; . /etc/ochenstarik-server-monitor-manager/control.env; set +a; systemctl stop ochenstarik-smm-control.service; /usr/local/lib/ochenstarik-server-monitor-manager/control/ochenstarik-smm-control backup-restore \"\$1\"; status=\$?; systemctl start ochenstarik-smm-control.service; exit \$status' sh '$backup_path'"
|
||||
for _ in {1..30}; do
|
||||
if api_get '/healthz' >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
api_get '/healthz' >/dev/null
|
||||
expect_reachable "$HOME_WG_IP"
|
||||
expect_blocked "$SECOND_WG_IP"
|
||||
else
|
||||
echo '[9/11] Restore check skipped; set SMM_ACCEPT_RESTORE=1 to enable it'
|
||||
fi
|
||||
|
||||
if [[ "${SMM_ACCEPT_REBOOT:-0}" == '1' ]]; then
|
||||
echo '[10/11] Rebooting the Hub and source Node and rechecking policy state'
|
||||
hub_ssh 'sudo systemctl reboot' || true
|
||||
source_ssh 'sudo systemctl reboot' || true
|
||||
sleep 15
|
||||
for _ in {1..30}; do
|
||||
if hub_ssh 'true' >/dev/null 2>&1 && source_ssh 'true' >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
expect_reachable "$HOME_WG_IP"
|
||||
expect_blocked "$SECOND_WG_IP"
|
||||
else
|
||||
echo '[10/11] Reboot check skipped; set SMM_ACCEPT_REBOOT=1 to enable it'
|
||||
fi
|
||||
|
||||
echo '[11/11] Revoking the temporary Operator certificate'
|
||||
api_post "/api/v1/control/devices/$CONTROL_DEVICE_ID/reenroll" \
|
||||
"$(jq -cn --arg key "$(cat /proc/sys/kernel/random/uuid)" \
|
||||
'{reason:"three-server acceptance completed",idempotencyKey:$key}')" \
|
||||
| jq -e '.entityType == "Operator"' >/dev/null
|
||||
if api_get '/api/v1/control/agents' >/dev/null 2>&1; then
|
||||
echo 'Revoked acceptance Operator certificate is still authorized' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo 'THREE_SERVER_ACCEPTANCE=PASS'
|
||||
43
tests/windows/Test-DesktopContracts.ps1
Normal file
43
tests/windows/Test-DesktopContracts.ps1
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
[CmdletBinding()]
|
||||
param()
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$root = (Resolve-Path (Join-Path $PSScriptRoot '..\..')).Path
|
||||
$pages = @('ServersPage', 'LinksPage', 'SessionsPage', 'SettingsPage')
|
||||
foreach ($page in $pages) {
|
||||
foreach ($extension in @('.xaml', '.xaml.cs')) {
|
||||
$path = Join-Path $root "src\ServerMonitorManager.Desktop\Pages\$page$extension"
|
||||
if (-not (Test-Path -LiteralPath $path)) {
|
||||
throw "Missing desktop page contract: $path"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$linksXaml = Get-Content -Raw -Encoding UTF8 -LiteralPath (
|
||||
Join-Path $root 'src\ServerMonitorManager.Desktop\Pages\LinksPage.xaml')
|
||||
$linksCode = Get-Content -Raw -Encoding UTF8 -LiteralPath (
|
||||
Join-Path $root 'src\ServerMonitorManager.Desktop\Pages\LinksPage.xaml.cs')
|
||||
$mainCode = Get-Content -Raw -Encoding UTF8 -LiteralPath (
|
||||
Join-Path $root 'src\ServerMonitorManager.Desktop\MainPage.xaml.cs')
|
||||
|
||||
$requiredXamlContracts = @(
|
||||
'x:Name="LinksList"',
|
||||
'AutomationProperties.Name=',
|
||||
'Click="ConnectButton_Click"',
|
||||
'Click="DisconnectButton_Click"'
|
||||
)
|
||||
foreach ($contract in $requiredXamlContracts) {
|
||||
if ($linksXaml.IndexOf($contract, [StringComparison]::Ordinal) -lt 0) {
|
||||
throw "Links page is missing required UI contract: $contract"
|
||||
}
|
||||
}
|
||||
if ($linksCode.IndexOf(
|
||||
'LinksList.SelectedItem as MeshLinkViewModel', [StringComparison]::Ordinal) -lt 0) {
|
||||
throw 'Links page must pass its selected Link to the command handler.'
|
||||
}
|
||||
if ($mainCode.IndexOf(
|
||||
'MeshLinksList.SelectedItem = selectedLink;', [StringComparison]::Ordinal) -lt 0) {
|
||||
throw 'Main page must synchronize the selected Link before disconnecting it.'
|
||||
}
|
||||
|
||||
Write-Host 'Windows desktop contracts passed.'
|
||||
Loading…
Reference in a new issue