Add Apache 2.0 and durable Agent buffering #2

Merged
ochenstarik-ui merged 2 commits from agent/add-apache-license into main 2026-07-16 16:30:42 +00:00
28 changed files with 436 additions and 35 deletions
Showing only changes of commit 9efd8324eb - Show all commits

View file

@ -118,9 +118,9 @@ In the application, generate or copy the monitoring SSH key, add the Hub profile
`v0.1.0-alpha.2` is an early testing release, not a production security appliance. Windows and Linux builds, six control-plane tests, Bash syntax checks, self-contained `linux-x64`/`linux-arm64` artifacts, and checksums are automated in GitHub Actions.
Implemented: Windows SSH monitoring, Hub/Node WireGuard installer, directional Links, one-time enrollment, mTLS Agent and Operator identities, SQLite control state, audit, authenticated event stream, and Windows Control API integration.
The current development branch implements Windows SSH monitoring, the Hub/Node WireGuard installer, directional Links, one-time enrollment, mTLS Agent and Operator identities, SQLite control state, audit, authenticated event streaming, Windows Control API integration, and a bounded durable Agent buffer with downsampling.
Still planned: certificate revoke/re-enroll, offline Agent buffering and downsampling, reconnect reconciliation, kill-switch failure tests, a 50100 Node load test, signed Windows installer, and desktop/mobile clients for additional platforms.
Still planned: certificate revoke/re-enroll, reconnect reconciliation, kill-switch failure tests, a 50100 Node load test, signed Windows installer, and desktop/mobile clients for additional platforms.
## License and project policy

View file

@ -50,7 +50,7 @@ Node устанавливает исходящее WireGuard-соединени
Текущий SSH snapshot содержит CPU/load, память, корневой диск, uptime и задержку. Следующая версия добавляет swap, inode, сетевые счётчики и выбранные systemd units.
Постоянный Linux agent появится после стабилизации трёхсерверного сценария. Он будет отправлять метрики исходящим mTLS-соединением, вести ограниченный локальный буфер и не открывать публичный API.
Постоянный Linux Agent отправляет метрики исходящим mTLS-соединением и не открывает публичный API. При потере Hub он сохраняет атомарный root-only буфер: последние точки остаются с полной детализацией, старые группы уплотняются с сохранением наиболее важного пика. Размер очереди ограничен, а подтверждённые Hub точки удаляются строго по порядку.
## 4. Links между серверами
@ -100,7 +100,7 @@ Hub остаётся маршрутизатором Mesh первого поко
Control Hub сохраняет inventory, heartbeat-метрики, направленные Links, idempotency и аудит в SQLite. Желаемое состояние отключения Link фиксируется до вызова ограниченного nftables wrapper; при ошибке фактическое состояние становится `Partial`. Отдельная mTLS identity `Operator` читает inventory, управляет Links и получает NDJSON event stream, а сертификат `Agent` ограничен heartbeat собственного `node_id`.
Локальный буфер Agent, downsampling и подключение event stream к WinUI выполняются следующими частями этапа.
Agent использует ограниченный долговечный буфер и повторяет доставку с тем же idempotency key. Hub принимает накопленные точки только в настроенном временном окне и сохраняет исходное время измерения. Event stream уже подключён к WinUI.
## 7. Целевые ограничения MVP

View file

@ -32,7 +32,7 @@ sudo ./ochenstarik-server-monitor-manager.sh hub
لا توجد كلمة مرور root مشتركة ولا يغادر مفتاح WireGuard الخاص عقدته. هويات monitoring وterminal وAgent وOperator وأتمتة AI منفصلة. يستخدم SSH أمراً إجبارياً بلا shell أو PTY أو forwarding؛ يقيّد mTLS الصلاحيات؛ تسمح nftables بالروابط الصريحة فقط؛ ويحفظ SQLite الحالة المطلوبة والتدقيق قبل تعديل الجدار الناري.
الإصدار `v0.1.0-alpha.2` للاختبار. يتضمن عميل Windows ومثبت Hub/Node وLinks وmTLS وSQLite والتدقيق والأحداث. المتبقي: التخزين دون اتصال، إلغاء الشهادات، مصالحة إعادة الاتصال، اختبار 50100 Node ومثبت Windows موقّع.
الإصدار `v0.1.0-alpha.2` للاختبار. يتضمن فرع التطوير الحالي عميل Windows ومثبت Hub/Node وLinks وmTLS وSQLite والتدقيق والأحداث ومخزناً محدوداً دون اتصال مع downsampling. المتبقي: إلغاء الشهادات، مصالحة إعادة الاتصال، اختبار 50100 Node ومثبت Windows موقّع.
## الترخيص

View file

@ -32,7 +32,7 @@ sudo ./ochenstarik-server-monitor-manager.sh hub
Es gibt kein gemeinsames Root-Passwort; der private WireGuard-Schlüssel bleibt auf dem Node. Monitoring-, Terminal-, Agent-, Operator- und KI-Automationsidentitäten sind getrennt. SSH nutzt einen forced-command ohne Shell/PTY/Forwarding; mTLS begrenzt Rollen; nftables erlaubt nur explizite Links; SQLite speichert Sollzustand und Audit vor der Firewalländerung.
`v0.1.0-alpha.2` ist eine Testversion. Windows-Client, Hub/Node-Installer, Links, mTLS, SQLite, Audit und Events sind vorhanden. Offen sind Offline-Puffer, Zertifikatswiderruf, Reconnect-Abgleich, Lasttest mit 50100 Nodes und signierter Windows-Installer.
`v0.1.0-alpha.2` ist eine Testversion. Der aktuelle Entwicklungszweig enthält bereits Windows-Client, Hub/Node-Installer, Links, mTLS, SQLite, Audit, Events und einen begrenzten Offline-Puffer mit Downsampling. Offen sind Zertifikatswiderruf, Reconnect-Abgleich, Lasttest mit 50100 Nodes und signierter Windows-Installer.
## Lizenz

View file

@ -34,7 +34,7 @@ Abra UDP `51820` y TCP `7443` en el Hub, cree códigos para los Nodes e instále
No hay contraseña root compartida ni claves WireGuard privadas de Nodes en el Hub. Las identidades de monitorización, terminal, Agent, Operator y automatización IA están separadas. SSH usa un forced-command sin shell, PTY ni forwarding; mTLS limita cada rol; nftables permite únicamente Links explícitos; SQLite conserva estado y auditoría antes de aplicar cambios.
`v0.1.0-alpha.2` es una versión de prueba. Ya incluye cliente Windows, instalador Hub/Node, Links, mTLS, SQLite, auditoría y eventos. Faltan el búfer offline, revocación de certificados, reconciliación tras reconexión, prueba de 50100 Nodes e instalador Windows firmado.
`v0.1.0-alpha.2` es una versión de prueba. La rama de desarrollo actual ya incluye cliente Windows, instalador Hub/Node, Links, mTLS, SQLite, auditoría, eventos y un búfer offline limitado con downsampling. Faltan la revocación de certificados, reconciliación tras reconexión, prueba de 50100 Nodes e instalador Windows firmado.
## Licencia

View file

@ -32,7 +32,7 @@ Ouvrez UDP `51820` et TCP `7443` sur le Hub, créez les codes et installez les a
Aucun mot de passe root n'est partagé et la clé WireGuard privée reste sur le Node. Les identités monitoring, terminal, Agent, Operator et automatisation IA sont séparées. SSH emploie une forced-command sans shell/PTY/forwarding ; mTLS limite les rôles ; nftables n'autorise que les Links explicites ; SQLite enregistre état et audit avant le changement de pare-feu.
`v0.1.0-alpha.2` est destiné aux tests. Client Windows, installateur Hub/Node, Links, mTLS, SQLite, audit et événements sont disponibles. Restent le tampon hors ligne, la révocation des certificats, la réconciliation, le test de 50100 Nodes et l'installateur Windows signé.
`v0.1.0-alpha.2` est destiné aux tests. La branche de développement actuelle contient déjà le client Windows, l'installateur Hub/Node, les Links, mTLS, SQLite, l'audit, les événements et un tampon hors ligne limité avec downsampling. Restent la révocation des certificats, la réconciliation, le test de 50100 Nodes et l'installateur Windows signé.
## Licence

View file

@ -32,7 +32,7 @@ Hub पर UDP `51820` और TCP `7443` खोलें, Node codes बना
Shared root password नहीं है और Node की WireGuard private key Node से बाहर नहीं जाती। Monitoring, terminal, Agent, Operator और AI automation identities अलग हैं। SSH forced-command shell/PTY/forwarding नहीं देता; mTLS roles सीमित करता है; nftables केवल explicit Links स्वीकारता है; SQLite पहले desired state और audit सहेजता है।
`v0.1.0-alpha.2` testing release है। Windows client, Hub/Node installer, Links, mTLS, SQLite, audit और event stream तैयार हैं। Offline buffer, certificate revocation, reconnect reconciliation, 50100 Node load test और signed Windows installer अभी बाकी हैं।
`v0.1.0-alpha.2` testing release है। Current development branch में Windows client, Hub/Node installer, Links, mTLS, SQLite, audit, event stream और downsampling वाला सीमित offline buffer तैयार हैं। Certificate revocation, reconnect reconciliation, 50100 Node load test और signed Windows installer अभी बाकी हैं।
## लाइसेंस

View file

@ -32,7 +32,7 @@ Hub で UDP `51820` と TCP `7443` を開き、Node コードを作成して他
共有 root パスワードはなく、Node の WireGuard 秘密鍵は Node 外に出ません。monitoring、terminal、Agent、Operator、AI automation の ID は分離されています。SSH は shell/PTY/forwarding のない forced-command、mTLS はロール制限、nftables は明示 Link のみを許可し、SQLite は firewall 変更前に状態と監査を保存します。
`v0.1.0-alpha.2` はテスト版です。Windows client、Hub/Node installer、Links、mTLS、SQLite、監査、イベントは実装済みです。オフラインバッファ、証明書失効、再接続調整、50100 Node 負荷試験、署名付き Windows installer は今後の課題です。
`v0.1.0-alpha.2` はテスト版です。現在の開発ブランチには Windows client、Hub/Node installer、Links、mTLS、SQLite、監査、イベント、downsampling 付きの制限オフラインバッファが実装済みです。証明書失効、再接続調整、50100 Node 負荷試験、署名付き Windows installer は今後の課題です。
## ライセンス

View file

@ -32,7 +32,7 @@ Hub에서 UDP `51820`과 TCP `7443`을 열고 Node 코드를 생성한 뒤 다
공유 root 암호가 없고 Node WireGuard 개인 키는 Node를 떠나지 않습니다. monitoring, terminal, Agent, Operator, AI automation identity는 분리됩니다. SSH는 shell/PTY/forwarding 없는 forced-command를 사용하고, mTLS는 역할을 제한하며, nftables는 명시된 Link만 허용합니다. SQLite는 방화벽 변경 전에 상태와 감사를 저장합니다.
`v0.1.0-alpha.2`는 테스트 릴리스입니다. Windows client, Hub/Node installer, Links, mTLS, SQLite, 감사와 event stream이 구현되었습니다. offline buffer, 인증서 폐기, 재연결 조정, 50100 Node 부하 시험과 서명된 Windows installer가 남아 있습니다.
`v0.1.0-alpha.2`는 테스트 릴리스입니다. 현재 개발 branch에는 Windows client, Hub/Node installer, Links, mTLS, SQLite, 감사, event stream과 downsampling이 적용된 제한 offline buffer가 구현되었습니다. 인증서 폐기, 재연결 조정, 50100 Node 부하 시험과 서명된 Windows installer가 남아 있습니다.
## 라이선스

View file

@ -32,7 +32,7 @@ Abra UDP `51820` e TCP `7443` no Hub, gere códigos e instale os demais servidor
Não há senha root compartilhada e a chave WireGuard privada nunca sai do Node. As identidades de monitoramento, terminal, Agent, Operator e automação de IA são separadas. SSH usa forced-command sem shell/PTY/forwarding; mTLS restringe funções; nftables permite apenas Links explícitos; SQLite registra estado e auditoria antes da mudança no firewall.
`v0.1.0-alpha.2` é uma versão de teste. Cliente Windows, instalador Hub/Node, Links, mTLS, SQLite, auditoria e eventos estão prontos. Restam buffer offline, revogação de certificado, reconciliação, teste com 50100 Nodes e instalador Windows assinado.
`v0.1.0-alpha.2` é uma versão de teste. A branch de desenvolvimento atual já contém cliente Windows, instalador Hub/Node, Links, mTLS, SQLite, auditoria, eventos e buffer offline limitado com downsampling. Restam revogação de certificado, reconciliação, teste com 50100 Nodes e instalador Windows assinado.
## Licença

View file

@ -38,7 +38,7 @@ sudo ./ochenstarik-server-monitor-manager.sh hub
Нет общих root-паролей; приватный WireGuard-ключ Node не покидает Node. Идентичности monitoring, terminal, Agent, Operator и AI-автоматизации разделены. Monitoring SSH использует forced-command без shell/PTY/forwarding. Agent может отправлять heartbeat только своего узла, Operator управляет inventory и Links. Отключение Link сначала сохраняется в SQLite, затем удаляет разрешение nftables; повтор запроса не повторяет побочный эффект.
`v0.1.0-alpha.2` предназначен для тестирования. Уже готовы Windows SSH-monitoring, Hub/Node installer, Links, mTLS, SQLite, аудит и поток событий. Остались offline-буфер Agent, отзыв сертификатов, reconnect reconciliation, нагрузочный тест 50100 Node и подписанный Windows installer. До стабильного релиза используйте тестовые или резервируемые серверы.
`v0.1.0-alpha.2` предназначен для тестирования. В текущей ветке разработки уже готовы Windows SSH-monitoring, Hub/Node installer, Links, mTLS, SQLite, аудит, поток событий и ограниченный offline-буфер Agent с downsampling. Остались отзыв сертификатов, reconnect reconciliation, нагрузочный тест 50100 Node и подписанный Windows installer. До стабильного релиза используйте тестовые или резервируемые серверы.
## Лицензия

View file

@ -32,7 +32,7 @@ Hub üzerinde UDP `51820` ve TCP `7443` açın, Node kodlarını üretin ve diğ
Ortak root parolası yoktur ve Node'un WireGuard özel anahtarı Node'dan çıkmaz. Monitoring, terminal, Agent, Operator ve AI automation kimlikleri ayrıdır. SSH shell/PTY/forwarding vermeyen forced-command kullanır; mTLS rolleri sınırlar; nftables yalnızca açık Links'e izin verir; SQLite güvenlik duvarı değişmeden önce durum ve audit kaydeder.
`v0.1.0-alpha.2` test sürümüdür. Windows client, Hub/Node installer, Links, mTLS, SQLite, audit ve event stream hazırdır. Offline buffer, sertifika iptali, yeniden bağlantı uzlaştırması, 50100 Node yük testi ve imzalı Windows installer sıradadır.
`v0.1.0-alpha.2` test sürümüdür. Güncel geliştirme dalında Windows client, Hub/Node installer, Links, mTLS, SQLite, audit, event stream ve downsampling kullanan sınırlı offline buffer hazırdır. Sertifika iptali, yeniden bağlantı uzlaştırması, 50100 Node yük testi ve imzalı Windows installer sıradadır.
## Lisans

View file

@ -34,7 +34,7 @@ sudo ./ochenstarik-server-monitor-manager.sh hub
系统不共享 root 密码Node 的 WireGuard 私钥不会离开本机。监控、终端、Agent、Operator 和 AI 自动化身份相互隔离。SSH 使用无 shell、PTY、转发权限的 forced-commandmTLS 限制角色nftables 仅允许明确 LinkSQLite 在执行防火墙变更前保存目标状态和审计。
`v0.1.0-alpha.2` 是测试版,已包含 Windows 客户端、Hub/Node 安装器、Links、mTLS、SQLite、审计和事件流。待完成离线缓冲、证书撤销、重连协调、50100 Node 压测和签名 Windows 安装器。
`v0.1.0-alpha.2` 是测试版。当前开发分支已包含 Windows 客户端、Hub/Node 安装器、Links、mTLS、SQLite、审计、事件流以及带降采样的有限离线缓冲。待完成证书撤销、重连协调、50100 Node 压测和签名 Windows 安装器。
## 许可证

View file

@ -74,7 +74,7 @@
- [x] SQLite inventory, policies, history и audit;
- [x] исходящие mTLS agent sessions;
- [x] защищённый Hub event stream для desktop client;
- [ ] ограниченный локальный буфер и downsampling;
- [x] ограниченный локальный буфер и downsampling;
- [x] idempotency key и защита от replay;
- [ ] тест нагрузки 50100 Node на одном Hub.

View file

@ -1,3 +1,4 @@
using System.Net;
using System.Net.Http.Json;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
@ -48,25 +49,94 @@ internal sealed class AgentClient(AgentOptions options)
password: null,
X509KeyStorageFlags.EphemeralKeySet);
using var client = CreateHttpClient(certificate);
var delay = TimeSpan.FromSeconds(options.HeartbeatSeconds);
var buffer = new MetricBuffer(options);
var collectionDelay = TimeSpan.FromSeconds(options.HeartbeatSeconds);
var retryDelay = collectionDelay;
var nextUploadAttempt = DateTimeOffset.MinValue;
while (!cancellationToken.IsCancellationRequested)
{
var heartbeat = LinuxMetrics.Collect(options.NodeId, "0.1.0");
using var response = await client.PostAsJsonAsync(
"api/v1/agents/heartbeat",
heartbeat,
SmmJsonContext.Default.AgentHeartbeat,
cancellationToken);
response.EnsureSuccessStatusCode();
var accepted = await response.Content.ReadFromJsonAsync(
SmmJsonContext.Default.AgentHeartbeatResponse,
cancellationToken)
?? throw new InvalidOperationException("Control service returned an empty heartbeat response.");
delay = TimeSpan.FromSeconds(Math.Clamp(accepted.NextHeartbeatSeconds, 10, 300));
await Task.Delay(delay, cancellationToken);
var iterationStartedAt = DateTimeOffset.UtcNow;
try
{
await buffer.EnqueueAsync(
LinuxMetrics.Collect(options.NodeId, "0.1.0"),
cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
Console.Error.WriteLine($"Metric collection failed: {exception.Message}");
}
if (DateTimeOffset.UtcNow >= nextUploadAttempt)
{
try
{
var pending = await buffer.PeekAsync(options.UploadBatchSize, cancellationToken);
foreach (var heartbeat in pending)
{
try
{
var accepted = await SendHeartbeatAsync(client, heartbeat, cancellationToken);
collectionDelay = TimeSpan.FromSeconds(
Math.Clamp(accepted.NextHeartbeatSeconds, 10, 300));
}
catch (PermanentHeartbeatException exception)
{
Console.Error.WriteLine(
$"Dropping rejected metric {heartbeat.SentAt:O}: {exception.Message}");
}
await buffer.AcknowledgeAsync(heartbeat.IdempotencyKey, cancellationToken);
}
retryDelay = collectionDelay;
nextUploadAttempt = DateTimeOffset.MinValue;
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
var jitter = TimeSpan.FromSeconds(Random.Shared.Next(0, 6));
nextUploadAttempt = DateTimeOffset.UtcNow + retryDelay + jitter;
retryDelay = TimeSpan.FromSeconds(Math.Min(
options.MaxRetrySeconds,
Math.Max(10, retryDelay.TotalSeconds * 2)));
Console.Error.WriteLine(
$"Control Hub unavailable; metrics remain buffered until {nextUploadAttempt:O}: "
+ exception.Message);
}
}
var elapsed = DateTimeOffset.UtcNow - iterationStartedAt;
var remaining = collectionDelay - elapsed;
await Task.Delay(remaining > TimeSpan.Zero ? remaining : TimeSpan.FromSeconds(1), cancellationToken);
}
}
private static async Task<AgentHeartbeatResponse> SendHeartbeatAsync(
HttpClient client,
AgentHeartbeat heartbeat,
CancellationToken cancellationToken)
{
using var response = await client.PostAsJsonAsync(
"api/v1/agents/heartbeat",
heartbeat,
SmmJsonContext.Default.AgentHeartbeat,
cancellationToken);
if (response.StatusCode is HttpStatusCode.BadRequest or HttpStatusCode.Conflict)
{
var details = await response.Content.ReadAsStringAsync(cancellationToken);
throw new PermanentHeartbeatException(
string.IsNullOrWhiteSpace(details)
? $"Control service returned {(int)response.StatusCode}."
: details);
}
response.EnsureSuccessStatusCode();
return await response.Content.ReadFromJsonAsync(
SmmJsonContext.Default.AgentHeartbeatResponse,
cancellationToken)
?? throw new InvalidOperationException("Control service returned an empty heartbeat response.");
}
private HttpClient CreateHttpClient(X509Certificate2? clientCertificate)
{
using var root = X509CertificateLoader.LoadCertificateFromFile(options.CertificateAuthorityPath);
@ -103,3 +173,5 @@ internal sealed class AgentClient(AgentOptions options)
}
}
}
internal sealed class PermanentHeartbeatException(string message) : Exception(message);

View file

@ -7,4 +7,9 @@ public sealed class AgentOptions
public string StateDirectory { get; init; } = "/var/lib/ochenstarik-server-monitor-manager/agent";
public string CertificateAuthorityPath { get; init; } = "/etc/ochenstarik-server-monitor-manager/control-ca.crt";
public int HeartbeatSeconds { get; init; } = 30;
public int BufferMaxSamples { get; init; } = 720;
public int BufferRecentSamples { get; init; } = 120;
public int BufferDownsampleFactor { get; init; } = 4;
public int UploadBatchSize { get; init; } = 20;
public int MaxRetrySeconds { get; init; } = 300;
}

View file

@ -0,0 +1,186 @@
using System.Text.Json;
using ServerMonitorManager.Core;
namespace ServerMonitorManager.Agent;
internal sealed class MetricBuffer
{
private readonly AgentOptions _options;
private readonly string _path;
private readonly SemaphoreSlim _gate = new(1, 1);
private List<AgentHeartbeat>? _samples;
public MetricBuffer(AgentOptions options)
{
_options = options;
_path = Path.Combine(options.StateDirectory, "metric-buffer.json");
}
public async Task EnqueueAsync(AgentHeartbeat heartbeat, CancellationToken cancellationToken)
{
await _gate.WaitAsync(cancellationToken);
try
{
var samples = await LoadAsync(cancellationToken);
if (samples.Any(sample => sample.IdempotencyKey == heartbeat.IdempotencyKey))
{
return;
}
samples.Add(heartbeat);
Compact(samples);
await PersistAsync(samples, cancellationToken);
}
finally
{
_gate.Release();
}
}
public async Task<IReadOnlyList<AgentHeartbeat>> PeekAsync(
int maximumCount,
CancellationToken cancellationToken)
{
await _gate.WaitAsync(cancellationToken);
try
{
var samples = await LoadAsync(cancellationToken);
return samples.Take(maximumCount).ToArray();
}
finally
{
_gate.Release();
}
}
public async Task AcknowledgeAsync(string idempotencyKey, CancellationToken cancellationToken)
{
await _gate.WaitAsync(cancellationToken);
try
{
var samples = await LoadAsync(cancellationToken);
if (samples.Count == 0)
{
return;
}
if (!string.Equals(samples[0].IdempotencyKey, idempotencyKey, StringComparison.Ordinal))
{
throw new InvalidOperationException("Metric acknowledgements must preserve queue order.");
}
samples.RemoveAt(0);
await PersistAsync(samples, cancellationToken);
}
finally
{
_gate.Release();
}
}
private async Task<List<AgentHeartbeat>> LoadAsync(CancellationToken cancellationToken)
{
if (_samples is not null)
{
return _samples;
}
Directory.CreateDirectory(_options.StateDirectory);
if (!File.Exists(_path))
{
_samples = [];
return _samples;
}
try
{
await using var stream = File.OpenRead(_path);
var loaded = await JsonSerializer.DeserializeAsync(
stream,
SmmJsonContext.Default.AgentHeartbeatArray,
cancellationToken) ?? [];
_samples = loaded
.Where(sample => string.Equals(sample.NodeId, _options.NodeId, StringComparison.Ordinal))
.OrderBy(sample => sample.SentAt)
.ToList();
Compact(_samples);
return _samples;
}
catch (JsonException)
{
var corruptPath = $"{_path}.corrupt-{DateTimeOffset.UtcNow:yyyyMMddHHmmss}";
File.Move(_path, corruptPath, overwrite: true);
_samples = [];
return _samples;
}
}
private void Compact(List<AgentHeartbeat> samples)
{
while (samples.Count > _options.BufferMaxSamples)
{
var recentCount = Math.Min(_options.BufferRecentSamples, _options.BufferMaxSamples);
if (recentCount >= samples.Count)
{
samples.RemoveRange(0, samples.Count - _options.BufferMaxSamples);
return;
}
var olderCount = samples.Count - recentCount;
var compacted = new List<AgentHeartbeat>(_options.BufferMaxSamples);
for (var start = 0; start < olderCount; start += _options.BufferDownsampleFactor)
{
var count = Math.Min(_options.BufferDownsampleFactor, olderCount - start);
compacted.Add(samples
.GetRange(start, count)
.MaxBy(ImportanceScore)!);
}
compacted.AddRange(samples.GetRange(olderCount, recentCount));
samples.Clear();
samples.AddRange(compacted.OrderBy(sample => sample.SentAt));
}
}
private async Task PersistAsync(List<AgentHeartbeat> samples, CancellationToken cancellationToken)
{
Directory.CreateDirectory(_options.StateDirectory);
var temporaryPath = $"{_path}.tmp";
await using (var stream = new FileStream(
temporaryPath,
FileMode.Create,
FileAccess.Write,
FileShare.None,
4096,
FileOptions.Asynchronous | FileOptions.WriteThrough))
{
await JsonSerializer.SerializeAsync(
stream,
samples.ToArray(),
SmmJsonContext.Default.AgentHeartbeatArray,
cancellationToken);
await stream.FlushAsync(cancellationToken);
}
SetOwnerOnlyPermissions(temporaryPath);
File.Move(temporaryPath, _path, overwrite: true);
}
private static double ImportanceScore(AgentHeartbeat sample)
{
var memory = Ratio(sample.MemoryUsedBytes, sample.MemoryTotalBytes);
var disk = Ratio(sample.DiskUsedBytes, sample.DiskTotalBytes);
return Math.Max(sample.LoadOne, Math.Max(memory, disk));
}
private static double Ratio(long value, long total)
=> total <= 0 ? 0 : (double)value / total;
private static void SetOwnerOnlyPermissions(string path)
{
if (OperatingSystem.IsLinux() || OperatingSystem.IsMacOS())
{
File.SetUnixFileMode(path, UnixFileMode.UserRead | UnixFileMode.UserWrite);
}
}
}

View file

@ -13,6 +13,19 @@ if (string.IsNullOrWhiteSpace(options.NodeId)
Console.Error.WriteLine("NodeId must contain lowercase letters, digits, or hyphens.");
return 2;
}
if (options.HeartbeatSeconds is < 10 or > 300
|| options.BufferMaxSamples is < 10 or > 10_000
|| options.BufferRecentSamples is < 1
|| options.BufferRecentSamples >= options.BufferMaxSamples
|| options.BufferDownsampleFactor is < 2 or > 100
|| options.UploadBatchSize is < 1 or > 100
|| options.MaxRetrySeconds is < 10 or > 3600)
{
Console.Error.WriteLine(
"Invalid buffer settings: heartbeat 10-300s, max samples 10-10000, recent samples below max, "
+ "downsample factor 2-100, batch size 1-100, retry 10-3600s.");
return 2;
}
using var shutdown = new CancellationTokenSource();
Console.CancelKeyPress += (_, eventArgs) =>

View file

@ -0,0 +1,3 @@
using System.Runtime.CompilerServices;
[assembly: InternalsVisibleTo("ServerMonitorManager.Control.Tests")]

View file

@ -3,5 +3,10 @@
"ControlUrl": "https://127.0.0.1:7443",
"StateDirectory": "/var/lib/ochenstarik-server-monitor-manager/agent",
"CertificateAuthorityPath": "/etc/ochenstarik-server-monitor-manager/control-ca.crt",
"HeartbeatSeconds": 30
"HeartbeatSeconds": 30,
"BufferMaxSamples": 720,
"BufferRecentSamples": 120,
"BufferDownsampleFactor": 4,
"UploadBatchSize": 20,
"MaxRetrySeconds": 300
}

View file

@ -12,5 +12,7 @@ public sealed class ControlOptions
public int HeartbeatSeconds { get; init; } = 30;
public int MaxBufferedMetricAgeHours { get; init; } = 24;
public string HubHelperPath { get; init; } = "/usr/local/libexec/ochenstarik-smm-policy-apply";
}

View file

@ -385,7 +385,7 @@ public sealed class ControlStore(IOptions<ControlOptions> options)
SELECT last_insert_rowid();
""";
insert.Parameters.AddWithValue("$node", heartbeat.NodeId);
insert.Parameters.AddWithValue("$now", now.ToString("O"));
insert.Parameters.AddWithValue("$now", heartbeat.SentAt.ToString("O"));
insert.Parameters.AddWithValue(
"$payload",
JsonSerializer.Serialize(heartbeat, SmmJsonContext.Default.AgentHeartbeat));

View file

@ -34,8 +34,9 @@ builder.Services.AddOptions<ControlOptions>()
.Validate(options =>
!string.IsNullOrWhiteSpace(options.DatabasePath)
&& !string.IsNullOrWhiteSpace(options.CertificateAuthorityPath)
&& options.HeartbeatSeconds is >= 10 and <= 300,
"Control paths are required and HeartbeatSeconds must be between 10 and 300.")
&& options.HeartbeatSeconds is >= 10 and <= 300
&& options.MaxBufferedMetricAgeHours is >= 1 and <= 168,
"Control paths are required, HeartbeatSeconds must be 10-300, and buffered metrics 1-168 hours.")
.ValidateOnStart();
builder.Services.AddSingleton<ControlStore>();
builder.Services.AddSingleton<CertificateAuthority>();
@ -224,7 +225,7 @@ agents.MapPost("/heartbeat", async (
{
if (!NodeIdValidator.IsValid(heartbeat.NodeId)
|| !IdempotencyKeyValidator.IsValid(heartbeat.IdempotencyKey)
|| heartbeat.SentAt < DateTimeOffset.UtcNow.AddMinutes(-5)
|| heartbeat.SentAt < DateTimeOffset.UtcNow.AddHours(-options.Value.MaxBufferedMetricAgeHours)
|| heartbeat.SentAt > DateTimeOffset.UtcNow.AddMinutes(1))
{
return Results.ValidationProblem(new Dictionary<string, string[]>

View file

@ -4,6 +4,7 @@
"CertificateAuthorityPath": "/etc/ochenstarik-server-monitor-manager/control-ca.pfx",
"CertificateAuthorityPassword": null,
"HeartbeatSeconds": 30,
"MaxBufferedMetricAgeHours": 24,
"HubHelperPath": "/usr/local/libexec/ochenstarik-smm-policy-apply"
},
"Logging": {

View file

@ -5,6 +5,7 @@ namespace ServerMonitorManager.Core;
[JsonSerializable(typeof(EnrollmentRequest))]
[JsonSerializable(typeof(EnrollmentResponse))]
[JsonSerializable(typeof(AgentHeartbeat))]
[JsonSerializable(typeof(AgentHeartbeat[]))]
[JsonSerializable(typeof(AgentHeartbeatResponse))]
[JsonSerializable(typeof(AgentSummary[]))]
[JsonSerializable(typeof(DeviceEnrollmentRequest))]

View file

@ -56,6 +56,14 @@ public sealed class ControlStoreTests : IAsyncDisposable
Assert.Equal(first, retry);
Assert.Equal(1, first.Sequence);
await using (var connection = new SqliteConnection(
$"Data Source={Path.Combine(_directory, "control.db")}"))
{
await connection.OpenAsync(cancellationToken);
var command = connection.CreateCommand();
command.CommandText = "SELECT recorded_at FROM metric_samples LIMIT 1;";
Assert.Equal(heartbeat.SentAt.ToString("O"), await command.ExecuteScalarAsync(cancellationToken));
}
await Assert.ThrowsAsync<IdempotencyConflictException>(() => store.RecordHeartbeatAsync(
heartbeat with { LoadOne = 0.9 },
30,

View file

@ -0,0 +1,103 @@
using ServerMonitorManager.Agent;
using ServerMonitorManager.Core;
using Xunit;
namespace ServerMonitorManager.Control.Tests;
public sealed class MetricBufferTests : IAsyncDisposable
{
private readonly string _directory = Path.Combine(
Path.GetTempPath(),
$"smm-buffer-tests-{Guid.NewGuid():N}");
[Fact]
public async Task BufferSurvivesRestartAndAcknowledgesInOrder()
{
var cancellationToken = TestContext.Current.CancellationToken;
var options = CreateOptions(maximum: 20, recent: 5, factor: 3);
var firstBuffer = new MetricBuffer(options);
await firstBuffer.EnqueueAsync(CreateHeartbeat(1), cancellationToken);
await firstBuffer.EnqueueAsync(CreateHeartbeat(2), cancellationToken);
await firstBuffer.EnqueueAsync(CreateHeartbeat(3), cancellationToken);
var restartedBuffer = new MetricBuffer(options);
var pending = await restartedBuffer.PeekAsync(10, cancellationToken);
Assert.Equal(["sample-1", "sample-2", "sample-3"], pending.Select(x => x.IdempotencyKey));
await restartedBuffer.AcknowledgeAsync("sample-1", cancellationToken);
var afterAcknowledgement = await new MetricBuffer(options).PeekAsync(10, cancellationToken);
Assert.Equal(["sample-2", "sample-3"], afterAcknowledgement.Select(x => x.IdempotencyKey));
}
[Fact]
public async Task BufferIsBoundedAndKeepsRecentSamplesAtFullResolution()
{
var cancellationToken = TestContext.Current.CancellationToken;
var options = CreateOptions(maximum: 10, recent: 4, factor: 3);
var buffer = new MetricBuffer(options);
for (var index = 1; index <= 30; index++)
{
await buffer.EnqueueAsync(CreateHeartbeat(index), cancellationToken);
}
var pending = await buffer.PeekAsync(100, cancellationToken);
Assert.InRange(pending.Count, 4, 10);
Assert.Equal(
["sample-27", "sample-28", "sample-29", "sample-30"],
pending.TakeLast(4).Select(x => x.IdempotencyKey));
Assert.True(pending.SequenceEqual(pending.OrderBy(x => x.SentAt)));
}
[Fact]
public async Task CorruptBufferIsQuarantinedInsteadOfStoppingAgent()
{
var cancellationToken = TestContext.Current.CancellationToken;
var options = CreateOptions(maximum: 20, recent: 5, factor: 3);
Directory.CreateDirectory(_directory);
await File.WriteAllTextAsync(
Path.Combine(_directory, "metric-buffer.json"),
"not-json",
cancellationToken);
var buffer = new MetricBuffer(options);
var pending = await buffer.PeekAsync(10, cancellationToken);
Assert.Empty(pending);
Assert.Single(Directory.GetFiles(_directory, "metric-buffer.json.corrupt-*"));
}
public ValueTask DisposeAsync()
{
if (Directory.Exists(_directory))
{
Directory.Delete(_directory, recursive: true);
}
return ValueTask.CompletedTask;
}
private AgentOptions CreateOptions(int maximum, int recent, int factor)
=> new()
{
NodeId = "home",
StateDirectory = _directory,
BufferMaxSamples = maximum,
BufferRecentSamples = recent,
BufferDownsampleFactor = factor
};
private static AgentHeartbeat CreateHeartbeat(int sequence)
=> new(
"home",
"test",
DateTimeOffset.UnixEpoch.AddMinutes(sequence),
sequence,
sequence,
100,
sequence,
100,
sequence,
sequence,
sequence,
$"sample-{sequence}");
}

View file

@ -6,6 +6,7 @@
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\ServerMonitorManager.Agent\ServerMonitorManager.Agent.csproj" />
<ProjectReference Include="..\..\src\ServerMonitorManager.Control\ServerMonitorManager.Control.csproj" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />
<PackageReference Include="xunit.v3" Version="3.2.2" />