fix(dashboard): unify LeaseManager, warm up initial CPU measurement, add live network speed, and clarify YAML comment support

This commit is contained in:
Hermes Team 2026-08-21 22:18:59 +07:00
parent 09d5e5dccb
commit 9f72e8ec39
7 changed files with 246 additions and 43 deletions

View file

@ -0,0 +1,70 @@
# Отчёт: Задание A7 — исправление двух дефектов в данных дашборда
Дата: 2026-08-21
## Идентификаторы и границы
- **START_HEAD (BASE_SHA)**: `09d5e5d1ae62c16f2c3d52dd2b2e88a09f3e4987`
- **Ветка**: `antigravity/dashboard-fixes`
- **origin/main**: `09d5e5d1ae62c16f2c3d52dd2b2e88a09f3e4987`
- **Граница зоны Codex**: ни один файл в `src/antigravity_provider/router/ui/**`, `hermes_hub_app.py`, `tests/test_ui_*.py` **НЕ изменялся** (`git diff --name-only` по этим путям пуст).
- **Тег `v0.1.1`**: **НЕ создавался** (в репозитории `hermes-hub`).
---
## 1. Исправление `active_calls` и унификация `LeaseManager` (P0-1)
- **Причина дефекта**: `RouterEngine` по умолчанию создавал собственный экземпляр `LeaseManager()`, в то время как `state_store.py` и `health_tracker.py` опрашивали синглтон `LeaseManager.get()`.
- **Исправление**:
- `RouterEngine.__init__` теперь инициализирует `self.leases = leases if leases is not None else LeaseManager.get()`.
- `state_store.py` считывает активные лизы через `get_router_engine().leases` (с запасным обращением к `LeaseManager.get()`), обеспечивая единый источник истины.
- `ProfileHealthRecord.active_leases` в `health_tracker.py` заполняется реальным числом занятых лизов для каждого профиля.
- В `cli_commands.py:print_router_status` добавлен вывод активных лизов в столбце `STATE` (`healthy (1 active)`), если `precord.active_leases > 0`.
- **Тест**: добавлен тест `test_active_calls_and_hub_snapshot_integration`, захватывающий лиз через путь роутера (`engine.leases.acquire("ag-w1")`) и проверяющий, что `HubSnapshot.metrics["active_calls_total"] == 1`, `metrics["active_calls_by_profile"]["ag-w1"] == 1` и `snapshot.get_profile("ag-w1").active_leases == 1`.
---
## 2. Устранение холодного 0% CPU на первом замере (P0-2)
- **Причина дефекта**: `psutil.cpu_percent(interval=None)` без предварительного замера по спецификации `psutil` возвращает `0.0%`.
- **Исправление**:
- При первом холодном вызове `HostMetricsService.collect()` выполняется прогрев счетчика через короткий неблокирующий интервал `psutil.cpu_percent(interval=0.05)`, после чего последующие вызовы считывают накопленный дельта-дифференциал через `interval=None`.
- Также счетчик `psutil` предварительно калибруется при импорте модуля `host_metrics.py`.
- **Тест**: добавлен юнит-тест `test_host_metrics_cpu_warmup_avoids_cold_zero`, проверяющий прогрев на холодном старте.
---
## 3. Расчет реальной скорости сети и семантика счетчиков (P1-4)
- В `HostMetricsService` и `HostMetricsSnapshot` реализован расчет мгновенной скорости сети в Мбит/с на основе дельты времени и переданных байт между выборками:
- `net_speed_mbps`: общая скорость сети (Mbps = (delta_sent + delta_recv) * 8 / (dt * 1_000_000));
- `net_sent_mbps` / `net_recv_mbps`: скорость отдачи / приема (Mbps);
- `net_bytes_sent` / `net_bytes_recv`: накопительные счетчики с момента загрузки машины (с явной семантикой в контракте).
- **Тест**: добавлен юнит-тест `test_host_metrics_network_live_speed`, проверяющий расчет Mbps между замерами.
---
## 4. Обновление контракта и статус YAML (P1-3, P1-5)
- В `docs/UI_STATE_CONTRACT.md`:
- В разделе 8.2 детально описаны поля сетевой скорости (`net_speed_mbps`, `net_sent_mbps`, `net_recv_mbps`) и накопительных байт (`net_bytes_sent`, `net_bytes_recv`).
- Добавлено примечание о warm-up поведении CPU.
- Добавлен раздел 9 **Configuration Preservation Status** с честной фиксацией статуса:
- Заголовочные комментарии и пустые строки до первого ключа сохраняются (`supported`);
- Внутренние комментарии внутри словарей нормализуются стандартным `safe_dump` (статус **«частично» / «partially supported»**).
---
## 5. Результаты проверок
- **Headless pytest** (Python 3.8):
`pytest -v` → **195 passed, 26 skipped, 3 deselected in 11.79s**
- **Full pytest** (Python 3.12 с `customtkinter`, `pillow`, `psutil`):
`& "C:\Users\trush\AppData\Local\Programs\Python\Python312\python.exe" -m pytest -v` → **195 passed, 26 skipped, 3 deselected in 11.02s**
- **Ruff linter**:
`ruff check .` → **All checks passed!**
- **Release Gate**:
`python scripts/release_gate.py`**7/7 PASSED** (`[RELEASE GATE: PASSED] All criteria verified. Ready for Candidate v0.1.1`)
- **Live Update Feed**:
`[MANIFEST_LIVE=True, PACKAGE_LIVE=True, PACKAGE_HASH_VERIFIED=True]` (sha256 `b5bbdea2a7a2157a26389266aab07ab3602bb00b4612065c48defec9d6fe909c`)
- **UI Zone Isolation**: `0 files modified in UI area`

View file

@ -262,13 +262,28 @@ Accessible at `HubSnapshot.metrics["host"]`:
| `disk_percent` | `Optional[float]` | `psutil` | Root disk partition utilization percentage (`0.0` to `100.0`), or `None` if unavailable. |
| `disk_used_gb` | `Optional[float]` | `psutil` | Used disk storage in gigabytes. |
| `disk_total_gb` | `Optional[float]` | `psutil` | Total disk storage in gigabytes. |
| `net_bytes_sent` | `Optional[int]` | `psutil` | Total network bytes sent since host boot. |
| `net_bytes_recv` | `Optional[int]` | `psutil` | Total network bytes received since host boot. |
| `net_speed_mbps` | `Optional[float]` | `psutil` | Live total network throughput in Megabits per second (Mbps) computed across sampling intervals, or `None` on initial sample. |
| `net_sent_mbps` | `Optional[float]` | `psutil` | Live outbound network throughput in Megabits per second (Mbps). |
| `net_recv_mbps` | `Optional[float]` | `psutil` | Live inbound network throughput in Megabits per second (Mbps). |
| `net_bytes_sent` | `Optional[int]` | `psutil` | Cumulative bytes sent since host boot (raw counter). |
| `net_bytes_recv` | `Optional[int]` | `psutil` | Cumulative bytes received since host boot (raw counter). |
> **Measurement Note (CPU Warm-up):** The first CPU measurement is pre-warmed during service initialization to prevent cold-start `0.0%` artifacts. Subsequent measurements read the differential counters non-blockingly.
### 8.3 Active Calls Telemetry (`source: "own_measurement"`)
- `HubSnapshot.metrics["active_calls_total"]`: `int` (Total ongoing concurrency leases managed across all profiles).
- `HubSnapshot.metrics["active_calls_total"]`: `int` (Total ongoing concurrency leases managed across all profiles by `RouterEngine`).
- `HubSnapshot.metrics["active_calls_by_profile"]`: `Dict[str, int]` (Active concurrency leases per profile ID).
- `ProfileViewModel.active_leases`: `int` (Current number of active leases for this specific profile).
---
## 9. Configuration Preservation Status
| Component | Status | Behavior & Details |
|---|---|---|
| **Header Comments & Structure** | **Supported** | All leading YAML comments, document banners, and blank lines before the first dictionary key (`existing_comments`) are preserved across file writes. |
| **Inline Section Annotations** | **Partially Supported** | Inline dictionary comments (such as comments inside `profiles`, `roles`, or `pricing`) are normalized during canonical YAML serialization (`safe_dump`). |

View file

@ -42,6 +42,8 @@ def print_router_status() -> int:
precord = engine.health.get_or_create(pid)
state_display = precord.overall_state
if precord.active_leases > 0:
state_display = f"{precord.overall_state} ({precord.active_leases} active)"
reset_display = "-"
if precord.overall_state != "healthy":
cooldown_remaining = max([int(f.reset_at - time.time()) for f in precord.families.values() if f.reset_at and f.reset_at > time.time()] or [0])

View file

@ -1,22 +1,30 @@
"""Hermes Hub — System Host Metrics Service using psutil.
Collects empirical host hardware indicators:
- CPU utilization percentage
- CPU utilization percentage (warmed up on initial sample)
- RAM (used/total MB, percent)
- Disk (used/total GB, percent for system root)
- Network I/O (bytes sent/recv)
- Network live speed (Mbps) and cumulative I/O counters (bytes sent/recv since boot)
Source: 'host_measurement'.
"""
from __future__ import annotations
import logging
import os
import threading
import time
from dataclasses import asdict, dataclass
from typing import Any, Dict, Optional
logger = logging.getLogger("hermes.router.host_metrics")
# Prime psutil internal CPU timer on module import
try:
import psutil
psutil.cpu_percent(interval=None)
except Exception:
pass
@dataclass
class HostMetricsSnapshot:
@ -29,8 +37,11 @@ class HostMetricsSnapshot:
disk_percent: Optional[float] = None
disk_used_gb: Optional[float] = None
disk_total_gb: Optional[float] = None
net_bytes_sent: Optional[int] = None
net_bytes_recv: Optional[int] = None
net_speed_mbps: Optional[float] = None # Live total network speed (Mbps) between samples
net_sent_mbps: Optional[float] = None # Live upload speed (Mbps)
net_recv_mbps: Optional[float] = None # Live download speed (Mbps)
net_bytes_sent: Optional[int] = None # Cumulative bytes sent since host boot
net_bytes_recv: Optional[int] = None # Cumulative bytes received since host boot
source: str = "host_measurement"
has_data: bool = True
@ -41,6 +52,21 @@ class HostMetricsSnapshot:
class HostMetricsService:
"""Safe, non-blocking collector for host performance indicators."""
_last_cpu_time: float = 0.0
_last_net_time: float = 0.0
_last_net_bytes_sent: int = 0
_last_net_bytes_recv: int = 0
_lock = threading.Lock()
@classmethod
def reset_state(cls) -> None:
"""Reset internal sampling baselines (useful for testing)."""
with cls._lock:
cls._last_cpu_time = 0.0
cls._last_net_time = 0.0
cls._last_net_bytes_sent = 0
cls._last_net_bytes_recv = 0
@classmethod
def collect(cls) -> HostMetricsSnapshot:
now = time.time()
@ -54,7 +80,15 @@ class HostMetricsService:
)
try:
cpu = psutil.cpu_percent(interval=None)
with cls._lock:
if cls._last_cpu_time == 0.0:
# Warm up CPU baseline with short interval to prevent cold start 0.0%
cpu = psutil.cpu_percent(interval=0.05)
cls._last_cpu_time = time.time()
else:
cpu = psutil.cpu_percent(interval=None)
cls._last_cpu_time = now
mem = psutil.virtual_memory()
# Disk usage of root drive / partition
@ -62,8 +96,27 @@ class HostMetricsService:
disk = psutil.disk_usage(root_path)
net = None
net_speed_mbps = None
net_sent_mbps = None
net_recv_mbps = None
try:
net = psutil.net_io_counters()
if net:
with cls._lock:
if cls._last_net_time > 0.0 and now > cls._last_net_time:
dt = now - cls._last_net_time
if dt >= 0.05: # Valid time delta
d_sent = max(0, net.bytes_sent - cls._last_net_bytes_sent)
d_recv = max(0, net.bytes_recv - cls._last_net_bytes_recv)
# Convert bytes/sec to Megabits/sec (Mbps)
net_sent_mbps = round((d_sent * 8.0) / (dt * 1_000_000.0), 2)
net_recv_mbps = round((d_recv * 8.0) / (dt * 1_000_000.0), 2)
net_speed_mbps = round(((d_sent + d_recv) * 8.0) / (dt * 1_000_000.0), 2)
cls._last_net_time = now
cls._last_net_bytes_sent = net.bytes_sent
cls._last_net_bytes_recv = net.bytes_recv
except Exception:
pass
@ -76,6 +129,9 @@ class HostMetricsService:
disk_percent=round(float(disk.percent), 1),
disk_used_gb=round(float(disk.used) / (1024 * 1024 * 1024), 1),
disk_total_gb=round(float(disk.total) / (1024 * 1024 * 1024), 1),
net_speed_mbps=net_speed_mbps,
net_sent_mbps=net_sent_mbps,
net_recv_mbps=net_recv_mbps,
net_bytes_sent=int(net.bytes_sent) if net else None,
net_bytes_recv=int(net.bytes_recv) if net else None,
source="host_measurement",

View file

@ -28,7 +28,7 @@ class RouterEngine:
self.config = config or load_router_config()
self.health = health or HealthTracker()
self.affinity = affinity or SessionAffinityTracker(ttl_seconds=self.config.session_affinity_ttl_seconds)
self.leases = leases or LeaseManager()
self.leases = leases if leases is not None else LeaseManager.get()
def reload_config(self) -> None:
self.config = load_router_config()

View file

@ -168,12 +168,18 @@ class HubStateStore:
host_data = {"source": "host_measurement", "has_data": False}
try:
from .session_affinity import LeaseManager
active_leases_total = LeaseManager.get().total_active_count()
active_leases_by_profile = LeaseManager.get().all_active_counts()
from .router_engine import get_router_engine
engine = get_router_engine()
active_leases_total = engine.leases.total_active_count()
active_leases_by_profile = engine.leases.all_active_counts()
except Exception:
active_leases_total = 0
active_leases_by_profile = {}
try:
from .session_affinity import LeaseManager
active_leases_total = LeaseManager.get().total_active_count()
active_leases_by_profile = LeaseManager.get().all_active_counts()
except Exception:
active_leases_total = 0
active_leases_by_profile = {}
metrics = {
"generation": gen,

View file

@ -1,22 +1,23 @@
"""Tests for Hermes Hub Dashboard Data (Assignment A6).
"""Tests for Hermes Hub Dashboard Data (Assignment A6 & A7).
Verifies:
- HubSnapshot includes structured telemetry breakdown (global, by_provider, by_role)
- Provider call_share calculation (e.g. 45% / 35% / 20%) and None on empty window
- Real host system metrics via psutil with 'host_measurement' provenance
- Active calls tracking from LeaseManager reflected in snapshot and ProfileViewModel
- Real host system metrics via psutil with CPU warm-up and live network throughput (Mbps)
- Active calls tracking from RouterEngine.leases reflected in snapshot and ProfileViewModel
"""
from __future__ import annotations
import os
from pathlib import Path
import collections
import time
from unittest.mock import MagicMock, patch
import pytest
from antigravity_provider.router.telemetry_service import TelemetryService
from antigravity_provider.router.host_metrics import HostMetricsService, HostMetricsSnapshot
from antigravity_provider.router.host_metrics import HostMetricsService
from antigravity_provider.router.session_affinity import LeaseManager
from antigravity_provider.router.state_store import HubStateStore, HubSnapshot
from antigravity_provider.router.router_engine import get_router_engine
from antigravity_provider.router.state_store import HubStateStore
from antigravity_provider.router.router_config import (
RolePolicy,
RouterConfig,
@ -44,12 +45,15 @@ def clean_services(tmp_path, monkeypatch):
old_engine = re_mod._ROUTER_ENGINE
re_mod._ROUTER_ENGINE = None
HostMetricsService.reset_state()
with patch.object(TelemetryService, "get", return_value=ts), \
patch.object(LeaseManager, "get", return_value=lm):
yield ts, lm
HubStateStore._instance = old_store
re_mod._ROUTER_ENGINE = old_engine
HostMetricsService.reset_state()
@pytest.mark.unit
@ -125,7 +129,8 @@ def test_provider_call_share_calculation(clean_services):
@pytest.mark.unit
def test_host_metrics_service_psutil():
"""P1-3: Verify HostMetricsService measures system host resources with 'host_measurement' provenance."""
"""P1-3 & P0-2: Verify HostMetricsService measures system host resources with CPU warm-up."""
HostMetricsService.reset_state()
snap = HostMetricsService.collect()
assert snap.source == "host_measurement"
@ -148,27 +153,53 @@ def test_host_metrics_service_psutil():
assert fail_snap.source == "host_measurement"
@pytest.mark.unit
def test_host_metrics_cpu_warmup_avoids_cold_zero():
"""P0-2: Verify initial CPU collection uses interval warm-up to avoid cold start 0.0%."""
HostMetricsService.reset_state()
# Mock psutil to verify that interval=0.05 is passed on cold start
with patch("psutil.cpu_percent", return_value=14.5) as mock_cpu:
snap = HostMetricsService.collect()
mock_cpu.assert_called_once_with(interval=0.05)
assert snap.cpu_percent == 14.5
# Second call should use interval=None
with patch("psutil.cpu_percent", return_value=22.0) as mock_cpu_2:
snap2 = HostMetricsService.collect()
mock_cpu_2.assert_called_once_with(interval=None)
assert snap2.cpu_percent == 22.0
@pytest.mark.unit
def test_host_metrics_network_live_speed():
"""P1-4: Verify live network throughput calculation (Mbps) between samples."""
HostMetricsService.reset_state()
NetCounters = collections.namedtuple("NetCounters", ["bytes_sent", "bytes_recv"])
# First sample: 10,000,000 sent, 20,000,000 recv
with patch("psutil.net_io_counters", return_value=NetCounters(10_000_000, 20_000_000)):
snap1 = HostMetricsService.collect()
assert snap1.net_bytes_sent == 10_000_000
assert snap1.net_bytes_recv == 20_000_000
assert snap1.net_speed_mbps is None # Initial baseline
time.sleep(0.1)
# Second sample: +1,250,000 bytes sent (10 Mbits) and +1,250,000 bytes recv (10 Mbits)
# Total 20 Mbits in ~0.1s -> ~200 Mbps
with patch("psutil.net_io_counters", return_value=NetCounters(11_250_000, 21_250_000)):
snap2 = HostMetricsService.collect()
assert snap2.net_bytes_sent == 11_250_000
assert snap2.net_bytes_recv == 21_250_000
assert snap2.net_speed_mbps is not None
assert snap2.net_speed_mbps > 0.0
@pytest.mark.unit
def test_active_calls_and_hub_snapshot_integration(clean_services, tmp_path, monkeypatch):
"""P0-1 & P1-4: Verify HubSnapshot exposes telemetry breakdown, host metrics, and active leases."""
ts, lm = clean_services
# Record 1 call
ts.record_call(
role="orchestrator",
profile_id="ag-w1",
provider="antigravity",
model="gemini-2.5-pro",
outcome="success",
latency_seconds=0.22,
prompt_tokens=150,
completion_tokens=50,
)
# Acquire an active lease on ag-w1
assert lm.acquire("ag-w1", max_concurrency=2) is True
assert lm.total_active_count() == 1
assert lm.active_count("ag-w1") == 1
"""P0-1: Verify HubSnapshot exposes active leases acquired via router engine path."""
ts, _ = clean_services
config = RouterConfig(
profiles={
@ -188,6 +219,24 @@ def test_active_calls_and_hub_snapshot_integration(clean_services, tmp_path, mon
)
save_router_config(config)
# Acquire lease directly on the router engine's lease manager
engine = get_router_engine()
assert engine.leases.acquire("ag-w1", max_concurrency=2) is True
assert engine.leases.total_active_count() == 1
assert engine.leases.active_count("ag-w1") == 1
# Record 1 call
ts.record_call(
role="orchestrator",
profile_id="ag-w1",
provider="antigravity",
model="gemini-2.5-pro",
outcome="success",
latency_seconds=0.22,
prompt_tokens=150,
completion_tokens=50,
)
store = HubStateStore.get()
snapshot = store.refresh(force_scan=True)
@ -202,6 +251,7 @@ def test_active_calls_and_hub_snapshot_integration(clean_services, tmp_path, mon
assert "host" in snapshot.metrics
assert snapshot.metrics["host"]["source"] == "host_measurement"
# Verify active calls in snapshot matches engine leases
assert "active_calls_total" in snapshot.metrics
assert snapshot.metrics["active_calls_total"] == 1
assert snapshot.metrics["active_calls_by_profile"].get("ag-w1") == 1
@ -211,6 +261,10 @@ def test_active_calls_and_hub_snapshot_integration(clean_services, tmp_path, mon
if prof:
assert prof.active_leases == 1
# Check HealthTracker ProfileHealthRecord.active_leases
prec = engine.health.get_or_create("ag-w1")
assert prec.active_leases == 1
# Release lease and verify count drops to 0
lm.release("ag-w1")
assert lm.total_active_count() == 0
engine.leases.release("ag-w1")
assert engine.leases.total_active_count() == 0