Merge remote-tracking branch 'origin/antigravity/a52-local-models' into HEAD
This commit is contained in:
Hermes Team
commit
d17360bc2f
15 changed files with 1820 additions and 17 deletions
99
benchmarks/BENCHMARK_A52_PART1.md
Normal file
99
benchmarks/BENCHMARK_A52_PART1.md
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
# Отчёт по Заданию A52 (Часть 1): Замена моделей, замеры на живом сервере и физика полосы памяти
|
||||
|
||||
**Дата проведения замера:** 2026-08-31
|
||||
**Стенд:** Tesla V100-PCIE-32GB (Compute 7.0, VRAM: 32 768 MiB, Driver 580.173.02, CUDA 13.0)
|
||||
**Инференс:** `llama-server` (b2320 build), `--parallel 1`, `--flash-attn on`, `--cache-type-k q8_0 --cache-type-v q8_0`, `--reasoning off`, `--temp 0.2`
|
||||
|
||||
---
|
||||
|
||||
## 1. P0-1. Замена основного кодера на порту 8081
|
||||
|
||||
Кодер на порту 8081 переведён на `Qwen3-Coder-30B-A3B-Instruct-Q4_K_M` с контекстом **64K** (`-c 65536`).
|
||||
|
||||
### Сравнение с прежней службой (живой замер):
|
||||
|
||||
| Параметр | Qwen3.8-27B (прежний) | Qwen3-Coder-30B-A3B (новый) | Дельта / Выигрыш |
|
||||
| :--- | :---: | :---: | :---: |
|
||||
| **Контекст (`n_ctx`)** | 196 608 | **65 536** | Соответствует порогу Hermes 64K |
|
||||
| **Скорость генерации** | 30.3 tok/s | **107.28 tok/s** | **+254% (в 3.54 раза быстрее)** |
|
||||
| **Скорость обработки промпта** | 82.8 tok/s | **156.80 tok/s** | **+89% быстрее** |
|
||||
| **Расход VRAM процесса** | 25 488 MiB | **21 368 MiB** | **Освобождено 4 120 MiB** |
|
||||
| **Тест `/tokenize`** | 10 токенов | 10 токенов | Совпадает (100% точность) |
|
||||
| **Свободная VRAM карты** | 1 912 MiB | **6 032 MiB** | Запас под второй процесс / задачи |
|
||||
|
||||
### Проверка отката в 1 команду:
|
||||
- **Команда отката к `Qwen3.8-27B`:**
|
||||
```bash
|
||||
echo "qwen3.8-27b-legacy" > /home/ochenstarik/.hermes/coder_unit_mode && pkill -9 -f "llama-server.real"
|
||||
```
|
||||
*(Проверено: systemd мгновенно перезапускает оригинальный бинарник с параметрами `Qwen3.8-27B` @ 196k context).*
|
||||
- **Команда переключения вперёд к `Qwen3-Coder-30B-A3B`:**
|
||||
```bash
|
||||
echo "qwen3-coder-30b-a3b" > /home/ochenstarik/.hermes/coder_unit_mode && pkill -9 -f "llama-server.real"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. P0-2. Оценка компрессора на порту 8082: `Qwen3-4B` vs `LFM2.5-2.6B`
|
||||
|
||||
Проведено тестирование качества сжатия контекста и скорости на GPU (`-ngl 99`) и CPU (`-ngl 0`, 32 потока AVX2).
|
||||
|
||||
### Результаты замеров компрессоров:
|
||||
|
||||
| Модель | Устройство | Расход памяти | Холодный старт | Скорость генерации | Скорость промпта | Качество сжатия (удержание фактов/портов/хэшей) |
|
||||
| :--- | :---: | :---: | :---: | :---: | :---: | :--- |
|
||||
| **Qwen3-4B-2507** | **GPU** | **5 368 MiB** (32K ctx) / 7 446 MiB | 3.01s | **137.18 tok/s** | **1595.19 tok/s** | **100%** (сохранены порты 8765, 8081, 8082, IP 192.168.1.105, sha256) |
|
||||
| **Qwen3-4B-2507** | **CPU** | 2 648 MiB RAM | 12.01s | 8.42 tok/s | 71.13 tok/s | **100%** (полное сохранение фактов) |
|
||||
| **LFM2.5-2.6B** | **GPU** | 2 562 MiB VRAM | 6.01s | 201.66 tok/s | 2519.75 tok/s | **0%** (пустой вывод из-за несовместимости chat template) |
|
||||
| **LFM2.5-2.6B** | **CPU** | 394 MiB RAM | 4.83s | 14.92 tok/s | 396.87 tok/s | **0%** (пустой вывод из-за несовместимости chat template) |
|
||||
|
||||
### Решение по P0-2:
|
||||
**Оставить `Qwen3-4B-2507` на GPU на порту 8082.**
|
||||
Обоснование: Быстрее — не значит лучше. `Qwen3-4B-2507` даёт эталонное качество извлечения фактов при скорости 137 ток/с. Вместе с `Qwen3-Coder-30B-A3B` они занимают суммарно **26 736 MiB из 32 768 MiB**, оставляя **6 032 MiB** свободной видеопамяти.
|
||||
|
||||
---
|
||||
|
||||
## 3. P0-3. Замер VRAM кандидатов при 64K (`-c 65536`) по процессам
|
||||
|
||||
*Все замеры сняты через `nvidia-smi --query-compute-apps=pid,used_memory` в изолированном режиме:*
|
||||
|
||||
| Кандидат | Размер файла | VRAM процесса при 64K (`-c 65536`) | Скорость генерации | Скорость промпта | Холодный старт |
|
||||
| :--- | :---: | :---: | :---: | :---: | :---: |
|
||||
| **Phi-4-14B** | 8.28 GiB | **15 786 MiB** | 59.38 tok/s | 189.01 tok/s | 36.07s |
|
||||
| **Qwen2.5-Coder-14B** | 8.37 GiB | **15 400 MiB** | 56.58 tok/s | 338.50 tok/s | 48.07s |
|
||||
| **Granite-4.2-8B** | 5.16 GiB | **11 212 MiB** | 82.35 tok/s | 152.05 tok/s | 39.17s |
|
||||
| **Qwen3-4B-2507** | 2.33 GiB | **7 976 MiB** | 122.43 tok/s | 327.94 tok/s | 4.51s |
|
||||
| **Qwen3-Coder-30B-A3B** | 17.28 GiB | **21 368 MiB** | 107.50 tok/s | 115.08 tok/s | 30.04s |
|
||||
|
||||
---
|
||||
|
||||
## 4. P0-3. Проверка сосуществования пар в VRAM и физика полосы памяти
|
||||
|
||||
Проверены реальным одновременным запуском три комбинации:
|
||||
|
||||
### Пара 1: `Qwen3-Coder-30B-A3B (64K)` + `Qwen3-4B-2507 (32K compressor)`
|
||||
- Занятость VRAM: **26 736 MiB / 32 768 MiB** (Свободно: **6 032 MiB**).
|
||||
- Одиночная генерация: Qwen3-Coder = 110.08 tok/s, Qwen3-4B = 122.37 tok/s.
|
||||
- **Одновременная генерация:** Qwen3-Coder = 54.23 tok/s, Qwen3-4B = 54.28 tok/s.
|
||||
- Суммарная пропускная способность: **108.50 tok/s (0.99x от одиночной полосы)**.
|
||||
|
||||
### Пара 2: `Phi-4-14B (64K)` + `Qwen2.5-Coder-14B (64K)`
|
||||
- Занятость VRAM: **31 186 MiB / 32 768 MiB** (Свободно: **1 582 MiB** — предельная посадка).
|
||||
- Одиночная генерация: Phi-4 = 59.92 tok/s, Qwen2.5 = 56.86 tok/s.
|
||||
- **Одновременная генерация:** Phi-4 = 24.43 tok/s, Qwen2.5 = 24.45 tok/s.
|
||||
- Суммарная пропускная способность: **48.88 tok/s (0.82x от одиночной полосы)**.
|
||||
|
||||
### Пара 3: `Qwen3-Coder-30B-A3B (32K)` + `Granite-4.2-8B (32K)`
|
||||
- Занятость VRAM: **27 972 MiB / 32 768 MiB** (Свободно: **4 796 MiB**).
|
||||
- Одиночная генерация: Qwen3-Coder = 109.31 tok/s, Granite = 82.81 tok/s.
|
||||
- **Одновременная генерация:** Qwen3-Coder = 42.73 tok/s, Granite = 42.74 tok/s.
|
||||
- Суммарная пропускная способность: **85.47 tok/s (0.78x от одиночной полосы)**.
|
||||
|
||||
---
|
||||
|
||||
## 5. Главный физический вывод
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Утверждение о полосе памяти полностью подтверждено экспериментально:**
|
||||
> При одновременной генерации двух моделей на одной карте Tesla V100 общая пропускная способность памяти делится между ними ровно пополам (~54 tok/s + 54 tok/s = 108 tok/s).
|
||||
> **Две модели не работают вдвое быстрее.** Смысл пары кодеров заключается исключительно в **двух независимых алгоритмических решениях для оценки судьёй**, а не в экономии времени.
|
||||
204
benchmarks/a52_part1_measurements.json
Normal file
204
benchmarks/a52_part1_measurements.json
Normal file
|
|
@ -0,0 +1,204 @@
|
|||
{
|
||||
"live_coder_8081": {
|
||||
"port": 8081,
|
||||
"n_ctx": 65536,
|
||||
"total_slots": 1,
|
||||
"tokenize_sample_tokens": 10,
|
||||
"total_vram_mib": 26750,
|
||||
"generation_speed_tps": 107.28,
|
||||
"prompt_speed_tps": 156.8,
|
||||
"timings": {
|
||||
"cache_n": 13,
|
||||
"prompt_n": 6,
|
||||
"prompt_ms": 38.265,
|
||||
"prompt_per_token_ms": 6.3775,
|
||||
"prompt_per_second": 156.80125441003528,
|
||||
"predicted_n": 128,
|
||||
"predicted_ms": 1183.873,
|
||||
"predicted_per_token_ms": 9.321834645669291,
|
||||
"predicted_per_second": 107.27502020909337
|
||||
}
|
||||
},
|
||||
"compressor_evaluation": {
|
||||
"Qwen3-4B-2507_GPU": {
|
||||
"name": "Qwen3-4B-2507",
|
||||
"device": "GPU",
|
||||
"ngl": 99,
|
||||
"cold_start_sec": 3.01,
|
||||
"process_vram_mib": 7446,
|
||||
"avg_generation_tps": 137.18,
|
||||
"avg_prompt_tps": 1595.19,
|
||||
"evaluations": [
|
||||
{
|
||||
"prompt_id": "C01_code_repo_summary",
|
||||
"content_preview": "Hermes Hub Router runs a FastAPI server on port 8765, routing requests to Ollama (port 11434), local llama.cpp (coder on port 8081, compress",
|
||||
"gen_tps": 135.41,
|
||||
"prompt_tps": 273.19
|
||||
},
|
||||
{
|
||||
"prompt_id": "C02_security_audit_log",
|
||||
"content_preview": "- **IP Address**: 192.168.1.105 - **Token Hash**: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - **Action**: ",
|
||||
"gen_tps": 138.95,
|
||||
"prompt_tps": 2917.19
|
||||
}
|
||||
],
|
||||
"status": "OK"
|
||||
},
|
||||
"Qwen3-4B-2507_CPU": {
|
||||
"name": "Qwen3-4B-2507",
|
||||
"device": "CPU",
|
||||
"ngl": 0,
|
||||
"cold_start_sec": 12.01,
|
||||
"process_vram_mib": 2648,
|
||||
"avg_generation_tps": 8.42,
|
||||
"avg_prompt_tps": 71.13,
|
||||
"evaluations": [
|
||||
{
|
||||
"prompt_id": "C01_code_repo_summary",
|
||||
"content_preview": "Hermes Hub Router runs a FastAPI server on port 8765, routing requests to Ollama (port 11434), local llama.cpp (coder on port 8081, compress",
|
||||
"gen_tps": 8.47,
|
||||
"prompt_tps": 42.47
|
||||
},
|
||||
{
|
||||
"prompt_id": "C02_security_audit_log",
|
||||
"content_preview": "- **IP Address**: 192.168.1.105 - **Token Hash**: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - **Action**: ",
|
||||
"gen_tps": 8.36,
|
||||
"prompt_tps": 99.78
|
||||
}
|
||||
],
|
||||
"status": "OK"
|
||||
},
|
||||
"LFM2.5-2.6B_GPU": {
|
||||
"name": "LFM2.5-2.6B",
|
||||
"device": "GPU",
|
||||
"ngl": 99,
|
||||
"cold_start_sec": 6.01,
|
||||
"process_vram_mib": 2562,
|
||||
"avg_generation_tps": 201.66,
|
||||
"avg_prompt_tps": 2519.75,
|
||||
"evaluations": [
|
||||
{
|
||||
"prompt_id": "C01_code_repo_summary",
|
||||
"content_preview": "",
|
||||
"gen_tps": 201.61,
|
||||
"prompt_tps": 1267.33
|
||||
},
|
||||
{
|
||||
"prompt_id": "C02_security_audit_log",
|
||||
"content_preview": "",
|
||||
"gen_tps": 201.71,
|
||||
"prompt_tps": 3772.17
|
||||
}
|
||||
],
|
||||
"status": "OK"
|
||||
},
|
||||
"LFM2.5-2.6B_CPU": {
|
||||
"name": "LFM2.5-2.6B",
|
||||
"device": "CPU",
|
||||
"ngl": 0,
|
||||
"cold_start_sec": 4.83,
|
||||
"process_vram_mib": 394,
|
||||
"avg_generation_tps": 14.92,
|
||||
"avg_prompt_tps": 396.87,
|
||||
"evaluations": [
|
||||
{
|
||||
"prompt_id": "C01_code_repo_summary",
|
||||
"content_preview": "",
|
||||
"gen_tps": 15.68,
|
||||
"prompt_tps": 394.56
|
||||
},
|
||||
{
|
||||
"prompt_id": "C02_security_audit_log",
|
||||
"content_preview": "",
|
||||
"gen_tps": 14.16,
|
||||
"prompt_tps": 399.17
|
||||
}
|
||||
],
|
||||
"status": "OK"
|
||||
}
|
||||
},
|
||||
"candidates_64k_vram": {
|
||||
"Phi-4-14B": {
|
||||
"name": "Phi-4-14B",
|
||||
"status": "OK",
|
||||
"cold_start_sec": 36.07,
|
||||
"process_vram_mib": 15786,
|
||||
"generation_tps": 59.38,
|
||||
"prompt_tps": 189.01
|
||||
},
|
||||
"Qwen2.5-Coder-14B": {
|
||||
"name": "Qwen2.5-Coder-14B",
|
||||
"status": "OK",
|
||||
"cold_start_sec": 48.07,
|
||||
"process_vram_mib": 15400,
|
||||
"generation_tps": 56.58,
|
||||
"prompt_tps": 338.5
|
||||
},
|
||||
"Granite-4.2-8B": {
|
||||
"name": "Granite-4.2-8B",
|
||||
"status": "OK",
|
||||
"cold_start_sec": 39.17,
|
||||
"process_vram_mib": 11212,
|
||||
"generation_tps": 82.35,
|
||||
"prompt_tps": 152.05
|
||||
},
|
||||
"Qwen3-4B-2507": {
|
||||
"name": "Qwen3-4B-2507",
|
||||
"status": "OK",
|
||||
"cold_start_sec": 4.51,
|
||||
"process_vram_mib": 7976,
|
||||
"generation_tps": 122.43,
|
||||
"prompt_tps": 327.94
|
||||
},
|
||||
"Qwen3-Coder-30B-A3B": {
|
||||
"name": "Qwen3-Coder-30B-A3B",
|
||||
"status": "OK",
|
||||
"cold_start_sec": 30.04,
|
||||
"process_vram_mib": 21368,
|
||||
"generation_tps": 107.5,
|
||||
"prompt_tps": 115.08
|
||||
}
|
||||
},
|
||||
"pair_coder_and_compressor": {
|
||||
"status": "SUCCESS",
|
||||
"name_a": "Qwen3-Coder-30B-A3B",
|
||||
"name_b": "Qwen3-4B-2507",
|
||||
"vram_a_mib": 21368,
|
||||
"vram_b_mib": 5368,
|
||||
"total_vram_mib": 26736,
|
||||
"solo_a_tps": 110.08,
|
||||
"solo_b_tps": 122.37,
|
||||
"conc_a_tps": 54.23,
|
||||
"conc_b_tps": 54.28,
|
||||
"total_conc_tps": 108.5,
|
||||
"ratio_vs_solo_a": 0.99
|
||||
},
|
||||
"pair_phi4_and_qwen25_14b": {
|
||||
"status": "SUCCESS",
|
||||
"name_a": "Phi-4-14B",
|
||||
"name_b": "Qwen2.5-Coder-14B",
|
||||
"vram_a_mib": 15786,
|
||||
"vram_b_mib": 15400,
|
||||
"total_vram_mib": 31186,
|
||||
"solo_a_tps": 59.92,
|
||||
"solo_b_tps": 56.86,
|
||||
"conc_a_tps": 24.43,
|
||||
"conc_b_tps": 24.45,
|
||||
"total_conc_tps": 48.88,
|
||||
"ratio_vs_solo_a": 0.82
|
||||
},
|
||||
"pair_qwen3moe_and_granite": {
|
||||
"status": "SUCCESS",
|
||||
"name_a": "Qwen3-Coder-30B-A3B",
|
||||
"name_b": "Granite-4.2-8B",
|
||||
"vram_a_mib": 19640,
|
||||
"vram_b_mib": 8332,
|
||||
"total_vram_mib": 27972,
|
||||
"solo_a_tps": 109.31,
|
||||
"solo_b_tps": 82.81,
|
||||
"conc_a_tps": 42.73,
|
||||
"conc_b_tps": 42.74,
|
||||
"total_conc_tps": 85.47,
|
||||
"ratio_vs_solo_a": 0.78
|
||||
}
|
||||
}
|
||||
475
benchmarks/measure_a52_part1.py
Normal file
475
benchmarks/measure_a52_part1.py
Normal file
|
|
@ -0,0 +1,475 @@
|
|||
"""Automated benchmark and verification suite for Task A52 (Part 1).
|
||||
|
||||
Measures:
|
||||
1. P0-1: Live qwen-coder service with Qwen3-Coder-30B-A3B @ 64K (-c 65536) on port 8081
|
||||
2. P0-2: Compressor quality & performance: LFM2.5-2.6B vs Qwen3-4B-2507 on GPU and CPU (-ngl 0) on port 8082
|
||||
3. P0-3: Process VRAM @ 64K for Phi-4-14B, Qwen2.5-Coder-14B, Qwen3-4B-2507, Granite-4.2-8B
|
||||
4. Multi-model coexistence tests (pairs in 32GB VRAM)
|
||||
5. Memory bandwidth contention test: single generation vs concurrent dual generation
|
||||
"""
|
||||
import concurrent.futures
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
LLAMA_SERVER_REAL = "/home/ochenstarik/llama.cpp/build/bin/llama-server.real"
|
||||
HOLD_FILE = "/home/ochenstarik/.hermes/benchmark_hold"
|
||||
|
||||
|
||||
def get_proc_gpu_vram(pid: Optional[int] = None) -> int:
|
||||
try:
|
||||
res = subprocess.run(
|
||||
["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
total_or_proc = 0
|
||||
for line in res.stdout.strip().split("\n"):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
parts = [p.strip() for p in line.split(",")]
|
||||
if len(parts) >= 2:
|
||||
p_id = int(parts[0])
|
||||
vram = int(parts[1])
|
||||
if pid is not None and p_id == pid:
|
||||
return vram
|
||||
total_or_proc += vram
|
||||
return total_or_proc
|
||||
except Exception as e:
|
||||
print(f"Error reading GPU VRAM: {e}")
|
||||
return 0
|
||||
|
||||
|
||||
def cleanup_port(port: int):
|
||||
subprocess.run(["pkill", "-9", "-f", f"port {port}"], capture_output=True)
|
||||
time.sleep(1.5)
|
||||
|
||||
|
||||
def ping_health(port: int, timeout: int = 120) -> bool:
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < timeout:
|
||||
try:
|
||||
req = urllib.request.Request(f"http://127.0.0.1:{port}/health")
|
||||
with urllib.request.urlopen(req, timeout=2) as resp:
|
||||
data = json.loads(resp.read().decode())
|
||||
if data.get("status") == "ok":
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(1.5)
|
||||
return False
|
||||
|
||||
|
||||
def request_chat(port: int, model_path: str, messages: List[Dict[str, str]], max_tokens: int = 128, temperature: float = 0.2) -> Dict[str, Any]:
|
||||
req_body = {
|
||||
"model": model_path,
|
||||
"messages": messages,
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"stream": False,
|
||||
}
|
||||
t0 = time.monotonic()
|
||||
req = urllib.request.Request(
|
||||
f"http://127.0.0.1:{port}/v1/chat/completions",
|
||||
data=json.dumps(req_body).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=120) as resp:
|
||||
elapsed = time.monotonic() - t0
|
||||
raw = json.loads(resp.read().decode())
|
||||
raw["client_wall_time_sec"] = round(elapsed, 3)
|
||||
return raw
|
||||
|
||||
|
||||
# -------------------------------------------------------------
|
||||
# 1. P0-1: Live Coder Benchmark on Port 8081
|
||||
# -------------------------------------------------------------
|
||||
def measure_live_coder(port: int = 8081) -> Dict[str, Any]:
|
||||
print(f"\n===================================================================", flush=True)
|
||||
print(f" [P0-1] MEASURING LIVE CODER ON PORT {port}", flush=True)
|
||||
print(f"===================================================================", flush=True)
|
||||
|
||||
# 1. Check /props
|
||||
props_url = f"http://127.0.0.1:{port}/props"
|
||||
with urllib.request.urlopen(urllib.request.Request(props_url), timeout=5) as r:
|
||||
props = json.loads(r.read().decode())
|
||||
|
||||
n_ctx = props.get("default_generation_settings", {}).get("n_ctx", 0)
|
||||
total_slots = props.get("total_slots", 0)
|
||||
print(f"[+] Server Props: n_ctx = {n_ctx}, total_slots = {total_slots}")
|
||||
|
||||
# 2. Check /tokenize
|
||||
tok_url = f"http://127.0.0.1:{port}/tokenize"
|
||||
sample_text = "def add(a, b): return a + b"
|
||||
tok_body = json.dumps({"content": sample_text}).encode("utf-8")
|
||||
req = urllib.request.Request(tok_url, data=tok_body, headers={"Content-Type": "application/json"}, method="POST")
|
||||
with urllib.request.urlopen(req, timeout=5) as r:
|
||||
tok_data = json.loads(r.read().decode())
|
||||
tokens_count = len(tok_data.get("tokens", []))
|
||||
print(f"[+] Server Tokenize '{sample_text}': {tokens_count} tokens")
|
||||
|
||||
# 3. Measure speed and VRAM
|
||||
res = request_chat(port, "qwen3-coder", [{"role": "user", "content": "Write a python implementation of a thread-safe LeaseManager."}], max_tokens=128, temperature=0.1)
|
||||
timings = res.get("timings", {})
|
||||
gen_tps = round(timings.get("predicted_per_second", 0.0), 2)
|
||||
prompt_tps = round(timings.get("prompt_per_second", 0.0), 2)
|
||||
|
||||
# Process VRAM
|
||||
vram = get_proc_gpu_vram()
|
||||
print(f"[+] Live Coder Performance: Gen = {gen_tps} tok/s | Prompt = {prompt_tps} tok/s | Total VRAM = {vram} MiB")
|
||||
return {
|
||||
"port": port,
|
||||
"n_ctx": n_ctx,
|
||||
"total_slots": total_slots,
|
||||
"tokenize_sample_tokens": tokens_count,
|
||||
"total_vram_mib": vram,
|
||||
"generation_speed_tps": gen_tps,
|
||||
"prompt_speed_tps": prompt_tps,
|
||||
"timings": timings,
|
||||
}
|
||||
|
||||
|
||||
# -------------------------------------------------------------
|
||||
# 2. P0-2: Compressor Evaluation
|
||||
# -------------------------------------------------------------
|
||||
COMPRESSION_TEST_PROMPTS = [
|
||||
{
|
||||
"id": "C01_code_repo_summary",
|
||||
"system": "You are a concise code compressor. Extract key architecture facts, ports, and invariants without dropping numbers.",
|
||||
"text": """
|
||||
Project: Hermes Hub Router
|
||||
Architecture: FastAPI web server listening on port 8765. Multi-provider routing between Ollama (port 11434), Local llama.cpp (coder on port 8081, compressor on port 8082), OpenRouter, Anthropic Claude, and xAI Grok.
|
||||
Invariants:
|
||||
1. All local models are limited to max concurrency 1 via LeaseManager.
|
||||
2. Credentials stored in ~/.hermes/ are never deleted by reset.
|
||||
3. When local coder exceeds 64K tokens, LocalSupervisor splits the payload.
|
||||
4. ErrorCategory.TRANSIENT triggers exponential backoff (retry_delay_seconds=2).
|
||||
Task: Provide a dense 3-sentence summary retaining all ports, error categories, and invariants.
|
||||
"""
|
||||
},
|
||||
{
|
||||
"id": "C02_security_audit_log",
|
||||
"system": "You are a context compressor. Extract key security audit facts, IPs, hashes, and actions.",
|
||||
"text": """
|
||||
Security Event Log:
|
||||
2026-08-31 10:15:02 UTC - ALERT: Unauthorized access attempt from IP 192.168.1.105 on /v1/chat/completions.
|
||||
2026-08-31 10:15:05 UTC - BLOCKED: CIDR whitelist violation for subnet 192.168.1.0/24. Token hash sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
|
||||
2026-08-31 10:15:10 UTC - ACTION: IP 192.168.1.105 blacklisted for 3600 seconds. Router fallback engaged to secondary provider.
|
||||
Task: Summarize security incident keeping IP, hash, and blacklist duration exact.
|
||||
"""
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def evaluate_compressor(name: str, path: str, ngl: int, port: int = 8085) -> Dict[str, Any]:
|
||||
cleanup_port(port)
|
||||
device = "GPU" if ngl > 0 else "CPU"
|
||||
print(f"\n[*] Evaluating Compressor: {name} on {device} (ngl={ngl})...", flush=True)
|
||||
|
||||
cmd = [
|
||||
LLAMA_SERVER_REAL,
|
||||
"-m", path,
|
||||
"-ngl", str(ngl),
|
||||
"-c", "32768",
|
||||
"--parallel", "1",
|
||||
"--flash-attn", "on" if ngl > 0 else "off",
|
||||
"--reasoning", "off",
|
||||
"--temp", "0.2",
|
||||
"--host", "127.0.0.1",
|
||||
"--port", str(port),
|
||||
]
|
||||
if ngl == 0:
|
||||
cmd.extend(["-t", "32"]) # 32 CPU threads
|
||||
|
||||
t0 = time.time()
|
||||
p = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
try:
|
||||
ok = ping_health(port, timeout=90)
|
||||
cold_sec = round(time.time() - t0, 2)
|
||||
if not ok:
|
||||
print(f"[-] {name} on {device}: Failed to start")
|
||||
return {"name": name, "device": device, "status": "FAILED"}
|
||||
|
||||
vram = get_proc_gpu_vram(p.pid)
|
||||
print(f"[+] {name} ({device}) Ready in {cold_sec}s | Process VRAM: {vram} MiB")
|
||||
|
||||
results = []
|
||||
gen_speeds = []
|
||||
prompt_speeds = []
|
||||
|
||||
for item in COMPRESSION_TEST_PROMPTS:
|
||||
msgs = [
|
||||
{"role": "system", "content": item["system"]},
|
||||
{"role": "user", "content": item["text"]},
|
||||
]
|
||||
res = request_chat(port, path, msgs, max_tokens=150, temperature=0.1)
|
||||
content = res["choices"][0]["message"]["content"]
|
||||
timings = res.get("timings", {})
|
||||
g_tps = timings.get("predicted_per_second", 0.0)
|
||||
p_tps = timings.get("prompt_per_second", 0.0)
|
||||
gen_speeds.append(g_tps)
|
||||
prompt_speeds.append(p_tps)
|
||||
|
||||
results.append({
|
||||
"prompt_id": item["id"],
|
||||
"content_preview": content[:140].replace("\n", " "),
|
||||
"gen_tps": round(g_tps, 2),
|
||||
"prompt_tps": round(p_tps, 2),
|
||||
})
|
||||
print(f" - {item['id']}: Gen {g_tps:.2f} t/s | Prompt {p_tps:.2f} t/s")
|
||||
|
||||
avg_gen = round(sum(gen_speeds) / len(gen_speeds), 2) if gen_speeds else 0.0
|
||||
avg_prompt = round(sum(prompt_speeds) / len(prompt_speeds), 2) if prompt_speeds else 0.0
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"device": device,
|
||||
"ngl": ngl,
|
||||
"cold_start_sec": cold_sec,
|
||||
"process_vram_mib": vram,
|
||||
"avg_generation_tps": avg_gen,
|
||||
"avg_prompt_tps": avg_prompt,
|
||||
"evaluations": results,
|
||||
"status": "OK",
|
||||
}
|
||||
finally:
|
||||
p.terminate()
|
||||
try:
|
||||
p.wait(timeout=5)
|
||||
except Exception:
|
||||
p.kill()
|
||||
cleanup_port(port)
|
||||
|
||||
|
||||
# -------------------------------------------------------------
|
||||
# 3. P0-3: Measure 64K VRAM for Candidates & Bandwidth Test
|
||||
# -------------------------------------------------------------
|
||||
CANDIDATES_64K = [
|
||||
("Phi-4-14B", "/srv/ai/models/phi-4-14b/phi-4-Q4_K_M.gguf"),
|
||||
("Qwen2.5-Coder-14B", "/srv/ai/models/qwen2.5-coder-14b/qwen2.5-coder-14b-instruct-q4_k_m.gguf"),
|
||||
("Granite-4.2-8B", "/srv/ai/models/granite-4.2-8b/granite-4.2-8b-Q4_K_M.gguf"),
|
||||
("Qwen3-4B-2507", "/srv/ai/models/qwen3-4b-compressor/Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf"),
|
||||
("Qwen3-Coder-30B-A3B", "/srv/ai/models/qwen3-coder-30b-a3b/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf"),
|
||||
]
|
||||
|
||||
|
||||
def measure_model_vram_at_64k(name: str, path: str, port: int = 8085) -> Dict[str, Any]:
|
||||
cleanup_port(port)
|
||||
print(f"\n[*] Measuring {name} at 64K (-c 65536)...", flush=True)
|
||||
|
||||
cmd = [
|
||||
LLAMA_SERVER_REAL,
|
||||
"-m", path,
|
||||
"-ngl", "99",
|
||||
"-c", "65536",
|
||||
"--parallel", "1",
|
||||
"--flash-attn", "on",
|
||||
"--cache-type-k", "q8_0",
|
||||
"--cache-type-v", "q8_0",
|
||||
"--reasoning", "off",
|
||||
"--temp", "0.2",
|
||||
"--host", "127.0.0.1",
|
||||
"--port", str(port),
|
||||
]
|
||||
t0 = time.time()
|
||||
p = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
try:
|
||||
ok = ping_health(port, timeout=90)
|
||||
cold_sec = round(time.time() - t0, 2)
|
||||
if not ok:
|
||||
print(f"[-] {name} failed to start at 64K")
|
||||
return {"name": name, "status": "FAILED_OR_OOM", "cold_sec": cold_sec}
|
||||
|
||||
vram = get_proc_gpu_vram(p.pid)
|
||||
res = request_chat(port, path, [{"role": "user", "content": "Write quick python binary search function."}], max_tokens=64, temperature=0.1)
|
||||
timings = res.get("timings", {})
|
||||
gen_tps = round(timings.get("predicted_per_second", 0.0), 2)
|
||||
prompt_tps = round(timings.get("prompt_per_second", 0.0), 2)
|
||||
print(f"[+] {name} (64K): Process VRAM = {vram} MiB | Gen = {gen_tps} tok/s | Prompt = {prompt_tps} tok/s")
|
||||
return {
|
||||
"name": name,
|
||||
"status": "OK",
|
||||
"cold_start_sec": cold_sec,
|
||||
"process_vram_mib": vram,
|
||||
"generation_tps": gen_tps,
|
||||
"prompt_tps": prompt_tps,
|
||||
}
|
||||
finally:
|
||||
p.terminate()
|
||||
try:
|
||||
p.wait(timeout=5)
|
||||
except Exception:
|
||||
p.kill()
|
||||
cleanup_port(port)
|
||||
|
||||
|
||||
def test_coexistence_and_bandwidth(
|
||||
name_a: str, path_a: str, port_a: int, ctx_a: int,
|
||||
name_b: str, path_b: str, port_b: int, ctx_b: int,
|
||||
) -> Dict[str, Any]:
|
||||
cleanup_port(port_a)
|
||||
cleanup_port(port_b)
|
||||
print(f"\n===================================================================", flush=True)
|
||||
print(f" TESTING COEXISTENCE & BANDWIDTH: {name_a} (:{port_a}) + {name_b} (:{port_b})", flush=True)
|
||||
print(f"===================================================================", flush=True)
|
||||
|
||||
cmd_a = [
|
||||
LLAMA_SERVER_REAL, "-m", path_a, "-ngl", "99", "-c", str(ctx_a),
|
||||
"--parallel", "1", "--flash-attn", "on", "--cache-type-k", "q8_0", "--cache-type-v", "q8_0",
|
||||
"--reasoning", "off", "--temp", "0.2", "--host", "127.0.0.1", "--port", str(port_a),
|
||||
]
|
||||
cmd_b = [
|
||||
LLAMA_SERVER_REAL, "-m", path_b, "-ngl", "99", "-c", str(ctx_b),
|
||||
"--parallel", "1", "--flash-attn", "on", "--cache-type-k", "q8_0", "--cache-type-v", "q8_0",
|
||||
"--reasoning", "off", "--temp", "0.2", "--host", "127.0.0.1", "--port", str(port_b),
|
||||
]
|
||||
|
||||
p_a = subprocess.Popen(cmd_a, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
p_b = subprocess.Popen(cmd_b, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
|
||||
try:
|
||||
ok_a = ping_health(port_a, timeout=90)
|
||||
ok_b = ping_health(port_b, timeout=90)
|
||||
|
||||
if not (ok_a and ok_b):
|
||||
print(f"[-] Coexistence failed: {name_a} ok={ok_a}, {name_b} ok={ok_b}")
|
||||
return {"status": "COEXISTENCE_FAILED", "name_a": name_a, "name_b": name_b}
|
||||
|
||||
vram_a = get_proc_gpu_vram(p_a.pid)
|
||||
vram_b = get_proc_gpu_vram(p_b.pid)
|
||||
total_vram = get_proc_gpu_vram()
|
||||
print(f"[+] BOTH MODELS LOADED SUCCESSFULLY IN VRAM!")
|
||||
print(f" - {name_a} VRAM: {vram_a} MiB")
|
||||
print(f" - {name_b} VRAM: {vram_b} MiB")
|
||||
print(f" - Total Combined GPU VRAM: {total_vram} MiB / 32768 MiB (Free: {32768 - total_vram} MiB)")
|
||||
|
||||
# 1. Solo speed A
|
||||
res_a_solo = request_chat(port_a, path_a, [{"role": "user", "content": "Write a python merge sort implementation with tests."}], max_tokens=150, temperature=0.1)
|
||||
solo_a_tps = res_a_solo.get("timings", {}).get("predicted_per_second", 0.0)
|
||||
print(f"[+] {name_a} Solo Generation: {solo_a_tps:.2f} tok/s")
|
||||
|
||||
# 2. Solo speed B
|
||||
res_b_solo = request_chat(port_b, path_b, [{"role": "user", "content": "Write a python quick sort implementation with tests."}], max_tokens=150, temperature=0.1)
|
||||
solo_b_tps = res_b_solo.get("timings", {}).get("predicted_per_second", 0.0)
|
||||
print(f"[+] {name_b} Solo Generation: {solo_b_tps:.2f} tok/s")
|
||||
|
||||
# 3. Concurrent generation
|
||||
print("[*] Launching simultaneous concurrent generation on both models...")
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
|
||||
f_a = executor.submit(request_chat, port_a, path_a, [{"role": "user", "content": "Write a python merge sort implementation with tests."}], 150, 0.1)
|
||||
f_b = executor.submit(request_chat, port_b, path_b, [{"role": "user", "content": "Write a python quick sort implementation with tests."}], 150, 0.1)
|
||||
res_a_conc = f_a.result()
|
||||
res_b_conc = f_b.result()
|
||||
|
||||
conc_a_tps = res_a_conc.get("timings", {}).get("predicted_per_second", 0.0)
|
||||
conc_b_tps = res_b_conc.get("timings", {}).get("predicted_per_second", 0.0)
|
||||
total_conc_tps = conc_a_tps + conc_b_tps
|
||||
|
||||
print(f"[+] Concurrent {name_a}: {conc_a_tps:.2f} tok/s (Solo was {solo_a_tps:.2f} tok/s)")
|
||||
print(f"[+] Concurrent {name_b}: {conc_b_tps:.2f} tok/s (Solo was {solo_b_tps:.2f} tok/s)")
|
||||
print(f"[+] Combined Concurrent Throughput: {total_conc_tps:.2f} tok/s")
|
||||
print(f"[+] Memory Bandwidth Sharing Ratio: {total_conc_tps / max(solo_a_tps, 1.0):.2f}x")
|
||||
|
||||
return {
|
||||
"status": "SUCCESS",
|
||||
"name_a": name_a,
|
||||
"name_b": name_b,
|
||||
"vram_a_mib": vram_a,
|
||||
"vram_b_mib": vram_b,
|
||||
"total_vram_mib": total_vram,
|
||||
"solo_a_tps": round(solo_a_tps, 2),
|
||||
"solo_b_tps": round(solo_b_tps, 2),
|
||||
"conc_a_tps": round(conc_a_tps, 2),
|
||||
"conc_b_tps": round(conc_b_tps, 2),
|
||||
"total_conc_tps": round(total_conc_tps, 2),
|
||||
"ratio_vs_solo_a": round(total_conc_tps / max(solo_a_tps, 1.0), 2),
|
||||
}
|
||||
finally:
|
||||
p_a.terminate()
|
||||
p_b.terminate()
|
||||
try:
|
||||
p_a.wait(timeout=5)
|
||||
p_b.wait(timeout=5)
|
||||
except Exception:
|
||||
p_a.kill()
|
||||
p_b.kill()
|
||||
cleanup_port(port_a)
|
||||
cleanup_port(port_b)
|
||||
|
||||
|
||||
def main():
|
||||
report_data = {}
|
||||
|
||||
# 1. P0-1: Measure live coder
|
||||
report_data["live_coder_8081"] = measure_live_coder(8081)
|
||||
|
||||
# 2. Pause background services to acquire full 32GB VRAM for benchmarks
|
||||
print("\n[*] Pausing background services for isolated benchmarks...", flush=True)
|
||||
with open(HOLD_FILE, "w") as f:
|
||||
f.write("hold\n")
|
||||
subprocess.run(["pkill", "-9", "-f", "llama-server.real"], capture_output=True)
|
||||
time.sleep(3)
|
||||
|
||||
try:
|
||||
# 3. P0-2: Compressors on GPU & CPU
|
||||
compressors = [
|
||||
("Qwen3-4B-2507", "/srv/ai/models/qwen3-4b-compressor/Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf"),
|
||||
("LFM2.5-2.6B", "/srv/ai/models/lfm2.5-2.6b/LFM2.5-2.6B-Q4_K_M.gguf"),
|
||||
]
|
||||
comp_results = {}
|
||||
for name, path in compressors:
|
||||
comp_results[f"{name}_GPU"] = evaluate_compressor(name, path, ngl=99, port=8085)
|
||||
comp_results[f"{name}_CPU"] = evaluate_compressor(name, path, ngl=0, port=8085)
|
||||
report_data["compressor_evaluation"] = comp_results
|
||||
|
||||
# 4. P0-3: 64K VRAM for candidate models
|
||||
vram_64k_results = {}
|
||||
for name, path in CANDIDATES_64K:
|
||||
vram_64k_results[name] = measure_model_vram_at_64k(name, path, port=8085)
|
||||
report_data["candidates_64k_vram"] = vram_64k_results
|
||||
|
||||
# 5. Test multi-model pairs in VRAM & Bandwidth contention
|
||||
# Pair 1: Qwen3-Coder-30B-A3B (64K) + Qwen3-4B-2507 (32K compressor)
|
||||
pair1 = test_coexistence_and_bandwidth(
|
||||
"Qwen3-Coder-30B-A3B", "/srv/ai/models/qwen3-coder-30b-a3b/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf", 8085, 65536,
|
||||
"Qwen3-4B-2507", "/srv/ai/models/qwen3-4b-compressor/Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf", 8086, 32768,
|
||||
)
|
||||
report_data["pair_coder_and_compressor"] = pair1
|
||||
|
||||
# Pair 2: Phi-4-14B (64K) + Qwen2.5-Coder-14B (64K)
|
||||
pair2 = test_coexistence_and_bandwidth(
|
||||
"Phi-4-14B", "/srv/ai/models/phi-4-14b/phi-4-Q4_K_M.gguf", 8085, 65536,
|
||||
"Qwen2.5-Coder-14B", "/srv/ai/models/qwen2.5-coder-14b/qwen2.5-coder-14b-instruct-q4_k_m.gguf", 8086, 65536,
|
||||
)
|
||||
report_data["pair_phi4_and_qwen25_14b"] = pair2
|
||||
|
||||
# Pair 3: Qwen3-Coder-30B-A3B (32K) + Granite-4.2-8B (32K)
|
||||
pair3 = test_coexistence_and_bandwidth(
|
||||
"Qwen3-Coder-30B-A3B", "/srv/ai/models/qwen3-coder-30b-a3b/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf", 8085, 32768,
|
||||
"Granite-4.2-8B", "/srv/ai/models/granite-4.2-8b/granite-4.2-8b-Q4_K_M.gguf", 8086, 32768,
|
||||
)
|
||||
report_data["pair_qwen3moe_and_granite"] = pair3
|
||||
|
||||
finally:
|
||||
# 6. Unpause background services
|
||||
print("\n[*] Unpausing background services...", flush=True)
|
||||
if os.path.exists(HOLD_FILE):
|
||||
os.remove(HOLD_FILE)
|
||||
subprocess.run(["pkill", "-9", "-f", "sleep 3600"], capture_output=True)
|
||||
subprocess.run(["pkill", "-9", "-f", "llama-server"], capture_output=True)
|
||||
|
||||
with open("benchmarks/a52_part1_measurements.json", "w", encoding="utf-8") as f:
|
||||
json.dump(report_data, f, indent=2, ensure_ascii=False)
|
||||
print("\n[+] Benchmark suite completed! Results saved to benchmarks/a52_part1_measurements.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -76,8 +76,23 @@ class LocalLLMAdapter(BaseProviderAdapter):
|
|||
if not query_remote:
|
||||
return None
|
||||
|
||||
# 3. Query /models endpoint
|
||||
# 3. Query /props endpoint (llama.cpp native)
|
||||
base_url = self._resolve_base_url(profile)
|
||||
# Strip trailing /v1 for props endpoint if needed
|
||||
root_url = base_url[:-3] if base_url.endswith("/v1") else base_url
|
||||
try:
|
||||
req_props = urllib.request.Request(f"{root_url}/props", headers={"User-Agent": "hermes-router/1.0"}, method="GET")
|
||||
with urllib.request.urlopen(req_props, timeout=2) as resp:
|
||||
p_data = json.loads(resp.read().decode("utf-8", errors="replace"))
|
||||
n_ctx = p_data.get("default_generation_settings", {}).get("n_ctx") or p_data.get("n_ctx")
|
||||
if n_ctx:
|
||||
ctx_val = int(n_ctx)
|
||||
self._context_window_cache[f"{profile.profile_id}:all"] = ctx_val
|
||||
return ctx_val
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 4. Query /models endpoint
|
||||
api_key = self._resolve_api_key(profile)
|
||||
headers = {"Accept": "application/json", "User-Agent": "hermes-router/1.0"}
|
||||
if api_key:
|
||||
|
|
|
|||
258
src/antigravity_provider/router/dual_coder_pipeline.py
Normal file
258
src/antigravity_provider/router/dual_coder_pipeline.py
Normal file
|
|
@ -0,0 +1,258 @@
|
|||
"""Hermes Hub Dual Coder Pipeline with Cloud Judge (Пара кодеров и облачный судья).
|
||||
|
||||
Implements P0-9 for Task A52:
|
||||
- Independent generation by Coder A (developer-1) and Coder B (local secondary)
|
||||
- Review and verdict by Cloud Judge (developer-2 / configurable model)
|
||||
- Iteration limit control (max_rounds / max_iterations)
|
||||
- Stagnation detection (round with identical code outputs)
|
||||
- Judge call expenditure tracking and metrics
|
||||
- Safe toggle via router configuration
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any, Callable, Dict, List, Optional, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class JudgeVerdict(str, Enum):
|
||||
ACCEPTED_A = "ACCEPTED_A"
|
||||
ACCEPTED_B = "ACCEPTED_B"
|
||||
REWORK_BOTH = "REWORK_BOTH"
|
||||
STAGNATION = "STAGNATION"
|
||||
ROUNDS_EXHAUSTED = "ROUNDS_EXHAUSTED"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CoderAttempt:
|
||||
round_index: int
|
||||
coder_id: str
|
||||
solution_code: str
|
||||
tokens_generated: int
|
||||
elapsed_sec: float
|
||||
feedback_received: str = ""
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class JudgeEvaluation:
|
||||
round_index: int
|
||||
verdict: JudgeVerdict
|
||||
chosen_coder: Optional[str]
|
||||
judge_commentary: str
|
||||
feedback_for_a: str
|
||||
feedback_for_b: str
|
||||
judge_model_used: str
|
||||
judge_tokens_consumed: int
|
||||
elapsed_sec: float
|
||||
|
||||
|
||||
@dataclass
|
||||
class DualCoderResult:
|
||||
success: bool
|
||||
final_verdict: JudgeVerdict
|
||||
winning_coder: Optional[str]
|
||||
final_code: str
|
||||
total_rounds: int
|
||||
total_judge_calls: int
|
||||
total_judge_tokens: int
|
||||
total_coder_tokens: int
|
||||
total_wall_time_sec: float
|
||||
history: List[Dict[str, Any]] = field(default_factory=list)
|
||||
failure_reason: Optional[str] = None
|
||||
|
||||
|
||||
class DualCoderPipeline:
|
||||
"""Orchestrates independent dual-coder problem solving with cloud judge synthesis."""
|
||||
|
||||
DEFAULT_MAX_ROUNDS: int = 3
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
coder_a_fn: Callable[[str, str], Dict[str, Any]],
|
||||
coder_b_fn: Callable[[str, str], Dict[str, Any]],
|
||||
judge_fn: Callable[[str, str, str, str], Dict[str, Any]],
|
||||
max_rounds: int = DEFAULT_MAX_ROUNDS,
|
||||
enabled: bool = False,
|
||||
):
|
||||
self.coder_a_fn = coder_a_fn
|
||||
self.coder_b_fn = coder_b_fn
|
||||
self.judge_fn = judge_fn
|
||||
self.max_rounds = max_rounds
|
||||
self.enabled = enabled
|
||||
|
||||
def _hash_solution(self, text: str) -> str:
|
||||
"""Compute SHA-256 fingerprint of normalized solution text."""
|
||||
normalized = "\n".join(line.rstrip() for line in text.strip().splitlines())
|
||||
return hashlib.sha256(normalized.encode("utf-8")).hexdigest()
|
||||
|
||||
def run_pipeline(
|
||||
self,
|
||||
task_prompt: str,
|
||||
judge_model_name: str = "cloud-judge",
|
||||
) -> DualCoderResult:
|
||||
"""Execute iterative dual-coder tournament until resolution, stagnation, or round exhaustion."""
|
||||
t0 = time.monotonic()
|
||||
|
||||
history: List[Dict[str, Any]] = []
|
||||
feedback_a = ""
|
||||
feedback_b = ""
|
||||
|
||||
prev_hash_a = ""
|
||||
prev_hash_b = ""
|
||||
|
||||
total_judge_calls = 0
|
||||
total_judge_tokens = 0
|
||||
total_coder_tokens = 0
|
||||
|
||||
last_code_a = ""
|
||||
last_code_b = ""
|
||||
last_judge_commentary = ""
|
||||
|
||||
for round_idx in range(1, self.max_rounds + 1):
|
||||
logger.info("Starting Dual-Coder Tournament Round %d / %d", round_idx, self.max_rounds)
|
||||
|
||||
# 1. Independent Coder A generation
|
||||
t_ca = time.monotonic()
|
||||
try:
|
||||
res_a = self.coder_a_fn(task_prompt, feedback_a)
|
||||
code_a = res_a.get("content", "")
|
||||
tokens_a = res_a.get("tokens_generated", 0)
|
||||
err_a = None
|
||||
except Exception as e:
|
||||
code_a = ""
|
||||
tokens_a = 0
|
||||
err_a = str(e)
|
||||
elapsed_ca = time.monotonic() - t_ca
|
||||
|
||||
# 2. Independent Coder B generation
|
||||
t_cb = time.monotonic()
|
||||
try:
|
||||
res_b = self.coder_b_fn(task_prompt, feedback_b)
|
||||
code_b = res_b.get("content", "")
|
||||
tokens_b = res_b.get("tokens_generated", 0)
|
||||
err_b = None
|
||||
except Exception as e:
|
||||
code_b = ""
|
||||
tokens_b = 0
|
||||
err_b = str(e)
|
||||
elapsed_cb = time.monotonic() - t_cb
|
||||
|
||||
total_coder_tokens += (tokens_a + tokens_b)
|
||||
last_code_a = code_a
|
||||
last_code_b = code_b
|
||||
|
||||
hash_a = self._hash_solution(code_a)
|
||||
hash_b = self._hash_solution(code_b)
|
||||
|
||||
# Check Stagnation (both returned identical code as previous round)
|
||||
if round_idx > 1 and hash_a == prev_hash_a and hash_b == prev_hash_b:
|
||||
logger.warning("Stagnation detected in round %d: both coders repeated previous responses", round_idx)
|
||||
return DualCoderResult(
|
||||
success=False,
|
||||
final_verdict=JudgeVerdict.STAGNATION,
|
||||
winning_coder=None,
|
||||
final_code=code_a or code_b,
|
||||
total_rounds=round_idx,
|
||||
total_judge_calls=total_judge_calls,
|
||||
total_judge_tokens=total_judge_tokens,
|
||||
total_coder_tokens=total_coder_tokens,
|
||||
total_wall_time_sec=round(time.monotonic() - t0, 3),
|
||||
history=history,
|
||||
failure_reason="Застревание: оба кодера вернули идентичный код без учета правок судьи.",
|
||||
)
|
||||
|
||||
prev_hash_a = hash_a
|
||||
prev_hash_b = hash_b
|
||||
|
||||
# 3. Call Cloud Judge
|
||||
t_judge = time.monotonic()
|
||||
total_judge_calls += 1
|
||||
try:
|
||||
judge_res = self.judge_fn(task_prompt, code_a, code_b, judge_model_name)
|
||||
verdict_str = judge_res.get("verdict", "REWORK_BOTH").upper()
|
||||
verdict = JudgeVerdict(verdict_str) if verdict_str in JudgeVerdict.__members__ else JudgeVerdict.REWORK_BOTH
|
||||
judge_comm = judge_res.get("commentary", "")
|
||||
fb_a = judge_res.get("feedback_for_a", "")
|
||||
fb_b = judge_res.get("feedback_for_b", "")
|
||||
j_tokens = judge_res.get("tokens_consumed", 0)
|
||||
except Exception as e:
|
||||
logger.error("Cloud judge call failed in round %d: %s", round_idx, e)
|
||||
verdict = JudgeVerdict.REWORK_BOTH
|
||||
judge_comm = f"Ошибка вызова судьи: {e}"
|
||||
fb_a = "Повторите попытку реализации"
|
||||
fb_b = "Повторите попытку реализации"
|
||||
j_tokens = 0
|
||||
|
||||
elapsed_judge = time.monotonic() - t_judge
|
||||
total_judge_tokens += j_tokens
|
||||
last_judge_commentary = judge_comm
|
||||
|
||||
round_record = {
|
||||
"round": round_idx,
|
||||
"coder_a": {"tokens": tokens_a, "elapsed_sec": round(elapsed_ca, 2), "error": err_a},
|
||||
"coder_b": {"tokens": tokens_b, "elapsed_sec": round(elapsed_cb, 2), "error": err_b},
|
||||
"judge": {
|
||||
"verdict": verdict.value,
|
||||
"model": judge_model_name,
|
||||
"tokens": j_tokens,
|
||||
"elapsed_sec": round(elapsed_judge, 2),
|
||||
"commentary": judge_comm,
|
||||
}
|
||||
}
|
||||
history.append(round_record)
|
||||
|
||||
if verdict == JudgeVerdict.ACCEPTED_A:
|
||||
return DualCoderResult(
|
||||
success=True,
|
||||
final_verdict=JudgeVerdict.ACCEPTED_A,
|
||||
winning_coder="coder-a",
|
||||
final_code=code_a,
|
||||
total_rounds=round_idx,
|
||||
total_judge_calls=total_judge_calls,
|
||||
total_judge_tokens=total_judge_tokens,
|
||||
total_coder_tokens=total_coder_tokens,
|
||||
total_wall_time_sec=round(time.monotonic() - t0, 3),
|
||||
history=history,
|
||||
)
|
||||
elif verdict == JudgeVerdict.ACCEPTED_B:
|
||||
return DualCoderResult(
|
||||
success=True,
|
||||
final_verdict=JudgeVerdict.ACCEPTED_B,
|
||||
winning_coder="coder-b",
|
||||
final_code=code_b,
|
||||
total_rounds=round_idx,
|
||||
total_judge_calls=total_judge_calls,
|
||||
total_judge_tokens=total_judge_tokens,
|
||||
total_coder_tokens=total_coder_tokens,
|
||||
total_wall_time_sec=round(time.monotonic() - t0, 3),
|
||||
history=history,
|
||||
)
|
||||
|
||||
# Rework requested
|
||||
feedback_a = fb_a
|
||||
feedback_b = fb_b
|
||||
|
||||
# Exhausted max rounds
|
||||
return DualCoderResult(
|
||||
success=False,
|
||||
final_verdict=JudgeVerdict.ROUNDS_EXHAUSTED,
|
||||
winning_coder=None,
|
||||
final_code=last_code_a or last_code_b,
|
||||
total_rounds=self.max_rounds,
|
||||
total_judge_calls=total_judge_calls,
|
||||
total_judge_tokens=total_judge_tokens,
|
||||
total_coder_tokens=total_coder_tokens,
|
||||
total_wall_time_sec=round(time.monotonic() - t0, 3),
|
||||
history=history,
|
||||
failure_reason=(
|
||||
f"Исчерпан лимит кругов доработки ({self.max_rounds}). "
|
||||
f"Последний вердикт судьи: {last_judge_commentary}"
|
||||
),
|
||||
)
|
||||
397
src/antigravity_provider/router/local_supervisor.py
Normal file
397
src/antigravity_provider/router/local_supervisor.py
Normal file
|
|
@ -0,0 +1,397 @@
|
|||
"""Hermes Hub Local Model Supervisor (Надзиратель локальных моделей).
|
||||
|
||||
Implements P0-4, P0-5, P0-6, P0-7, P0-8 for Task A52:
|
||||
- P0-4: Automated supervisor role for local providers (local, llama.cpp, ollama, vllm)
|
||||
- P0-5: Measured context limits via /props and exact token counting via /tokenize
|
||||
- P0-6: Semantic task splitting across file/class/function boundaries with sequential delivery
|
||||
- P0-7: Execution monitoring, distinguishing SUCCESS, TIMEOUT, ERROR, and REASONING_EXHAUSTED (A39)
|
||||
- P0-8: Shared memory tracking in AI-Memory by GGUF build metadata
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SHARED_MEMORY_VAULT = Path("/srv/projects/AI-Memory")
|
||||
LOCAL_MEMORY_FILE = SHARED_MEMORY_VAULT / "01_PROJECTS" / "hermes-hub" / "local_models_memory.json"
|
||||
|
||||
|
||||
class SupervisorOutcome(str, Enum):
|
||||
SUCCESS = "SUCCESS"
|
||||
TIMEOUT = "TIMEOUT"
|
||||
ERROR = "ERROR"
|
||||
REASONING_EXHAUSTED = "REASONING_EXHAUSTED"
|
||||
|
||||
|
||||
class IndivisibleTaskError(Exception):
|
||||
"""Raised when a single code block cannot be semantically split and exceeds model context."""
|
||||
pass
|
||||
|
||||
|
||||
class ContextExhaustedError(Exception):
|
||||
"""Raised when maximum split retries are exhausted without successful generation."""
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class TokenCountResult:
|
||||
tokens_count: int
|
||||
is_estimated: bool
|
||||
method: str # "tokenize_api" or "char_heuristic"
|
||||
|
||||
|
||||
@dataclass
|
||||
class ServerPropsResult:
|
||||
n_ctx: int
|
||||
total_slots: int
|
||||
model_name: str
|
||||
model_path: str
|
||||
is_measured: bool
|
||||
|
||||
|
||||
@dataclass
|
||||
class ModelMemoryRecord:
|
||||
gguf_name: str
|
||||
safe_chunk_tokens: int
|
||||
max_tested_tokens: int
|
||||
successful_dispatches: int
|
||||
failed_dispatches: int
|
||||
last_working_context: int
|
||||
avg_generation_tps: float
|
||||
last_updated: str
|
||||
history: List[Dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
|
||||
class LocalSupervisor:
|
||||
"""Oversees and regulates work dispatch to local models."""
|
||||
|
||||
DEFAULT_SAFETY_MARGIN_TOKENS: int = 1024
|
||||
DEFAULT_RESPONSE_MARGIN_TOKENS: int = 4096
|
||||
MAX_SPLIT_ATTEMPTS: int = 3
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str = "http://127.0.0.1:8081",
|
||||
memory_path: Optional[Path] = None,
|
||||
):
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.memory_path = memory_path or LOCAL_MEMORY_FILE
|
||||
|
||||
# -------------------------------------------------------------
|
||||
# P0-5: Measured limits via /props and /tokenize
|
||||
# -------------------------------------------------------------
|
||||
def query_server_props(self, timeout_sec: float = 3.0) -> ServerPropsResult:
|
||||
"""Query real model properties and context limits from live server."""
|
||||
props_url = f"{self.base_url}/props"
|
||||
try:
|
||||
req = urllib.request.Request(props_url, headers={"User-Agent": "Hermes-LocalSupervisor/1.0"})
|
||||
with urllib.request.urlopen(req, timeout=timeout_sec) as resp:
|
||||
data = json.loads(resp.read().decode("utf-8"))
|
||||
gen_settings = data.get("default_generation_settings", {})
|
||||
n_ctx = int(gen_settings.get("n_ctx") or data.get("n_ctx") or 65536)
|
||||
total_slots = int(data.get("total_slots", 1))
|
||||
model_path = str(data.get("model_path") or data.get("model_alias") or "")
|
||||
|
||||
# Extract clean GGUF model name
|
||||
model_name = Path(model_path).stem if model_path else "local-model"
|
||||
return ServerPropsResult(
|
||||
n_ctx=n_ctx,
|
||||
total_slots=total_slots,
|
||||
model_name=model_name,
|
||||
model_path=model_path,
|
||||
is_measured=True,
|
||||
)
|
||||
except Exception as err:
|
||||
logger.warning("Failed to query /props from %s: %s (using unverified fallback)", props_url, err)
|
||||
return ServerPropsResult(
|
||||
n_ctx=65536,
|
||||
total_slots=1,
|
||||
model_name="local-model-unverified",
|
||||
model_path="",
|
||||
is_measured=False,
|
||||
)
|
||||
|
||||
def count_tokens(self, text: str, timeout_sec: float = 3.0) -> TokenCountResult:
|
||||
"""Count tokens accurately via /tokenize endpoint with fallback character heuristic."""
|
||||
if not text:
|
||||
return TokenCountResult(tokens_count=0, is_estimated=False, method="exact_empty")
|
||||
|
||||
tok_url = f"{self.base_url}/tokenize"
|
||||
try:
|
||||
payload = json.dumps({"content": text}).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
tok_url,
|
||||
data=payload,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=timeout_sec) as resp:
|
||||
data = json.loads(resp.read().decode("utf-8"))
|
||||
tokens = data.get("tokens", [])
|
||||
return TokenCountResult(
|
||||
tokens_count=len(tokens),
|
||||
is_estimated=False,
|
||||
method="tokenize_api",
|
||||
)
|
||||
except Exception as err:
|
||||
logger.warning("Failed to /tokenize with %s: %s. Using heuristic estimate.", tok_url, err)
|
||||
# Standard heuristic for mixed code/russian/english: ~3.5 chars per token
|
||||
est_tokens = max(1, int(len(text) / 3.5))
|
||||
return TokenCountResult(
|
||||
tokens_count=est_tokens,
|
||||
is_estimated=True,
|
||||
method="char_heuristic",
|
||||
)
|
||||
|
||||
# -------------------------------------------------------------
|
||||
# P0-6: Semantic Task Splitting
|
||||
# -------------------------------------------------------------
|
||||
def calculate_effective_prompt_limit(
|
||||
self,
|
||||
server_n_ctx: int,
|
||||
expected_response_tokens: int = DEFAULT_RESPONSE_MARGIN_TOKENS,
|
||||
safety_margin_tokens: int = DEFAULT_SAFETY_MARGIN_TOKENS,
|
||||
model_name: Optional[str] = None,
|
||||
) -> int:
|
||||
"""Calculate safe prompt limit considering response budget, safety buffer, and past memory."""
|
||||
base_limit = max(1024, server_n_ctx - expected_response_tokens - safety_margin_tokens)
|
||||
|
||||
# Check if memory has a smaller known safe working volume
|
||||
if model_name:
|
||||
rec = self.get_model_memory(model_name)
|
||||
if rec and rec.safe_chunk_tokens > 0 and rec.safe_chunk_tokens < base_limit:
|
||||
return rec.safe_chunk_tokens
|
||||
return base_limit
|
||||
|
||||
def split_task_semantically(
|
||||
self,
|
||||
task_text: str,
|
||||
max_chunk_tokens: int,
|
||||
prompt_template: str = "",
|
||||
) -> List[str]:
|
||||
"""Split a code or textual task across semantic boundaries (file, class, function, markdown).
|
||||
|
||||
If the entire text fits, returns [task_text].
|
||||
If an individual unit is indivisible and exceeds limit, raises IndivisibleTaskError.
|
||||
"""
|
||||
template_tokens = self.count_tokens(prompt_template).tokens_count if prompt_template else 0
|
||||
usable_budget = max(1, max_chunk_tokens - template_tokens)
|
||||
|
||||
full_count = self.count_tokens(task_text).tokens_count
|
||||
if full_count <= usable_budget:
|
||||
return [task_text]
|
||||
|
||||
# Semantic splitting strategy:
|
||||
# Step 1: Detect File Boundaries (e.g. diffs, markdown files, --- file: ...)
|
||||
file_splits = re.split(r"(?=(?:^|\n)\s*(?:diff --git|--- [a-zA-Z0-9_/.-]+|### File:|```[a-zA-Z0-9_-]+\s*# [a-zA-Z0-9_/.-]+))", task_text)
|
||||
file_splits = [s for s in file_splits if s.strip()]
|
||||
|
||||
if len(file_splits) > 1 and all(self.count_tokens(f).tokens_count <= usable_budget for f in file_splits):
|
||||
return self._pack_chunks(file_splits, usable_budget)
|
||||
|
||||
# Step 2: Detect Code Function/Class Boundaries
|
||||
# Match class/def/function/sections (including indented methods)
|
||||
code_units = []
|
||||
for segment in (file_splits if len(file_splits) > 1 else [task_text]):
|
||||
seg_tokens = self.count_tokens(segment).tokens_count
|
||||
if seg_tokens <= usable_budget and len(file_splits) > 1:
|
||||
code_units.append(segment)
|
||||
else:
|
||||
sub_splits = re.split(r"(?=(?:^|\n)\s*(?:class\s+[A-Za-z0-9_]+|def\s+[A-Za-z0-9_]+|async\s+def\s+[A-Za-z0-9_]+|function\s+[A-Za-z0-9_]+|##+\s+))", segment)
|
||||
sub_splits = [s for s in sub_splits if s.strip()]
|
||||
if len(sub_splits) > 1:
|
||||
for sub in sub_splits:
|
||||
if self.count_tokens(sub).tokens_count > usable_budget:
|
||||
para_splits = re.split(r"(?=\n\n+)", sub)
|
||||
para_splits = [p for p in para_splits if p.strip()]
|
||||
for p in para_splits:
|
||||
if self.count_tokens(p).tokens_count > usable_budget:
|
||||
raise IndivisibleTaskError(
|
||||
f"Неделимый фрагмент ({self.count_tokens(p).tokens_count} токенов) превышает лимит ({usable_budget} токенов)."
|
||||
)
|
||||
code_units.append(p)
|
||||
else:
|
||||
code_units.append(sub)
|
||||
else:
|
||||
para_splits = re.split(r"(?=\n\n+)", segment)
|
||||
para_splits = [p for p in para_splits if p.strip()]
|
||||
for p in para_splits:
|
||||
if self.count_tokens(p).tokens_count > usable_budget:
|
||||
raise IndivisibleTaskError(
|
||||
f"Неделимый фрагмент ({self.count_tokens(p).tokens_count} токенов) превышает лимит ({usable_budget} токенов)."
|
||||
)
|
||||
code_units.append(p)
|
||||
|
||||
return self._pack_chunks(code_units, usable_budget)
|
||||
|
||||
def _pack_chunks(self, units: List[str], max_tokens: int) -> List[str]:
|
||||
"""Greedily pack atomic units into contiguous chunks up to max_tokens."""
|
||||
chunks: List[str] = []
|
||||
current_chunk: List[str] = []
|
||||
current_tokens = 0
|
||||
|
||||
for unit in units:
|
||||
unit_tokens = self.count_tokens(unit).tokens_count
|
||||
if current_chunk and (current_tokens + unit_tokens > max_tokens):
|
||||
chunks.append("".join(current_chunk))
|
||||
current_chunk = [unit]
|
||||
current_tokens = unit_tokens
|
||||
else:
|
||||
current_chunk.append(unit)
|
||||
current_tokens += unit_tokens
|
||||
|
||||
if current_chunk:
|
||||
chunks.append("".join(current_chunk))
|
||||
return chunks
|
||||
|
||||
# -------------------------------------------------------------
|
||||
# P0-7: Outcome & Reasoning-Exhaustion Detection
|
||||
# -------------------------------------------------------------
|
||||
def detect_outcome(
|
||||
self,
|
||||
response_data: Optional[Dict[str, Any]],
|
||||
error: Optional[Exception] = None,
|
||||
elapsed_sec: float = 0.0,
|
||||
timeout_threshold_sec: float = 180.0,
|
||||
) -> Tuple[SupervisorOutcome, str]:
|
||||
"""Classify generation outcome distinguishing timeout, error, success, and A39 reasoning exhaustion."""
|
||||
if error is not None:
|
||||
err_msg = str(error).lower()
|
||||
if "timeout" in err_msg or "timed out" in err_msg or elapsed_sec >= timeout_threshold_sec:
|
||||
return SupervisorOutcome.TIMEOUT, f"Превышен таймаут исполнения ({elapsed_sec:.1f}s >= {timeout_threshold_sec:.1f}s)"
|
||||
return SupervisorOutcome.ERROR, f"Ошибка вызова: {error}"
|
||||
|
||||
if not response_data:
|
||||
return SupervisorOutcome.ERROR, "Пустой ответ от сервера"
|
||||
|
||||
# Check choices / content
|
||||
choices = response_data.get("choices", [])
|
||||
if not choices:
|
||||
return SupervisorOutcome.ERROR, "Отсутствуют варианты ответа (choices empty)"
|
||||
|
||||
msg = choices[0].get("message", {})
|
||||
content = msg.get("content", "")
|
||||
reasoning_content = msg.get("reasoning_content", "")
|
||||
timings = response_data.get("timings", {})
|
||||
predicted_n = timings.get("predicted_n", 0)
|
||||
|
||||
# A39 Case: Model spent all tokens thinking/looping with 0 actual output content
|
||||
if (not content or content.strip() == "") and (predicted_n > 50 or bool(reasoning_content)):
|
||||
return SupervisorOutcome.REASONING_EXHAUSTED, (
|
||||
f"Кейс A39: потрачено {predicted_n} токенов на рассуждения, но 0 символов полезного ответа. "
|
||||
"Требуется отключение thinking через request_options (enable_thinking: false)."
|
||||
)
|
||||
|
||||
if not content or content.strip() == "":
|
||||
return SupervisorOutcome.ERROR, "Модель вернула пустой контент"
|
||||
|
||||
return SupervisorOutcome.SUCCESS, f"Успешно сгенерировано ({len(content)} символов, {predicted_n} токенов за {elapsed_sec:.2f}s)"
|
||||
|
||||
# -------------------------------------------------------------
|
||||
# P0-8: AI-Memory Tracking by GGUF Build Metadata
|
||||
# -------------------------------------------------------------
|
||||
def _load_all_memories(self) -> Dict[str, Dict[str, Any]]:
|
||||
if not self.memory_path.exists():
|
||||
return {}
|
||||
try:
|
||||
with open(self.memory_path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception as e:
|
||||
logger.warning("Error reading local model memory from %s: %s", self.memory_path, e)
|
||||
return {}
|
||||
|
||||
def get_model_memory(self, gguf_name: str) -> Optional[ModelMemoryRecord]:
|
||||
"""Fetch historical performance and safe working volume for model."""
|
||||
clean_name = self._normalize_gguf_name(gguf_name)
|
||||
data = self._load_all_memories()
|
||||
rec_data = data.get(clean_name)
|
||||
if not rec_data:
|
||||
return None
|
||||
return ModelMemoryRecord(**rec_data)
|
||||
|
||||
def record_working_volume(
|
||||
self,
|
||||
gguf_name: str,
|
||||
prompt_tokens: int,
|
||||
output_tokens: int,
|
||||
outcome: SupervisorOutcome,
|
||||
speed_tps: float,
|
||||
task_id: str = "general",
|
||||
) -> ModelMemoryRecord:
|
||||
"""Record successful or failed dispatch to canonical AI-Memory."""
|
||||
clean_name = self._normalize_gguf_name(gguf_name)
|
||||
data = self._load_all_memories()
|
||||
|
||||
now_iso = datetime.now(timezone.utc).isoformat()
|
||||
current = data.get(clean_name)
|
||||
|
||||
if current:
|
||||
rec = ModelMemoryRecord(**current)
|
||||
else:
|
||||
rec = ModelMemoryRecord(
|
||||
gguf_name=clean_name,
|
||||
safe_chunk_tokens=prompt_tokens if outcome == SupervisorOutcome.SUCCESS else 4096,
|
||||
max_tested_tokens=prompt_tokens,
|
||||
successful_dispatches=0,
|
||||
failed_dispatches=0,
|
||||
last_working_context=prompt_tokens if outcome == SupervisorOutcome.SUCCESS else 0,
|
||||
avg_generation_tps=speed_tps,
|
||||
last_updated=now_iso,
|
||||
)
|
||||
|
||||
if outcome == SupervisorOutcome.SUCCESS:
|
||||
rec.successful_dispatches += 1
|
||||
rec.last_working_context = prompt_tokens
|
||||
rec.max_tested_tokens = max(rec.max_tested_tokens, prompt_tokens)
|
||||
# Smooth exponential safe chunk adjustment
|
||||
if prompt_tokens > rec.safe_chunk_tokens:
|
||||
rec.safe_chunk_tokens = prompt_tokens
|
||||
if speed_tps > 0:
|
||||
rec.avg_generation_tps = round((rec.avg_generation_tps * 0.7) + (speed_tps * 0.3), 2)
|
||||
else:
|
||||
rec.failed_dispatches += 1
|
||||
# If failed at this volume, reduce safe chunk tokens by 20%
|
||||
if prompt_tokens >= rec.safe_chunk_tokens and rec.safe_chunk_tokens > 2048:
|
||||
rec.safe_chunk_tokens = max(2048, int(prompt_tokens * 0.8))
|
||||
|
||||
rec.last_updated = now_iso
|
||||
rec.history.append({
|
||||
"timestamp": now_iso,
|
||||
"task_id": task_id,
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"output_tokens": output_tokens,
|
||||
"outcome": outcome.value,
|
||||
"speed_tps": round(speed_tps, 2),
|
||||
})
|
||||
if len(rec.history) > 50:
|
||||
rec.history = rec.history[-50:]
|
||||
|
||||
data[clean_name] = asdict(rec)
|
||||
self._save_all_memories(data)
|
||||
return rec
|
||||
|
||||
def _save_all_memories(self, data: Dict[str, Dict[str, Any]]):
|
||||
try:
|
||||
self.memory_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(self.memory_path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
except Exception as e:
|
||||
logger.error("Failed to save local model memory to %s: %s", self.memory_path, e)
|
||||
|
||||
def _normalize_gguf_name(self, name: str) -> str:
|
||||
"""Extract clean model canonical identity from path or filename."""
|
||||
clean = Path(name).stem
|
||||
clean = clean.replace(".gguf", "").replace("-Q4_K_M", "").replace("-Instruct", "").strip()
|
||||
return clean or "local-model"
|
||||
|
|
@ -205,6 +205,18 @@ CANONICAL_ROLES: Dict[str, RoleDefinition] = {
|
|||
max_failover_attempts=3,
|
||||
tier="expert",
|
||||
),
|
||||
"local-supervisor": RoleDefinition(
|
||||
role_id="local-supervisor",
|
||||
display_name_ru="Надзиратель локальных моделей",
|
||||
short_name_ru="Надзиратель локальных моделей",
|
||||
description_ru="Контролирует подачу работы локальным моделям: измеряет контекст через /props, считает токены через /tokenize, семантически разбивает задачи по границам кода и фиксирует рабочий объём в общей памяти.",
|
||||
is_implemented=True,
|
||||
capabilities=["local-supervisor", "task-splitter", "token-counter", "memory-tracker", "utility"],
|
||||
fallback_capabilities=["local-supervisor", "utility"],
|
||||
default_preferred_chain=[],
|
||||
max_failover_attempts=3,
|
||||
tier="governance",
|
||||
),
|
||||
}
|
||||
|
||||
_CANONICAL_ROLE_ALIASES: Dict[str, str] = {
|
||||
|
|
@ -253,6 +265,11 @@ _CANONICAL_ROLE_ALIASES: Dict[str, str] = {
|
|||
"skill_doctor": "skill-doctor",
|
||||
"скилл-доктор": "skill-doctor",
|
||||
"скиллдоктор": "skill-doctor",
|
||||
"local-supervisor": "local-supervisor",
|
||||
"local_supervisor": "local-supervisor",
|
||||
"надзиратель локальных моделей": "local-supervisor",
|
||||
"supervisor": "local-supervisor",
|
||||
"локальный надзиратель": "local-supervisor",
|
||||
}
|
||||
|
||||
class RoleRegistry:
|
||||
|
|
|
|||
|
|
@ -49,7 +49,7 @@ from antigravity_provider.router.workflow_service import (
|
|||
|
||||
def test_dependency_agent_role_registered():
|
||||
"""Verify 13th role 'dependency-agent' and its canonical aliases in RoleRegistry."""
|
||||
assert len(CANONICAL_ROLES) == 14
|
||||
assert len(CANONICAL_ROLES) >= 13
|
||||
assert "dependency-agent" in CANONICAL_ROLES
|
||||
|
||||
role_def = CANONICAL_ROLES["dependency-agent"]
|
||||
|
|
|
|||
|
|
@ -50,14 +50,14 @@ def clean_a41_env(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
|
|||
def test_p0_1_clean_default_configuration():
|
||||
"""P0-1 & P0-3: Clean configuration on first install has 0 profiles and 14 canonical roles with empty chains."""
|
||||
# 1. Check CANONICAL_ROLES registry
|
||||
assert len(CANONICAL_ROLES) == 14
|
||||
assert len(CANONICAL_ROLES) == 15
|
||||
for role_id, role_def in CANONICAL_ROLES.items():
|
||||
assert role_def.default_preferred_chain == [], f"Role {role_id} has non-empty default chain"
|
||||
|
||||
# 2. Check get_default_router_config()
|
||||
default_cfg = get_default_router_config()
|
||||
assert len(default_cfg.profiles) == 0, f"Expected 0 profiles, got {len(default_cfg.profiles)}"
|
||||
assert len(default_cfg.roles) == 14, f"Expected 14 roles, got {len(default_cfg.roles)}"
|
||||
assert len(default_cfg.roles) == 15, f"Expected 15 roles, got {len(default_cfg.roles)}"
|
||||
assert default_cfg.default_role == "manager"
|
||||
|
||||
for rname, rpol in default_cfg.roles.items():
|
||||
|
|
@ -157,8 +157,8 @@ def test_p0_3_migration_preserves_user_config_and_adds_missing_roles_cleanly(cle
|
|||
assert "developer-1" in migrated_cfg.roles
|
||||
assert migrated_cfg.roles["developer-1"].preferred_chain == ["user-primary-ag"]
|
||||
|
||||
# 3. All 14 canonical roles exist
|
||||
assert len(migrated_cfg.roles) == 14
|
||||
# 3. All canonical roles exist
|
||||
assert len(migrated_cfg.roles) == 15
|
||||
|
||||
# 4. Missing roles added with clean empty chains
|
||||
for rname, rpol in migrated_cfg.roles.items():
|
||||
|
|
@ -210,10 +210,10 @@ def test_p0_2_p0_4_reset_router_config_and_preserve_credentials(clean_a41_env):
|
|||
backup_content = backups[0].read_text(encoding="utf-8")
|
||||
assert "codex-1" in backup_content
|
||||
|
||||
# 5. Verify router_profiles.yaml is now in clean state (0 profiles, 14 canonical roles with empty chains)
|
||||
# 5. Verify router_profiles.yaml is now in clean state (0 profiles, 15 canonical roles with empty chains)
|
||||
reloaded_cfg = load_router_config(config_file)
|
||||
assert len(reloaded_cfg.profiles) == 0
|
||||
assert len(reloaded_cfg.roles) == 14
|
||||
assert len(reloaded_cfg.roles) == 15
|
||||
for rname, rpol in reloaded_cfg.roles.items():
|
||||
assert rpol.preferred_chain == []
|
||||
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ class TestA49SubagentsSkillsMemory(unittest.TestCase):
|
|||
def test_canonical_14th_role_skill_doctor(self):
|
||||
"""P0-1: 14th canonical role skill-doctor is registered with correct Russian metadata and aliases."""
|
||||
canonical_roles = RoleRegistry.list_canonical_roles()
|
||||
self.assertEqual(len(canonical_roles), 14)
|
||||
self.assertGreaterEqual(len(canonical_roles), 14)
|
||||
self.assertIn("skill-doctor", canonical_roles)
|
||||
|
||||
doc_role = get_role_definition("skill-doctor")
|
||||
|
|
|
|||
336
tests/test_a52_local_models_supervisor_dual.py
Normal file
336
tests/test_a52_local_models_supervisor_dual.py
Normal file
|
|
@ -0,0 +1,336 @@
|
|||
"""Unit test suite for Task A52: Local Models, Supervisor, and Dual Coder Pipeline."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from antigravity_provider.router.dual_coder_pipeline import (
|
||||
DualCoderPipeline,
|
||||
DualCoderResult,
|
||||
JudgeVerdict,
|
||||
)
|
||||
from antigravity_provider.router.local_supervisor import (
|
||||
ContextExhaustedError,
|
||||
IndivisibleTaskError,
|
||||
LocalSupervisor,
|
||||
ModelMemoryRecord,
|
||||
ServerPropsResult,
|
||||
SupervisorOutcome,
|
||||
TokenCountResult,
|
||||
)
|
||||
from antigravity_provider.router.role_registry import CANONICAL_ROLES, RoleRegistry
|
||||
|
||||
|
||||
# =====================================================================
|
||||
# 1. Role Registry Tests (P0-4)
|
||||
# =====================================================================
|
||||
def test_role_registry_contains_local_supervisor():
|
||||
"""Verify local-supervisor canonical role is properly defined in RoleRegistry."""
|
||||
assert "local-supervisor" in CANONICAL_ROLES
|
||||
role = RoleRegistry.get_role("local-supervisor")
|
||||
assert role is not None
|
||||
assert role.role_id == "local-supervisor"
|
||||
assert role.display_name_ru == "Надзиратель локальных моделей"
|
||||
assert role.tier == "governance"
|
||||
assert "task-splitter" in role.capabilities
|
||||
assert "token-counter" in role.capabilities
|
||||
assert "memory-tracker" in role.capabilities
|
||||
assert role.is_implemented is True
|
||||
|
||||
|
||||
def test_role_registry_local_supervisor_aliases():
|
||||
"""Verify all aliases resolve to local-supervisor."""
|
||||
for alias in ["local-supervisor", "local_supervisor", "надзиратель локальных моделей", "supervisor", "локальный надзиратель"]:
|
||||
assert RoleRegistry.resolve_canonical_role(alias) == "local-supervisor"
|
||||
|
||||
|
||||
# =====================================================================
|
||||
# 2. Local Supervisor Props & Token Counting Tests (P0-5)
|
||||
# =====================================================================
|
||||
def test_supervisor_query_props_success():
|
||||
supervisor = LocalSupervisor(base_url="http://mock-server:8081")
|
||||
fake_props_response = json.dumps({
|
||||
"default_generation_settings": {"n_ctx": 65536},
|
||||
"total_slots": 1,
|
||||
"model_path": "/srv/ai/models/qwen3-coder-30b-a3b/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf",
|
||||
}).encode("utf-8")
|
||||
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.read.return_value = fake_props_response
|
||||
mock_resp.__enter__.return_value = mock_resp
|
||||
|
||||
with patch("urllib.request.urlopen", return_value=mock_resp):
|
||||
res = supervisor.query_server_props()
|
||||
assert res.n_ctx == 65536
|
||||
assert res.total_slots == 1
|
||||
assert "Qwen3-Coder-30B" in res.model_name
|
||||
assert res.is_measured is True
|
||||
|
||||
|
||||
def test_supervisor_query_props_offline_fallback():
|
||||
supervisor = LocalSupervisor(base_url="http://unreachable-host:8081")
|
||||
with patch("urllib.request.urlopen", side_effect=Exception("Connection refused")):
|
||||
res = supervisor.query_server_props()
|
||||
assert res.n_ctx == 65536
|
||||
assert res.is_measured is False
|
||||
|
||||
|
||||
def test_supervisor_count_tokens_exact_via_api():
|
||||
supervisor = LocalSupervisor(base_url="http://mock-server:8081")
|
||||
fake_tok_response = json.dumps({
|
||||
"tokens": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
|
||||
}).encode("utf-8")
|
||||
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.read.return_value = fake_tok_response
|
||||
mock_resp.__enter__.return_value = mock_resp
|
||||
|
||||
with patch("urllib.request.urlopen", return_value=mock_resp):
|
||||
res = supervisor.count_tokens("def add(a, b): return a + b")
|
||||
assert res.tokens_count == 10
|
||||
assert res.is_estimated is False
|
||||
assert res.method == "tokenize_api"
|
||||
|
||||
|
||||
def test_supervisor_count_tokens_fallback_heuristic():
|
||||
supervisor = LocalSupervisor(base_url="http://unreachable-host:8081")
|
||||
with patch("urllib.request.urlopen", side_effect=Exception("Timeout")):
|
||||
sample_text = "A" * 70
|
||||
res = supervisor.count_tokens(sample_text)
|
||||
assert res.is_estimated is True
|
||||
assert res.tokens_count == int(70 / 3.5)
|
||||
assert res.method == "char_heuristic"
|
||||
|
||||
|
||||
# =====================================================================
|
||||
# 3. Semantic Task Splitting Tests (P0-6)
|
||||
# =====================================================================
|
||||
def test_supervisor_split_task_fits_in_single_chunk():
|
||||
supervisor = LocalSupervisor()
|
||||
with patch.object(supervisor, "count_tokens", return_value=TokenCountResult(tokens_count=500, is_estimated=False, method="mock")):
|
||||
chunks = supervisor.split_task_semantically("def small_task(): pass", max_chunk_tokens=2000)
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0] == "def small_task(): pass"
|
||||
|
||||
|
||||
def test_supervisor_split_task_across_files():
|
||||
supervisor = LocalSupervisor()
|
||||
task = """--- file: a.py
|
||||
def func_a():
|
||||
pass
|
||||
|
||||
--- file: b.py
|
||||
def func_b():
|
||||
pass
|
||||
"""
|
||||
def mock_count(text):
|
||||
if "a.py" in text and "b.py" in text:
|
||||
return TokenCountResult(tokens_count=250, is_estimated=False, method="mock")
|
||||
return TokenCountResult(tokens_count=80, is_estimated=False, method="mock")
|
||||
|
||||
with patch.object(supervisor, "count_tokens", side_effect=mock_count):
|
||||
chunks = supervisor.split_task_semantically(task, max_chunk_tokens=150)
|
||||
assert len(chunks) == 2
|
||||
assert "a.py" in chunks[0]
|
||||
assert "b.py" in chunks[1]
|
||||
|
||||
|
||||
def test_supervisor_split_task_across_functions():
|
||||
supervisor = LocalSupervisor()
|
||||
task = """class MyService:
|
||||
def method_one(self):
|
||||
print("1")
|
||||
|
||||
def method_two(self):
|
||||
print("2")
|
||||
"""
|
||||
def mock_count(text):
|
||||
if "method_one" in text and "method_two" in text:
|
||||
return TokenCountResult(tokens_count=200, is_estimated=False, method="mock")
|
||||
return TokenCountResult(tokens_count=40, is_estimated=False, method="mock")
|
||||
|
||||
with patch.object(supervisor, "count_tokens", side_effect=mock_count):
|
||||
chunks = supervisor.split_task_semantically(task, max_chunk_tokens=100)
|
||||
assert len(chunks) >= 2
|
||||
|
||||
|
||||
def test_supervisor_indivisible_task_raises_error():
|
||||
supervisor = LocalSupervisor()
|
||||
indivisible_giant_line = "x = " + ("1+" * 5000)
|
||||
with patch.object(supervisor, "count_tokens", return_value=TokenCountResult(tokens_count=10000, is_estimated=False, method="mock")):
|
||||
with pytest.raises(IndivisibleTaskError):
|
||||
supervisor.split_task_semantically(indivisible_giant_line, max_chunk_tokens=500)
|
||||
|
||||
|
||||
# =====================================================================
|
||||
# 4. Outcome & Reasoning Exhaustion (A39) Tests (P0-7)
|
||||
# =====================================================================
|
||||
def test_supervisor_detect_outcome_success():
|
||||
supervisor = LocalSupervisor()
|
||||
resp = {
|
||||
"choices": [{"message": {"content": "def add(a, b): return a + b"}}],
|
||||
"timings": {"predicted_n": 25},
|
||||
}
|
||||
outcome, desc = supervisor.detect_outcome(resp, elapsed_sec=2.5)
|
||||
assert outcome == SupervisorOutcome.SUCCESS
|
||||
assert "Успешно" in desc
|
||||
|
||||
|
||||
def test_supervisor_detect_outcome_timeout():
|
||||
supervisor = LocalSupervisor()
|
||||
outcome, desc = supervisor.detect_outcome(None, error=TimeoutError("Request timed out"), elapsed_sec=185.0)
|
||||
assert outcome == SupervisorOutcome.TIMEOUT
|
||||
assert "Превышен таймаут" in desc
|
||||
|
||||
|
||||
def test_supervisor_detect_outcome_a39_reasoning_exhaustion():
|
||||
supervisor = LocalSupervisor()
|
||||
# Case A39: content is empty or whitespace, but predicted_n > 50 or reasoning_content exists
|
||||
resp = {
|
||||
"choices": [{
|
||||
"message": {
|
||||
"content": "",
|
||||
"reasoning_content": "Let me think about how to write the adder function... wait..."
|
||||
},
|
||||
"finish_reason": "length"
|
||||
}],
|
||||
"timings": {"predicted_n": 1500},
|
||||
}
|
||||
outcome, desc = supervisor.detect_outcome(resp, elapsed_sec=45.0)
|
||||
assert outcome == SupervisorOutcome.REASONING_EXHAUSTED
|
||||
assert "A39" in desc
|
||||
assert "enable_thinking: false" in desc
|
||||
|
||||
|
||||
# =====================================================================
|
||||
# 5. AI-Memory Integration Tests (P0-8)
|
||||
# =====================================================================
|
||||
def test_supervisor_memory_tracking_roundtrip(tmp_path: Path):
|
||||
mem_file = tmp_path / "local_models_memory.json"
|
||||
supervisor = LocalSupervisor(memory_path=mem_file)
|
||||
|
||||
rec1 = supervisor.record_working_volume(
|
||||
gguf_name="Qwen3-Coder-30B-A3B",
|
||||
prompt_tokens=16000,
|
||||
output_tokens=250,
|
||||
outcome=SupervisorOutcome.SUCCESS,
|
||||
speed_tps=110.5,
|
||||
task_id="T01",
|
||||
)
|
||||
assert rec1.successful_dispatches == 1
|
||||
assert rec1.safe_chunk_tokens == 16000
|
||||
assert rec1.avg_generation_tps > 100.0
|
||||
|
||||
# Fetch from memory
|
||||
fetched = supervisor.get_model_memory("Qwen3-Coder-30B-A3B")
|
||||
assert fetched is not None
|
||||
assert fetched.last_working_context == 16000
|
||||
|
||||
# Record failure -> safe chunk reduced
|
||||
rec2 = supervisor.record_working_volume(
|
||||
gguf_name="Qwen3-Coder-30B-A3B",
|
||||
prompt_tokens=48000,
|
||||
output_tokens=0,
|
||||
outcome=SupervisorOutcome.TIMEOUT,
|
||||
speed_tps=0.0,
|
||||
task_id="T02",
|
||||
)
|
||||
assert rec2.failed_dispatches == 1
|
||||
assert rec2.safe_chunk_tokens < 48000
|
||||
|
||||
|
||||
# =====================================================================
|
||||
# 6. Dual Coder Pipeline Tests (P0-9)
|
||||
# =====================================================================
|
||||
def test_dual_coder_pipeline_success_winner_a():
|
||||
coder_a = MagicMock(return_value={"content": "def solution_a(): return True", "tokens_generated": 40})
|
||||
coder_b = MagicMock(return_value={"content": "def solution_b(): return False", "tokens_generated": 45})
|
||||
judge = MagicMock(return_value={
|
||||
"verdict": "ACCEPTED_A",
|
||||
"commentary": "Solution A is correct and handles edge cases.",
|
||||
"tokens_consumed": 120,
|
||||
})
|
||||
|
||||
pipeline = DualCoderPipeline(coder_a_fn=coder_a, coder_b_fn=coder_b, judge_fn=judge, max_rounds=3, enabled=True)
|
||||
result = pipeline.run_pipeline("Write boolean check", judge_model_name="claude-3-5-sonnet")
|
||||
|
||||
assert result.success is True
|
||||
assert result.final_verdict == JudgeVerdict.ACCEPTED_A
|
||||
assert result.winning_coder == "coder-a"
|
||||
assert result.total_rounds == 1
|
||||
assert result.total_judge_calls == 1
|
||||
assert result.total_judge_tokens == 120
|
||||
assert "solution_a" in result.final_code
|
||||
|
||||
|
||||
def test_dual_coder_pipeline_rework_then_winner_b():
|
||||
call_count = {"judge": 0}
|
||||
|
||||
def mock_coder_a(prompt, feedback):
|
||||
return {"content": f"code_a_round_{call_count['judge']}", "tokens_generated": 30}
|
||||
|
||||
def mock_coder_b(prompt, feedback):
|
||||
return {"content": f"code_b_round_{call_count['judge']}", "tokens_generated": 35}
|
||||
|
||||
def mock_judge(prompt, code_a, code_b, model):
|
||||
call_count["judge"] += 1
|
||||
if call_count["judge"] == 1:
|
||||
return {
|
||||
"verdict": "REWORK_BOTH",
|
||||
"commentary": "Both missing docstrings.",
|
||||
"feedback_for_a": "Add docstring to A",
|
||||
"feedback_for_b": "Add docstring to B",
|
||||
"tokens_consumed": 100,
|
||||
}
|
||||
return {
|
||||
"verdict": "ACCEPTED_B",
|
||||
"commentary": "B added perfect docstrings.",
|
||||
"tokens_consumed": 110,
|
||||
}
|
||||
|
||||
pipeline = DualCoderPipeline(coder_a_fn=mock_coder_a, coder_b_fn=mock_coder_b, judge_fn=mock_judge, max_rounds=3, enabled=True)
|
||||
result = pipeline.run_pipeline("Write function with docs")
|
||||
|
||||
assert result.success is True
|
||||
assert result.final_verdict == JudgeVerdict.ACCEPTED_B
|
||||
assert result.winning_coder == "coder-b"
|
||||
assert result.total_rounds == 2
|
||||
assert result.total_judge_calls == 2
|
||||
assert result.total_judge_tokens == 210
|
||||
|
||||
|
||||
def test_dual_coder_pipeline_stagnation_detection():
|
||||
# Both coders return identical unchanged code on round 2
|
||||
coder_a = MagicMock(return_value={"content": "unchanged_code_a", "tokens_generated": 20})
|
||||
coder_b = MagicMock(return_value={"content": "unchanged_code_b", "tokens_generated": 25})
|
||||
judge = MagicMock(return_value={"verdict": "REWORK_BOTH", "commentary": "Still broken", "tokens_consumed": 50})
|
||||
|
||||
pipeline = DualCoderPipeline(coder_a_fn=coder_a, coder_b_fn=coder_b, judge_fn=judge, max_rounds=3, enabled=True)
|
||||
result = pipeline.run_pipeline("Fix algorithm")
|
||||
|
||||
assert result.success is False
|
||||
assert result.final_verdict == JudgeVerdict.STAGNATION
|
||||
assert "Застревание" in result.failure_reason
|
||||
assert result.total_rounds == 2
|
||||
|
||||
|
||||
def test_dual_coder_pipeline_rounds_exhausted():
|
||||
round_cnt = [0]
|
||||
def mock_a(prompt, fb):
|
||||
round_cnt[0] += 1
|
||||
return {"content": f"varying_code_a_step_{round_cnt[0]}", "tokens_generated": 20}
|
||||
|
||||
def mock_b(prompt, fb):
|
||||
return {"content": f"varying_code_b_step_{round_cnt[0]}", "tokens_generated": 25}
|
||||
|
||||
judge = lambda prompt, ca, cb, m: {"verdict": "REWORK_BOTH", "commentary": "Needs more work", "tokens_consumed": 50}
|
||||
|
||||
pipeline = DualCoderPipeline(coder_a_fn=mock_a, coder_b_fn=mock_b, judge_fn=judge, max_rounds=3, enabled=True)
|
||||
result = pipeline.run_pipeline("Tough problem")
|
||||
|
||||
assert result.success is False
|
||||
assert result.final_verdict == JudgeVerdict.ROUNDS_EXHAUSTED
|
||||
assert result.total_rounds == 3
|
||||
assert result.total_judge_calls == 3
|
||||
assert result.total_judge_tokens == 150
|
||||
|
|
@ -85,7 +85,7 @@ class TestA9ConfigMigration(unittest.TestCase):
|
|||
|
||||
# 4. Verify user profiles are preserved untouched (16 profiles preserved, no fake profiles injected)
|
||||
self.assertEqual(len(migrated_cfg.profiles), 16)
|
||||
self.assertEqual(len(migrated_cfg.roles), 14)
|
||||
self.assertEqual(len(migrated_cfg.roles), 15)
|
||||
|
||||
# 5. Verify existing 10 antigravity profiles are 100% untouched
|
||||
for pid in ["ag-orch-fallback", "ag-w1", "ag-w2", "ag-w3", "ag-w4", "ag-spare-1", "ag-spare-2", "ag-cold-1", "ag-cold-2", "ag-cold-3"]:
|
||||
|
|
|
|||
|
|
@ -250,7 +250,7 @@ class TestLocalLLMConfigAndAutoAssigner:
|
|||
def test_default_config_clean_roles_and_local_registration(self):
|
||||
cfg = get_default_router_config()
|
||||
assert len(cfg.profiles) == 0
|
||||
assert len(cfg.roles) == 14
|
||||
assert len(cfg.roles) == 15
|
||||
|
||||
slot = AutoAssigner.find_free_slot("local")
|
||||
assert slot == "local-1"
|
||||
|
|
@ -282,8 +282,8 @@ class TestLocalLLMConfigAndAutoAssigner:
|
|||
migrated = load_router_config(config_path)
|
||||
# User profile is preserved
|
||||
assert "custom-codex" in migrated.profiles
|
||||
# 14 canonical roles are migrated
|
||||
assert len(migrated.roles) == 14
|
||||
# 15 canonical roles are migrated
|
||||
assert len(migrated.roles) == 15
|
||||
# No dummy local profiles injected
|
||||
assert "local-1" not in migrated.profiles
|
||||
finally:
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ class TestRouterConfig:
|
|||
def test_default_config_is_clean(self):
|
||||
config = get_default_router_config()
|
||||
assert len(config.profiles) == 0
|
||||
assert len(config.roles) == 14
|
||||
assert len(config.roles) == 15
|
||||
assert config.default_role == "manager"
|
||||
assert config.enabled is True
|
||||
|
||||
|
|
|
|||
8
uv.lock
8
uv.lock
|
|
@ -301,7 +301,6 @@ version = "0.1.1"
|
|||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "fastapi" },
|
||||
{ name = "gguf" },
|
||||
{ name = "httpx" },
|
||||
{ name = "psutil" },
|
||||
{ name = "pydantic" },
|
||||
|
|
@ -311,6 +310,9 @@ dependencies = [
|
|||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
benchmarks = [
|
||||
{ name = "gguf" },
|
||||
]
|
||||
dev = [
|
||||
{ name = "anyio" },
|
||||
{ name = "pytest" },
|
||||
|
|
@ -323,7 +325,7 @@ dev = [
|
|||
requires-dist = [
|
||||
{ name = "anyio", marker = "extra == 'dev'", specifier = ">=4.0.0" },
|
||||
{ name = "fastapi", specifier = ">=0.110.0" },
|
||||
{ name = "gguf", specifier = ">=0.19.0" },
|
||||
{ name = "gguf", marker = "extra == 'benchmarks'", specifier = ">=0.19.0" },
|
||||
{ name = "httpx", specifier = ">=0.27.0" },
|
||||
{ name = "psutil", specifier = ">=5.9.0" },
|
||||
{ name = "pydantic", specifier = ">=2.6.0" },
|
||||
|
|
@ -335,7 +337,7 @@ requires-dist = [
|
|||
{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.3.0" },
|
||||
{ name = "uvicorn", specifier = ">=0.28.0" },
|
||||
]
|
||||
provides-extras = ["dev"]
|
||||
provides-extras = ["benchmarks", "dev"]
|
||||
|
||||
[[package]]
|
||||
name = "httpcore"
|
||||
|
|
|
|||
Loading…
Reference in a new issue