Skip to content

Evaluation · The gate every Agent change must pass

Standardized metrics, Pareto analysis, hard gate judgment, and regret—every Agent change must pass Evaluation before commit; otherwise rollback or re-propose.

GATE P0 Python/C++
Core capabilities ↓ Evaluation example Source ↗
Evaluation Gate
# Agent proposes change → evaluation gate judges
$ eval.run(snap_before, snap_after)
Gate: {WNS: -0.042ns, TNS: -2.1ns, HPWL: 1.25e7}
$ eval.judge(gate, proposal)
→ PASS: all hard gates satisfied, on Pareto front
iMapsynthesisiFPfloorplaniPDNpoweriPLplacementiCTSclockiTOoptimizationiRTroutingiSTAtimingAiEDAdesign dataiPCLlayout modeliMapsynthesisiFPfloorplaniPDNpoweriPLplacementiCTSclockiTOoptimizationiRTroutingiSTAtimingAiEDAdesign dataiPCLlayout model

Core capabilities

Metric Schema

Standardized metrics—WNS/TNS/HPWL/density/power/DRC count/congestion—with direction (minimize/maximize), thresholds, and units. Custom extensions supported; all compared in one framework.

Gate

Hard gate judgment—no critical false negatives; constraints must be complete. If an Agent change worsens critical-path WNS beyond threshold or adds DRC violations, gate FAIL and commit rejected.

Pareto

Pareto frontier analysis for multi-objective trade-offs. Locate the proposal on WNS vs HPWL vs density—dominated by existing candidates or extending the frontier.

Regret

Quantify loss from not picking the best candidate. Regret compares the chosen commit to the global best among peers—guides better trade-offs in the next iteration.

Coverage

Track verification coverage: which objects were evaluated, corner cases covered, constraint combos verified—avoid hidden issues on uncovered paths.

Evaluation example

from ieda import AgentClient
from ieda.evaluation import MetricSchema, Gate, ParetoFront

client = AgentClient()

# Define evaluation metrics
metrics = MetricSchema([
    {"name": "wns", "direction": "maximize", "gate": {"min": -0.050}},
    {"name": "hpwl", "direction": "minimize"},
    {"name": "density", "direction": "minimize", "gate": {"max": 0.85}},
    {"name": "drc_count", "direction": "minimize", "gate": {"max": 0}},
])

# Run evaluation on a snapshot pair
result = client.call("eval.run",
    before="snap_7f3a",
    after="snap_8b2c",
    metrics=metrics)

# Gate judgment
judgment = Gate.judge(result)
if judgment.passed:
    print(f"PASS — all hard gates satisfied")
    print(f"Pareto rank: {judgment.pareto_rank}")
    print(f"Regret (vs best): {judgment.regret:.4f}")
else:
    print(f"FAIL — {judgment.failures}")

Metric schema

MetricDefinition {
  name: str                    // Metric name, e.g. "wns", "hpwl", "drc_count"
  direction: "minimize" | "maximize" | "target"
  unit?: str                   // Unit, e.g. "ns", "um", "count"
  gate?: GateRule {            // Hard gate rule
    min?: f64                  // Minimum allowed value
    max?: f64                  // Maximum allowed value
    strict?: bool              // Strict inequality (default false = inclusive)
  }
  weight?: f64                 // Weight in multi-objective aggregation
}

GateJudgment {
  passed: bool                 // Whether gate passed
  metrics: {}MetricValue       // Actual metric values
  pareto_rank: int             // Rank on current Pareto frontier
  regret: f64                  // Regret vs best candidate
  failures: []GateFailure      // Failed gates
  coverage: f64                // Verification coverage (0.0 ~ 1.0)
}

Verification Hub

Full verification loop—from formal verification to silicon comparison.

Toolchain →

Agent Runtime

How Agents run the explore-evaluate-commit loop on the platform.

Platform →

Performance

Track Evaluation compute cost—latency and resources per gate judgment.

Performance →