NashMarkAI Open-Source Evaluation Module
GenAI Evaluation Matrix
A provider-neutral Python framework for testing what generative AI systems actually do across continuing conversations.
GAIEM executes controlled multi-turn benchmarks, preserves raw evidence before interpretation, applies versioned evaluators and produces reproducible reports without requiring another model call.
Purpose
Beyond isolated prompt demonstrations
Static benchmarks can show whether a model answers one question. GAIEM tests whether the model preserves facts, instructions, uncertainty and source boundaries as the conversation develops.
Continuing State
Tests whether the model can reconstruct the accumulated conversation state rather than merely react to the latest turn.
Evidence First
Stores raw provider responses and transcripts before behavioural interpretation or report generation.
Provider Neutral
Uses a common provider contract so local and API-based models can be evaluated under the same benchmark conditions.
Versioned Comparison
Freezes benchmark, evaluator and model configuration identities so results remain traceable and comparable.
Implemented Architecture
Execution and evidence are separated from interpretation
The operational flow prevents a report or evaluator from becoming the source record.
Evaluation Matrix
What GAIEM measures
The v0.1.0 registry separates distinct behaviours instead of collapsing performance into one opaque score.
Context availability is not the same as effective state retention.
GAIEM gives the model the accumulated transcript, then measures whether the model can actually preserve and reconstruct the benchmark-defined state.
Frozen v0.1.0 Benchmark
Four continuing-conversation risk profiles
The baseline is designed around realistic cumulative-state failure, not trivia alone.
Clinical State Retention
Tests whether symptoms, medication, allergy, chronology, worsening conditions and uncertainty remain present across a developing account.
Source Integrity
Tests whether supplied evidence remains separated from unsupported claims and whether citation or conclusion invention is resisted.
Cumulative Factual State
Tests retention of named facts, ownership, dates, responsibilities, corrections and the complete project record.
Instruction Persistence
Tests whether output rules, prohibited content, ordering, scope and superseding corrections survive later turns.
Measured Baseline
Llama 3.2 empirical anchor
The completed local Ollama run produced all 36 expected responses with no provider or evaluator execution errors. The strict cumulative-recall probes showed an omission-dominated failure pattern.
What the result means
The model received the complete transcript but omitted most configured facts when explicitly asked to reconstruct cumulative state.
This is an empirical baseline for one model and configuration. It is not a universal claim about all models or products.
Architecture Boundary
Measured evidence and predicted outcomes remain separate
Operational evaluation system
Benchmark loading, continuing-conversation execution, provider abstraction, raw evidence storage, evaluator dispatch, reporting and the frozen Llama 3.2 baseline.
Predictive Model Outcome Forecasting Layer
A hierarchical probabilistic layer intended to estimate likely model outcomes, uncertainty and the most informative next model test after a broader empirical calibration set exists.
Open Source
Reproducible, auditable and self-contained
The repository contains benchmark definitions, provider interfaces, evaluators, evidence-preserving run logic, reporting components and the frozen empirical baseline.
- Local model execution through Ollama
- External provider integration through adapters
- Append-only run directories and preserved evidence
- Report regeneration without contacting the model
- GPL-3.0-or-later source licence
git clone https://github.com/NashMarkAI/GAIEM
cd GAIEM
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python runner.py --validate-only
python runner.pyGAIEM v0.1.0
Evaluate behaviour. Preserve the evidence.
Open the source, inspect the benchmark and run the same continuing-conversation conditions against a local or configured model endpoint.