Scorra
LLM evaluation platform that helps teams score, compare, and rank model responses against curated datasets — with multi-evaluator workflows, inter-rater agreement analytics, AI-assisted judging, and portable exports.
- 3 Workflows Single, Pairwise & Ranking
- 4 Providers OpenAI, Anthropic, Groq & Gemini
- Fleiss κ Inter-rater agreement
- 4 Roles Org-scoped RBAC
The Problem
Evaluating LLM outputs at scale is messy. Teams end up with prompts in spreadsheets, scores spread across Notion docs, and no way to know how much two evaluators actually agree. Without a structured system, it is impossible to tell whether a model is genuinely better or whether the evaluators just happened to be in a good mood. Inter-rater agreement, weighted scoring criteria, and reproducible export formats all get lost in the noise.
What Was Built
Scorra is a pnpm-workspaces monorepo built on Turborepo. The NestJS API handles three evaluation workflows — single-response scoring, pairwise A/B comparison, and multi-response ranking — backed by PostgreSQL via Prisma, Redis-based Bull job queues, and Passport/JWT auth with org-scoped roles. An org admin creates datasets (CSV, JSON, JSONL uploads), defines scoring criteria with per-dimension weights and ranges, spins up evaluation tasks, and invites evaluators. Each evaluator works through an independent queue so responses can be reviewed multiple times without overlap. The Next.js 15 frontend (App Router, TanStack Query, Zustand, React Hook Form, Zod, Tailwind CSS 4) is role-aware — admins see org-wide dashboards and analytics, evaluators see only their task queue. An AI judge powered by OpenAI, Anthropic, Groq, or Gemini can auto-evaluate responses or pre-fill scores for evaluator calibration, with full deterministic fallbacks when no API key is configured. Results export as JSONL, CSV, or JSON with optional filters.
Highlights
- Three evaluation workflows: SINGLE (weighted scoring 1–10), PAIRWISE (A/B winner), RANKING (ordered responses)
- Multi-evaluator queues with independent assignment so the same item can be reviewed by multiple evaluators without coordination overhead
- Inter-rater agreement analytics including overall agreement rate and Fleiss' kappa across a task
- AI judge with provider-agnostic interface (OpenAI, Anthropic, Groq, Gemini) and confidence-scored fallbacks when no key is set
- Dataset versioning and cloning — CSV, JSON, JSONL upload with DatasetRow and ModelResponse tracking
- Role system (SUPER_ADMIN → ORG_ADMIN → EVALUATOR → VIEWER) enforced server-side via JWT claims and NestJS RolesGuard
- Bull job queue with exponential backoff, retry logic, and configurable cleanup for async export and AI evaluation jobs
- Export engine producing JSONL, CSV, and JSON with per-task filters
Engineering Decisions
The trickiest design problem was the multi-evaluator queue: each evaluator needs an independent, non-overlapping sequence of items so the same response gets reviewed by all assigned evaluators without any one person seeing the same item twice. This required tracking per-evaluator completion state separately from task-level progress rather than using a shared cursor. The AI judge layer was designed to be provider-agnostic — a single interface dispatches to OpenAI, Anthropic, Groq, or Gemini — while still returning a normalized confidence score so the frontend can signal to evaluators how much to trust a suggestion. When no key is configured, all AI paths degrade to deterministic heuristics with confidence set to zero, keeping the platform fully functional offline.
Architecture
- Next.js 15 frontendApp Router, TanStack Query & Zustand
- NestJS APIAuth, datasets, evaluations & exports
- Bull + RedisAI judging & async export jobs
- PostgreSQL + PrismaRelational data & migrations
In Development
Scout
AI-native startup job-search workspace. Save startups and jobs from anywhere on the web and Scout researches each company with AI, matches open roles against your CV with semantic search, and carries you from application through outreach in one place.
Wildcard
A real-time browser-based multiplayer card game for 2–10 players, built around color/number matching, action cards, and wild cards. Create or join a table with a room code — no installs, no accounts required.
In Development
AccountOS
Programmable virtual account infrastructure built on Nomba, enabling rule-driven automation, immutable ledgers, treasury management, and auditability for fintech products.