Benchmark agent memory and RAG systems with MemoryBench
Use MemoryBench to run repeatable conversational memory and RAG benchmarks across providers, datasets, judge models, checkpoints, and structured reports.
npx skills add agentskillexchange/skills --skill benchmark-agent-memory-and-rag-systems-with-memorybench
Use MemoryBench when an operator needs evidence about whether an agent memory or RAG provider is retrieving and answering correctly before changing memory infrastructure. The agent/operator installs the benchmark runner, configures one or more providers and judge models, runs datasets such as LoCoMo, LongMemEval, or ConvoMem through ingest, index, search, answer, and evaluate stages, then compares reports and failures. Invoke it instead of manually spot-checking memory answers when provider swaps, prompt changes, retrieval settings, or memory backends need repeatable evaluation. The scope boundary is benchmarking and diagnosing memory/RAG behavior with explicit datasets and metrics; it is not a generic memory product, vector database, or hosted retrieval service listing.
What this skill actually does
Inputs and prerequisites: Bun, MemoryBench repository, at least one memory/RAG provider API key, at least one judge model API key, benchmark datasets.
Setup notes: Clone https://github.com/supermemoryai/memorybench, run bun install, copy .env.example to .env.local, add provider and judge API keys, then run bun run src/index.ts run -p -b or use compare/status/show-failures for evaluation workflows.
Source and verification boundary: use https://supermemory.ai/docs/memorybench/overview as the canonical reference before running the workflow; keep commands, API calls, CLI usage, and generated outputs reviewable against that upstream source.
Framework fit: publish this as a Multi-Framework workflow only when the operator can invoke the documented toolchain directly, rather than treating the upstream project as a generic product listing.