ASE Weekly Changelog: Evals, Private RAG, and Code Sandboxes

This week’s practical thread is evaluation before trust, private knowledge before broad retrieval, and sandboxed execution before letting generated code touch real systems. The three routes fit together because they answer a common operator question: what evidence should exist before an agent’s output becomes part of normal work?

In Short

The current ASE queue is pointing operators toward three connected habits. Use eval workflows when agent behavior needs a repeatable pass/fail decision. Use private RAG workspaces when the agent needs internal knowledge but should not roam across every folder or document system. Use sandboxed code execution when the agent needs to run snippets, tests, or examples without inheriting access to the host machine.

This is not a popularity roundup. It is a practical routing note for teams that already know agents can help, but need better boundaries before they widen usage. The safest version of the stack keeps evaluation criteria, source access, and execution limits visible enough for a human reviewer to challenge.

Who this is for

This changelog is for platform teams, developer productivity leads, AI enablement teams, and small engineering groups that are moving from experiments into repeatable agent workflows. It is especially useful if the team has already shipped a few agent-assisted tasks and is now asking how to make the next round more reviewable.

It also fits teams with mixed risk. A documentation assistant, a code helper, and an internal research agent may all live in the same workspace, but they should not share the same acceptance gate. Evals, private retrieval, and sandboxes let operators split those risks instead of approving the agent as one vague system.

Starter workflow

Start with the behavior you need to trust. If the agent drafts release notes, classifies support tickets, proposes code changes, or answers from internal documents, write down the decision that matters. Is the output factually grounded? Does it follow the policy? Does it preserve tone? Does it avoid unsafe tool calls? Those questions become the eval criteria.

Next, decide what context the agent is allowed to use. Private RAG should not mean dumping every internal document into a shared index. Build a narrow workspace around the task: source folders, owners, retention, refresh cadence, and a few retrieval examples that a reviewer can inspect. The official ASE blog source for this queue is the blog itself, so the claims here stay at the operational-pattern level rather than naming vendor metrics or stale adoption numbers.

Then isolate execution. If the agent needs to run generated code, use a sandbox boundary before local shells, production hosts, or customer-connected environments enter the picture. Keep the submitted code, input, output, timeout, and error state as part of the review packet. A sandbox result is not automatic approval, but it is much better evidence than a model saying the code should work.

Finally, connect the three records. The eval says whether the behavior met the standard. The RAG workspace explains which sources were available. The sandbox shows what actually ran. Together, they give reviewers a compact trail that can be reused next time instead of reconstructed from chat history.

Recommended ASE skills

Build and optimize agent eval workflows with Kiln fits the first gate: turning examples, rubrics, and model behavior into a repeatable evaluation process. Use it when the team needs to compare prompt variants, catch regressions, or decide whether an agent workflow is ready for wider use.

Run private RAG and MCP skill workspaces with Yuxi fits the context boundary. Use it when agents need project knowledge, but the team still wants local control over sources, workspaces, and MCP-driven access patterns.

Run sandboxed multi-language code execution with Judge0 fits the execution boundary. Use it when code needs to be run as evidence, not as a leap of faith on the same machine where sensitive work happens.

Run repeatable model and agent eval suites with Inspect AI fits teams that need scoring traces, task definitions, and repeatable evaluation runs alongside their agent changes.

Give MCP agents disposable code execution sandboxes with OpenSandbox MCP fits workflows where agents need temporary execution environments through MCP instead of direct local shell access.

Those five links are intentionally narrow. Add more tools only after you can explain which missing control they cover: better scoring, tighter source access, stronger runtime isolation, or clearer human approval.

What to watch

Watch for eval theater. A rubric that never blocks a release is documentation, not a gate. Keep the failing examples and borderline cases because they teach the team where the agent is still brittle.

Watch for retrieval sprawl. Private RAG can become risky when old documents, copied exports, or ambiguous permissions stay indexed after the task changes. Tie each workspace to an owner and a cleanup rule.

Watch for sandbox overconfidence. A passing sandbox run proves only that the submitted code ran under those constraints. It does not prove the code is secure, maintainable, or appropriate for production. Keep the sandbox as one check inside the larger review path.

FAQ

Is this a full agent platform stack? No. It is a practical review chain for teams that need evidence around behavior, context, and execution. Identity, deployment, monitoring, and approval workflows still matter.

Should evals come before retrieval work? Usually yes. If you do not know what good behavior looks like, adding more context can hide the problem. Define the success criteria first, then give the agent the smallest useful source set.

When is a code sandbox enough? A sandbox is enough for quick execution evidence, examples, and low-risk checks. It is not enough for production approval when the code touches secrets, infrastructure, customer data, or persistent systems.

How should a small team start? Pick one workflow, one source set, and one sandboxed execution path. Keep the evidence packet short: eval result, allowed sources, executed code, output, and the human decision.