Evaluating Risk in Open-Weight Models

A model only produces text. It can answer a question, write code, or propose a tool call, but it cannot run any of them. The agent harness is the software that connects those outputs to tools, executes operations, and feeds results back to the model. When that loop continues without approval at every step, it enables autonomous action. This post is about the model itself. The harness and the runtime are what execute, and both are out of scope in this blog.  Later posts in the series will cover them.

Consider the same model in two deployments. One drafts a financial report for an analyst to review. The second model has tools to retrieve files, run calculations, and email the report. Both need guardrails for inaccurate claims and sensitive-data disclosure. The second also needs controls over which files it can access, what code it can execute, and where it can send information.

Open weight models let teams run, evaluate, and modify model weights subject to the license.  Open-weight models let your data stay under your control because you download and run the model on your own local hardware or private cloud infrastructure rather than sending your data through a third-party vendor’s API. You can fine-tune models or train new ones from scratch that are specific for your business use case, and you can gain more detailed insight into reasoning traces and other telemetry that provide useful information on how and why your users have engaged with your model. 

When you self-host a model, many of the responsibilities around safety, security, and trustworthiness that could be at least in part delegated to a model provider become your responsibility.

A model on its own is a component. What your users touch is a system: the model plus a harness, guardrails, retrieval, tool access, sandboxing, and runtime enforcement. You cannot judge the safety of the system by testing the component alone.  Many of the controls that make a deployment safe sit outside the weights entirely. 

NVIDIA and Microsoft’s Risk Management for Open-Weight Models in Enterprise provides a starting point: a vocabulary for inventorying risks, evaluations that measure these risks, and guidance for acting on the results. This blog covers the model component, because you have to be able to characterize the engine before you can reason about the vehicle. The series that follows will address the harness, the runtime, and the full agentic system.

This post introduces the guide and evaluation resources enterprises can use as they develop their own approach.

Which risks should you test?

Start with what your deployment does, the data it handles, and who uses its outputs. Use these five categories to identify the potential risks.

Table 1. Initial risk categories and illustrative questions for adapting them to an enterprise deployment.

Within each category, more specific risk types help teams assign owners and map evaluations and mitigations. For example, the Content category includes groundedness, whether an output is supported by the information provided to the model.

Which of these types matter, and how much, depends on your users, workflow goals, languages, data, and deployment architecture.  For example, a document summarization service might need groundedness assessments for unsupported statements and PII disclosure. A customer-facing assistant may need tests for harmful content, deception, and attempts to bypass safeguards.

The taxonomy is a starting point, while your specific use cases and their associated risks define how to apply it. You can use our evaluations as a starting point and should modify them to your specific requirements. Expect for evaluations to change, be challenged, or even require customization as new evidence, capabilities, and deployment patterns emerge.

How to test your deployment

The paper draws on ISO/IEC 23894 guidance on AI risk management and is aligned with the NIST AI Risk Management Framework. Apply the process in five steps.

1. Define scope, context, and criteria. Identify the model version, modifications, intended users, permitted uses, data, and connected services. Mark which of the five categories apply. If the service never handles personal data, much of Data Protection is out of scope. If it makes no high-impact decisions about people, parts of Bias matter less. Establish the criteria your organization will use to judge the risks that remain, before you interpret any evaluation results.
2. Identify and prioritize risks. For each risk type that applies, describe a concrete risk scenario, who could be affected, the potential consequences, and the evidence you would need to assess it. Add novel risks specific to your deployment where the taxonomy leaves gaps. Weigh likelihood, existing controls, and uncertainty, then assign owners who can act on the findings.
3. Evaluate the model and deployed system. Assess the model independently, as well as the configuration users will encounter, including system instructions, filters, and other controls. Combine relevant benchmarks for the risks you identified in the previous steps with representative scenarios and adversarial testing that reflect those risks.
4. Mitigate and evaluate again. Apply controls, such as NeMo Guardrails or comparable open frameworks, where they can address a given risk. Options include model changes, input and output filtering, access restrictions, and human review. Repeat evaluations to measure whether the controls work and whether they interfere with legitimate use.
5. Monitor and reassess. Use operational evidence, incidents, user feedback, and newly discovered vulnerabilities to update the assessment. Reevaluate the deployment after material changes to the model, configuration, controls, or user population.

Evaluation results show how a model behaves under the conditions you tested, and nothing more. For example, a 4% performance shortfall (96%) on a benchmark does not tell you how often that shortfall reaches a user, or what it costs when it does. Teams need to interpret results in the context of their own deployment and record where coverage is incomplete.

Making results comparable over time requires records. Store model version, prompts, datasets, scoring methods, and configuration with each result. That record makes it possible to compare changes and distinguish model behavior from the effects of surrounding controls.

Example: evaluating a financial-summary model

Consider a company plans to use an open-weight model to summarize financial documents for analysts, including analysis drafting, summarization, classification, model building, and question answering. The service accepts prompts and returns text or CSVs. The model cannot retrieve files, call tools, execute code or send reports. 

1. Define the use case. Record the model version and configuration. Analysts will review summaries before using them in client reports. Set acceptance criteria before testing—for example, a fabricated financial figure or disclosure of confidential information should block deployment until addressed. 
2. Describe what could go wrong. The model could invent a number, omit a qualification that changes a conclusion, or reproduce sensitive information that should be excluded from the summary. Assign an owner to investigate each type of finding. 
3. Run relevant tests. Test the model on representative financial documents. Compare summaries with the sources, checking figures, dates, and qualifications. Include cases designed to elicit sensitive information or override the summarization instructions. Record the prompts, expected behavior, scoring method, and results.
   1. Suppose 4 of 100 summaries contain an unsupported claim. Inspect each case to understand its consequences. That result describes this test set; it does not establish how often errors will occur in production.
4. Apply controls and test again. If figures are unsupported, revise the instructions to require source references and check those references against the supplied documents. If sensitive information appears, remove unnecessary sensitive inputs and test appropriate output filters. Repeat the original tests and add fresh cases. Check whether accuracy improves without losing information analysts need. If the results still fail the acceptance criteria, try another model or postpone deployment.
5. Reassess after changes. Repeat relevant tests when the model, prompts, or document types change. Use errors reported by analysts to expand the test set. Keep a record of unresolved problems and who is responsible for deciding whether the model remains suitable for use.

What to change when a test finds a problem

When an evaluation finds a problem, the controls available to you fall into a few types, and which one fits depends on where the problem originates.

Change the input. System instructions, retrieval scoping, and prompt templates narrow what the model is asked to do. This is the cheapest control and the fastest to iterate on. Prompt-based controls are soft by nature, so treat them as a first reduction rather than a fix.

Filter the output. Output controls do two different jobs. Content classifiers score free text against risk categories. Schema validation checks that structured output has the shape you expect, which is what catches a malformed CSV or an out-of-range value before it reaches a downstream system. Measure both what gets through and what legitimate traffic gets blocked: a filter that stops 99% of risky responses and a third of valid requests has moved the problem, not solved it.

Restrict who can reach it. Authentication, rate limits, role-based access, and separating internal from external users change exposure rather than behavior. A behavior that is unacceptable for external customers may be tolerable for a small internal team that knows the system’s limits.

Review the output before it is used. Human review of generated content on high-consequence paths catches a bad output before it is acted on. It costs latency and headcount, so apply it where the consequence justifies the cost.

Change the model. Selecting a different model or checkpoint addresses the behavior at its source, and choosing between candidates on evaluation evidence is exactly what this guidance is for. Post-training against a specific weakness is a further step, and the recipes for it come in a later paper.

Decline the deployment. If no combination of controls brings residual risk under the criteria you set, the answer is that this model does not go into this use case. Recording that decision and the evidence behind it is as much a risk-management output as a passing result.

After controls are added, evaluate again. Two questions matter: whether the control worked, and whether it broke something that was working.

Choosing evaluations: BBQ and garak 

The paper’s appendix maps 14 evaluations to selected risk types across all five categories: two for Cybersecurity, one for Data Protection, nine for Content, one for Bias, and one for Human-AI Interaction. The weighting is deliberate rather than even, because Content covers the widest span of risk types and has the most mature public benchmarks. Start from the risk you care about and take the evaluation it points to. The two examples below show how to read what an evaluation gives you, and what it does not.

Garak: probing for vulnerabilities

NVIDIA’s open-source Generative AI Red-teaming and Assessment Kit (garak) probes for vulnerabilities including risky content generation, jailbreaks, prompt injection, and data leakage.

Choose probes relevant to the risk types you prioritized, inspect flagged responses, and confirm whether each finding represents a material risk in your deployment.

Detectors are heuristics, so read findings in both directions.  Flagged responses will include false positives, and a probe that does not fire is not evidence the behavior is absent. Results also describe the endpoint you scanned rather than the model in isolation, so run garak against the configuration your users will encounter.

Record the garak, probe, and detector versions alongside the model configuration. See the garak FAQ for more.

BBQ: testing social bias

The Bias Benchmark for Question Answering (BBQ) tests social biases in question-answering tasks. It pairs ambiguous contexts, which withhold the information needed to answer, with disambiguated contexts that supply it.  Both reveal whether a model falls back on stereotypes when evidence is missing and whether it answers correctly once evidence is present.

BBQ can inform assessment within the Bias category and Content category for groundedness:  the ambiguous conditions test whether a model declines to assert what its context does not support.

BBQ focuses on United States English-speaking contexts. Supplement it with scenarios relevant to your users, languages, and workflows. NVIDIA’s internal adaptation supports OpenAI-compatible endpoints, automates metric review, and extends the method to audio inputs. 

Start with one deployment

Use the evaluation mapping in Risk Management for Open-Weight Models in Enterprise to choose tests for one or more deployments your organization knows well. 

Record the problems you find, the controls you apply, the retest results, and any remaining gaps. Assign an owner to decide whether the evidence supports deployment and when to reassess. 

For an agentic deployment, also test what the harness permits the agent to do and whether the surrounding controls enforce those limits.  

The joint GitHub repository will provide evaluation resources and updates.

Risk Management for Open-Weight Models in Enterprise was developed jointly by Microsoft and NVIDIA, with Andrew Berkley and April Chen of Microsoft leading the development and editing of the paper.