Unit Testing LLMs: Evaluating LLM Quality in .NET with `Microsoft.Extensions.AI.Evaluation`
How do we unit test an LLM that never gives the same answer twice? Microsoft.Extensions.AI.Evaluation makes it easy.
This is Part 4 of our Microsoft.Extensions.AI series. In Part 1 we built a chat app with IChatClient — but how do we know its answers are any good? This week, we'll write a unit test that grades our LLM!
You're welcome to skip the definitions and jump directly to the code below.
Why Can't We Just Write Normal Unit Tests?
Assert.AreEqual is useless for LLMs because the exact wording changes on every run. Instead, we grade the meaning.
Microsoft.Extensions.AI.Evaluation uses a second LLM to score the first LLM's answer against our criteria. It has ready-made evaluators that each return a score from 1 (bad) to 5 (great).
What is EquivalenceEvaluator?
EquivalenceEvaluator answers one question: "Does the LLM's response mean the same thing as my expected answer?"
We provide the expected answer (the "ground truth") via EquivalenceEvaluatorContext, and the judge compares the two on that 1-5 scale. Score a 4 or 5, and our test passes!
Code
Now that we have those pesky definitions out of the way, let's write some code!
You can find the completed code sample here.
Prerequisites
If you'd like to follow along, here are the steps.
We'll use Ollama as both the LLM under test AND the judge, so this entire test suite runs completely free!
- Download + install Ollama and pull a model:
ollama pull qwen3.5 - Create a File -> New NUnit Test Project in Visual Studio
- Add the
Microsoft.Extensions.AI,Microsoft.Extensions.AI.Evaluation,Microsoft.Extensions.AI.Evaluation.QualityandOllamaSharpNuGet Packages to the test project.
If you're unfamiliar with adding NuGet Packages, you can find more information here.
Now we can finally write some code!
Unit Test
using Microsoft.Extensions.AI;
using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.Quality;
using OllamaSharp;
namespace AiEvaluation.UnitTests;
public class ChatClientTests
{
[Test]
public async Task LlmResponse_MatchesExpectedAnswer()
{
// Arrange
IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "qwen3.5");
var equivalenceEvaluator = new EquivalenceEvaluator();
var equivalenceContext = new EquivalenceEvaluatorContext("The sky is blue");
var question = new ChatMessage(ChatRole.User, "What color is the sky on a clear day? Answer in one short sentence.");
// Act
ChatResponse response = await client.GetResponseAsync([question]);
EvaluationResult result = await equivalenceEvaluator.EvaluateAsync(
[question],
response,
new ChatConfiguration(client),
[equivalenceContext]);
NumericMetric equivalenceMetric = result.Get<NumericMetric>(EquivalenceEvaluator.EquivalenceMetricName);
Assert.That(equivalenceMetric.Value, Is.GreaterThanOrEqualTo(4)); // 1 = completely different, 5 = fully equivalent
}
}
Now let's click Run and see what gets printed to the console:
Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1
It works! The LLM answered in its own words, the judge compared the meaning to our ground truth, and the score cleared our threshold. No brittle string comparisons required! 🎉
More Evaluators
EquivalenceEvaluator is just one member of the family. Microsoft.Extensions.AI.Evaluation.Quality also ships the following evaluators:
RelevanceEvaluatorCoherenceEvaluatorFluencyEvaluatorCompletenessEvaluatorGroundednessEvaluatorTaskAdherenceEvaluatorToolCallAccuracyEvaluator
All use the same EvaluateAsync pattern above. Learn one, and you've learned them all 💯
Conclusion
As .NET developers, we can finally bring our unit testing discipline to AI features, thanks to Microsoft.Extensions.AI.Evaluation.
And we've only scratched the surface! The Microsoft.Extensions.AI.Evaluation.Reporting generates reports that you can use to augmentadd to your CI pipeline.
You can find the completed code sample here: https://github.com/Dometrain/from-zero-to-hero-microsoft-extensions-ai
Next week, we'll check whether our AI's answers are safe to ship using Microsoft.Extensions.AI.Evaluation.Safety. Make sure to subscribe to this blog so you don't miss it! 👇
To learn even more about Microsoft Extensions AI, check out my course on Dometrain:
Brandon Minnick