Context
Researcher in Microsoft 365 Copilot produces reports from a user's request, grounded in supporting sources and shown with citations. Since 2025 a large part of my work has been on the quality of what it produces: defining what a good response looks like, measuring it, and finding out why responses fall short.
I did not develop the underlying AI models. My work sits around them, in evaluation, analysis and the surrounding system. It is collaborative work with engineering, research and product colleagues; where I describe my own contribution, I say so.
Problem
An AI-generated answer can look correct and still fall short. It can:
- repeat the same point several times in different words, so it reads as thorough but says little;
- rely on sources that are out of date for the timeframe the user asked about;
- lose context the user provided, such as a meeting description or a clarification, somewhere between retrieval and the final answer.
None of these show up if the only question is "does this answer seem right?". Each needs its own definition of quality and its own way of measuring it, and evaluation results vary even when nothing has changed.
My contribution
- Designed and implemented evaluation metrics, including claim density and temporal relevance.
- Ran offline experiments and baseline evaluation runs, and analysed the results.
- Built a reusable offline workflow for testing changes to generated output.
- Investigated retrieval and context-preservation failures, and implemented a change to how context is passed between stages.
- Helped teammates adopt the offline evaluation workflows through walkthroughs and shared documentation.
Experiment direction, product decisions and system design were shared with the wider team.
Technical approach
Claim density
A metric that measures how much distinct information a generated report contains. It breaks a response into individual claims, identifies claims that repeat information already stated, and quantifies the redundancy. This made verbosity something the team could track objectively instead of judging it by eye. Colleagues later reused the implementation as a building block for related quality metrics.
Temporal relevance
A metric that checks whether a response uses evidence from the right time. For time-sensitive requests, such as questions about "this quarter", it extends citation evaluation to the timeframe the user asked for by examining the dates and timestamps in the supporting citations. I integrated it into the offline evaluation pipeline. Its purpose is to catch responses that look relevant but rely on outdated evidence.
Baselines before conclusions
To separate real changes from noise, I led regular baseline runs: the same evaluation repeated over time, with no change to the system, to measure natural variability. With a colleague, I added the results to a dashboard so the team could watch trends.
Offline experimentation workflow
A reusable workflow that applies transformations to generated outputs after generation, such as removing redundancy or applying a template, and then evaluates the result. Ideas could be tried offline before anyone changed the production system. I documented it, and teammates used it to prototype their own improvements.
Context preservation
AI workflows often run in stages, and information passed from one stage to the next can be incomplete. I implemented a change so that all relevant properties were passed and preserved across stages rather than a partial set; missing properties can lead to missing signals, incomplete results and redundant tool calls. Separately, with colleagues, I investigated why meeting-related content surfaced inconsistently in generated summaries and documented the failure modes: missing meeting descriptions, lost clarification context, and links to files that did not surface reliably.
Challenges and trade-offs
An experiment that changed direction. I drove a multi-phase experiment to make generated reports more concise without losing key information. Prompt changes moved the metrics less than expected. Removing duplicate content after generation looked promising offline, and I built a proof of concept with colleagues. A design review with the team that owned the wider system then showed that a post-processing step did not fit its architecture, so that approach did not ship. The experiment still produced a reusable metric, a tested approach and a clear understanding of the constraints.
Noise versus signal. Baseline runs surfaced some shifts that we could not yet explain. They remain open questions rather than findings.
Investigations that stay open. The meeting-context investigation identified the failure modes, but the underlying issue was still open when I last recorded it.
Outcome
- Completed: the claim density and temporal relevance metrics, integrated into offline evaluation; the reusable offline workflow; regular baseline evaluation runs.
- Concluded without shipping: the post-processing approach to verbosity, stopped after a design review.
- Implemented, rollout pending at last record: the context-preservation change, so no effect is claimed here.
- Open: the meeting-context retrieval issue, and unexplained shifts in some baseline metrics.
Reflection
- Define quality before improving it. "Better" needs a definition that can be measured, and different failures need different measurements.
- Know your noise. Without a baseline of natural variability, it is easy to mistake noise for an improvement.
- Investigations are valuable even when the issue stays open. A documented failure mode gives the next person a head start.
- Fit matters as much as results. A promising offline result still has to work with the architecture of the system it is meant for.
Questions I would explore next: how closely claim-level and time-based metrics agree with human judgements of quality, and which context-preservation failures matter most to users. Some of the same questions about memory, retrieval and evaluation shape my independent project, Memory Lab.