AI Triage Case Study: What a Real Fleet Review Needs

An AI triage case study is worthless if it hides the raw queue, human corrections, and missed evidence. This article does not claim an Artemes customer result. It gives you a reproducible fleet exercise and the proof a real deployment should produce.

The problem is not getting a model to rank findings. The problem is proving why one item moved, which current facts support the move, and what happens when the evidence is incomplete. A polished priority list without that record is faster opinion, not better triage.

This worked case uses labeled assumptions so you can replace them with your own counts. Its threat inputs come from public sources. Its endpoint queries use official table definitions. No customer outcome is implied.

What should an AI triage case study prove?

It should prove that the workflow improves a decision, not merely its presentation. For every case, show the original finding, source facts, target identity, observation time, local state, missing evidence, model claims, applied policy, reviewer decision, action owner, and final verification.

Report the denominator. Ten successful examples mean little if the system processed 10,000 cases and people quietly repaired the rest. Show the full sample, exclusions, invalid records, disagreements, false escalations, misses, and cases that stayed unknown.

Keep the model outside the grading role. An independent query, source record, reviewer, or replay test should determine whether the route was correct. The framework for validating AI findings gives each claim an observed, confirmed, actionable, or closed state.

What fleet and queue does the worked case use?

Assume a mixed Windows and Linux fleet of 500 monitored assets and a weekly import of 1,200 vulnerability records. These are exercise inputs, not observed Artemes results. The records include duplicates, missing asset IDs, stale inventory, several software versions, and findings with no current runtime evidence.

The operating goal is narrow: prepare an evidence bound recommendation for each valid finding. The allowed routes are urgent repair, planned repair, collect more evidence, apply a temporary control, or send to closure review. The system may not close a finding or change an endpoint.

A case passes the quality gate only with correct asset joins, cited public facts, current observations, explicit unknowns, and reviewer agreement on the route. A case fails if it invents a package, assumes reachability, omits a conflicting control, or cites a source that does not support the claim.

How does the workflow clean the queue first?

Start with identity. Normalize the vulnerability ID, asset ID, source ID, detected artifact, version, detection time, and collector time. Reject records that cannot name an asset or affected component. Keep them in an input error queue rather than asking AI to guess the missing join.

Resolve duplicates to one case key: vulnerability, asset, artifact, and source scope. Preserve every original record as evidence, but create one decision. If three scanners report the same package on the same server, the business needs one owned repair path, not three tickets.

Keep the operating sequence visible. The older vulnerability triage workflow shows how qualification, evidence, ownership, and verification fit together before AI enters the process.

In this exercise, suppose 1,200 imported records collapse to 900 unique case keys and 60 more fail identity checks. The working denominator is 840 valid cases. Those numbers are transparent assumptions. Your run should print actual counts and the reason for every exclusion.

Which public threat facts belong in each case?

Attach the vendor advisory, affected version range, fixed version, CVSS vector where available, CISA Known Exploited Vulnerabilities state, EPSS probability, source date, and retrieval time. Keep raw values separate from policy. A KEV entry proves known exploitation in the wild. It does not prove the affected build exists on this endpoint.

FIRST explains that its current EPSS training window contains more than 1.6 million recorded daily exploitation events. It also warns that a model predicting no exploitation for every vulnerability can show about 97 to 98 percent accuracy because the outcome is rare. The official EPSS method is a useful reminder: rank models by coverage, effort, and efficiency, not a comfortable accuracy number.

Store the probability as a fact with a date. Do not turn it into a verdict. A low probability item can still deserve urgent work on a critical exposed asset, while a high probability item may need evidence collection if the package match is uncertain.

How does current endpoint evidence change the route?

Ask only questions that can change the decision. Is the package present? Is the version affected? Is the related process running? Is it listening? Which identity runs it? Does an observed control block the claimed path? Save the host identifier and collection time with every answer.

On Windows, a read only software check can use the programs table:

A second query can join a current listener to its process:

The column names and platform scope are defined in the official osquery table specifications. The programs table represents Windows Installer products and may contain blank fields, so a missing row is not automatic proof that software is absent. The listener query also needs adequate permissions and a fresh collection time. Preserve those limits as missing evidence.

What does one complete triage record look like?

Consider case 0042 in the exercise. The source finding claims that an affected package version is installed on asset WKS 117. The public record supplies the affected range and fix. Current inventory confirms the named version. No related process or listener appears in the collection. Network policy for the asset group remains unavailable.

The correct route is not closed and not automatically urgent. Record package presence as observed, runtime as not observed at the collection time, and network path as unknown. Recommend a fresh process check during the normal workload window and collection of the applicable network policy. Assign an owner and due date based on the external exploitation signal and business role.

Model output can draft that reasoning. Deterministic checks should verify that every evidence ID exists, every claim names its time, the affected range was compared correctly, and the chosen route allows unknown network state. A reviewer then confirms or changes the route.

How much analyst work can this workflow remove?

Use the 840 valid cases from the exercise. Suppose identity and duplicate processing removes four manual minutes per case. That is 3,360 minutes, or 56 hours. If evidence assembly removes another six minutes but review adds three, the net gain is three minutes per case, or 42 hours. Total gross capacity is 98 hours for the run.

Now subtract upkeep and correction. If analysts spend 18 hours fixing evidence joins and engineers spend 12 hours maintaining the workflow, net capacity becomes 68 hours. Again, these are exercise inputs. The case study is credible only when your system emits the actual minutes, corrections, and exclusions.

Capacity is not the only outcome. Measure accepted routes, reviewer changes, engineering acceptance, time to owner, reopened cases, and confirmed misses. The AI vulnerability prioritization guide shows how to keep evidence and policy visible instead of hiding them in a score.

What recent threat data changes the case?

Verizon published its 2026 DBIR on May 19 after analyzing more than 31,000 incidents and more than 22,000 confirmed breaches. Vulnerability exploitation accounted for 31 percent of initial access, up 55 percent from the prior year, while median resolution time for a critical vulnerability reached 43 days. The current Verizon breach research raises the bar for a triage case. A model must shorten the path to a correct owner and verified repair. Faster prose is not enough.

The report also found that attackers verifiably researched or used generative AI across a median of 15 attack techniques. That development makes current local evidence more important. Public severity alone cannot tell a defender whether an attacker can turn one affected component into a path through this environment.

Where should people stay in the loop?

People should own high consequence routing, conflicting evidence, policy exceptions, changes to production, risk acceptance, and closure. The system can prepare the record and run allowed read only checks. It should stop when a needed source is missing or the proposed action exceeds the task contract.

Review the decision packet, not a long model conversation. Show the claim, decisive evidence, contradiction, unknown, route, owner, and next verification step. Record the reviewer change as structured feedback. Repeated corrections should become a data or policy fix, not permanent manual cleanup.

Microsoft reported on March 13, 2024 that experienced analysts in its randomized controlled trial worked 22 percent faster and were 7 percent more accurate with Security Copilot. The trial results support testing speed and quality together. A local fleet exercise should use the same discipline rather than borrowing the result.

Which scorecard should the case study publish?

Publish input count, valid denominator, duplicate rate, identity failure rate, evidence completeness, route distribution, reviewer agreement, material error rate, miss rate, median active minutes, 90th percentile active minutes, queue wait, correction hours, operating cost, and verified closure rate. Break each result down by task class and asset group.

Add a short error review. Name the five most common failure reasons and the control added for each. If the same stale inventory or asset join problem dominates every week, the model is not the main issue. Fix the evidence pipeline.

Frequently asked questions

Is this an Artemes customer case study?

No. It is a reproducible exercise with labeled assumptions and public sources. Artemes does not claim customer outcomes that have not been approved and verified.

Can AI close low risk findings automatically?

Not at the start. Earn that authority on a narrow finding class through replay, independent evidence, stable error rates, and a fresh closure check.

What if endpoint evidence is missing?

Route the case to evidence collection and state the gap. Missing data is not proof of safety and should not be converted into a clean verdict.

How many cases make a useful pilot?

Use enough cases to include normal volume, rare errors, difficult assets, and different shifts. Report the count and confidence range instead of relying on a universal sample size.

Executive takeaway

Demand a case record you can replay. The useful output is not a ranked list. It is a sourced explanation of what is known, what remains unknown, which policy chose the route, who owns the next step, and what evidence will prove closure.

Start with one finding class and 100 historical cases. Run the evidence checks, blind the reviewer to the model verdict, record every disagreement, and expand only after quality holds. Artemes AI uses deep endpoint context with AI driven analysis to support that kind of reviewable decision, while practitioners keep control of the operational outcome.