LLM System Analysis: From Telemetry to Judgment

LLM system analysis does not fail because the model cannot read telemetry. It fails because teams hand the model stale rows, weak identity, and missing context, then treat fluent text as proof.

A language model can connect a package record, a running process, a listening port, an owner, and a vendor advisory. That is useful. It can also connect facts collected from different hosts or different hours and produce a clean, wrong story. The operating problem is evidence construction, not prompt style.

The right design is strict. Bind every observation to one asset and one time. Separate facts from inference. Make missing data visible. Let deterministic checks test material claims. Keep policy outside the model. Then collect fresh evidence after any action.

What is LLM system analysis?

LLM system analysis is the use of a large language model to interpret current machine state and explain what that state means for security or operations. Inputs may include installed software, processes, services, ports, users, configuration, recent events, asset role, network exposure, deployment source, and external intelligence.

The model is not the sensor. It is not the source of truth. Its job is to compare observations, identify a plausible relationship, expose contradictions, ask for missing evidence, and produce a candidate judgment that another control can test.

That distinction matters. A summary says what the input contains. Analysis says what the facts support, which facts oppose the conclusion, and what must be checked next. A decision adds policy, ownership, consequence, and authority. One model call should not quietly collapse all three jobs.

Why is system context the hard part?

Security teams already have plenty of data. They do not have a coherent view at the moment a decision is due. The SANS Institute published its 2026 SOC Survey findings on June 11, 2026 after collecting 444 practitioner responses and 69 responses from CISOs and senior security leaders. In that 2026 SANS SOC research, 24 percent of leaders named limited visibility across the enterprise as the biggest barrier to effective security operations.

More tools do not fix that gap. A package inventory may update every day, an EDR event may arrive in seconds, the ownership record may be six months old, and the network path may have changed during a deployment. Put those values into one prompt without their clocks and the model cannot know that they describe different realities.

The September 3, 2025 SANS AI survey found that 33 percent of respondents used AI to investigate incidents and 26 percent used it to respond. Yet 66 percent reported excessive false positives from AI systems. The lesson is not that models are useless. It is that unbounded input creates unbounded claims.

What belongs in an evidence bundle?

Start with a durable asset identifier, not a hostname alone. Record the tenant, environment, operating system, service role, owner, and deployment source. Add the source and collection time to every observation. If two systems disagree, preserve both values and the disagreement. Do not ask the model to choose a convenient truth.

Next, collect only facts needed for the question. A vulnerability relevance check may need package version, affected function, process state, configuration, listener, network path, privilege, and compensating control. It probably does not need a month of raw endpoint events. More tokens can hide the answer as easily as too few tokens.

Each field needs four labels: source, observed time, expected freshness, and quality. Quality can be direct, derived, reported, or unknown. That small distinction prevents a model from treating an owner's description of a control as equivalent to a direct configuration check.

Add invalidation rules to the bundle. A reboot can end a process fact. A deployment can replace package and configuration facts. A network policy release can invalidate an access test. Store the event that should force a refresh, not only a fixed expiration time. The model should never carry an old conclusion across a state change because the prose still reads well.

• Direct fact: the process exists on asset 7 at 14:03 UTC.
• Derived fact: the process path maps to package release 4.8.2.
• Reported fact: the service owner says the route is private.
• Unknown: no current route test is available.

How do you collect system state without flooding the model?

Ask narrow questions at collection time. The osquery project gives a useful example because it exposes operating system facts as SQL. Its official site uses this query to find running processes whose original binary is no longer on disk:

SELECT name, path, pid
FROM processes
WHERE on_disk = 0;

One returned row is a lead, not a verdict. The process may be malicious, a legitimate updater may have replaced the file, or the source may be reporting an expected state. Ask for parent process, user, hash, signing information, network activity, and nearby events only if those facts can change the decision.

Choose snapshot and change data deliberately. The official osquery logging guide explains that scheduled queries normally emit differences between runs, while snapshot queries emit a full point in time result. Differences are efficient, but reconstructing current state requires the prior rows, epoch, and counter. Feed a model one added event without that history and it may describe a process as current after it has stopped.

How should LLM system analysis work?

First, define the question in a testable form. Bad: "Is this host secure?" Better: "Does the current evidence support exposure to CVE X on asset 7, and which missing fact would change the answer?" A bounded question creates a bounded evidence request.

Second, normalize values before the model sees them. Convert times to UTC. Resolve package aliases. Preserve raw values beside normalized ones. Distinguish absent from false. Reject observations outside their freshness window. A null listener field must not become "no listener exists."

Third, require a structured response. The model should return the claim, supporting evidence IDs, opposing evidence IDs, unknowns, a proposed next test, confidence reason, and no action. Deterministic code should confirm that every cited evidence ID exists and that no expired fact supports a material conclusion.

Fourth, apply policy after analysis. Policy decides whether the evidence supports urgent remediation, planned work, a temporary control, accepted risk, or more collection. The model can explain the choice. It should not invent the organization's tolerance or grant itself authority.

Finally, verify with a fresh source. If a firewall rule changed, test the route. If a package changed, collect the installed version again. If a process was stopped, confirm both absence and service health. A ticket status is not system state.

What should happen when evidence conflicts?

Conflict is useful. It tells you where the system model is weak. Suppose inventory says OpenSSL 3.0.12 is installed, the running binary reports another library path, and the container manifest names a patched image. A weak prompt asks the model to choose. A sound workflow asks which runtime artifact serves traffic and collects that proof.

Set a contradiction policy. Direct runtime evidence usually outranks an old inventory record, but only for the runtime question it answers. Desired state from infrastructure code explains intent, not necessarily reality. An owner record assigns responsibility, not technical truth.

Unknown should be a valid final state. Route it with a named evidence request and a clock. Forced certainty produces attractive reports and bad work queues.

How much context should one analysis use?

Use simple cost and attention math. Assume one investigation starts with 18,000 raw events at an average of 120 tokens each. That is 2.16 million tokens before instructions or output. If deterministic filters reduce the case to 90 relevant facts at 80 tokens each, the model sees 7,200 evidence tokens. The reduction is 300 to 1.

The point is not a cheaper prompt. It is a cleaner decision surface. Preserve the raw events outside the model context, keep stable evidence IDs, and let the analyst expand one branch when needed. Compression without traceability creates another information loss problem.

How do you stop telemetry from becoming instructions?

Treat every collected string as untrusted data. File names, command lines, ticket comments, web content, and log fields can contain text that looks like a command to a model. The model must see a clear boundary between policy instructions and observed values. Tool arguments need schema validation and separate authorization.

This became harder to dismiss in 2026. On March 23, NIST described a public red team competition that tested 13 frontier models with more than 250,000 attack attempts from over 400 participants. The NIST agent hijacking analysis reported at least one successful attack against every target model. A prompt guard alone is not an authorization system.

Give the analysis service read access to the smallest useful data set. Put write actions behind another identity, policy gate, and approval boundary. Log the model version, prompt version, evidence IDs, tool calls, policy result, and final human decision. Reconstructability is a control.

How should teams test the analysis?

Build a replay set from closed investigations. Include confirmed exposure, confirmed absence, stale data, mixed host identity, missing fields, conflicting controls, poisoned text, and cases where the correct answer is unknown. Score claims, evidence citation, abstention, unnecessary collection, and unsafe tool requests separately.

Sample live decisions by consequence. Review every proposed destructive action, every high consequence downgrade, and a random share of ordinary cases. Track how often analysts reverse the conclusion and why. "Model accuracy" is too broad to improve an operating system.

Artemes applies deep endpoint context with AI driven analysis to narrow security findings and explain exact remediation. The useful part is not fluency. It is the chain from observed fact to bounded judgment to fresh proof.

Frequently asked questions

Can an LLM analyze raw endpoint logs directly?

It can, but raw logs are a poor decision interface. Normalize identity and time, select facts tied to one question, preserve evidence IDs, and keep the full log outside the prompt for replay.

Is retrieval augmented generation enough for system analysis?

No. Retrieval can find relevant material. It does not prove that the source is current, tied to the right asset, or authorized for this decision. Those controls belong in the evidence layer.

Should the model calculate the final risk score?

Let the model explain evidence and propose a classification. Use visible policy or deterministic math for the final queue decision when consistency and audit matter.

What is the safest first use case?

Start with read only evidence collection and a proposed conclusion on a narrow finding class. Compare it with analyst decisions before allowing any state change.

Executive takeaway

Stop asking a model to understand the whole environment. Pick one decision. Define the facts it needs, bind those facts to asset and time, require support and opposition, and test each material claim. Keep policy and execution outside the model. Then verify the result from fresh state.

For the broader operating model, start with the AI vulnerability management guide. The earlier analysis of AI reading endpoint data covers the control plane in more detail.