Osquery Performance Tuning: Keeping the Agent Lightweight

An osquery agent that hurts the host gets uninstalled. Performance tuning is not a nice to have. It is the difference between a fleet that stays deployed and a security project that quietly gets ripped out.

Every endpoint agent competes for the same CPU, memory, and battery as the work the machine is actually there to do. Push too hard and the complaints start: a developer's laptop fan spins up during a build, a database server misses its latency target, a helpdesk ticket blames the security agent. Once that reputation sets in, the tool loses the argument regardless of how good the telemetry is. So the goal of tuning is not raw efficiency for its own sake. It is staying invisible enough that nobody has a reason to remove you.

The good news is that osquery was designed with this fight in mind. It ships with a watchdog that caps its own resource use, and it exposes its own performance data as tables you can query. This piece covers what makes a query expensive, how the watchdog protects the host, how to find your worst offenders, and the settings that keep the agent lightweight at scale.

What makes an osquery query expensive?

Cost comes from three places: how much the query has to gather, how often it runs, and whether it returns the whole result every time. A query against processes or listening_ports is cheap. A query that walks the entire filesystem, hashes files, or joins several large tables is not, because osquery has to materialize each table before it can join.

The classic mistake is a broad file query. Something like a hash of every file under a large tree forces osquery to stat and read across the disk on every run. Do that on a short interval across a fleet and you have built a distributed I/O storm with your own monitoring. The second mistake is frequency. A query that costs a little is fine once an hour and painful once a minute, and people reach for short intervals reflexively. The way you write the query matters as much as which table you hit, which is why the fundamentals in writing osquery queries pay off directly in performance.

How does the watchdog keep the agent in line?

Osqueryd runs a watchdog process that monitors the memory and CPU of the worker executing the schedule. If the worker crosses a limit, the watchdog kills and restarts it, and the query that caused the breach gets denylisted for 24 hours so it cannot keep hammering the host. There is exponential backoff on repeated offenders. This is a safety net, not a tuning strategy, and it is on by default.

The defaults are conservative. The watchdog caps worker memory at roughly 200 megabytes and CPU at about 10 percent of a core sustained over a short window. A more restrictive profile drops those to 100 megabytes and 5 percent. You control the behavior through command line flags:

# watchdog levels: 0 = normal, 1 = restrictive, -1 = disabled
--watchdog_level=0
--watchdog_memory_limit=200
--watchdog_utilization_limit=10

Read the trade the watchdog is making. It protects the host, which is correct, but the cost of that protection is that a badly tuned query does not error loudly. It gets benched, and collection for that query silently stops. So a watchdog that never trips is the goal, because a watchdog that trips means you have a blind spot you may not know about. The full flag reference lives in the official osquery command line flags docs.

How do you find your expensive queries?

You do not have to guess. Osquery keeps performance statistics on its own schedule and exposes them through the osquery_schedule table. Query it and the worst offenders sort straight to the top:

SELECT name, executions, output_size,
       wall_time, average_memory
FROM osquery_schedule
ORDER BY average_memory DESC
LIMIT 10;

That single query tells you which scheduled queries burn the most memory, how long they run in wall time, how many times they have executed, and how much data they produce. Watch average_memory against the watchdog limit and output_size for queries dumping far more rows than you actually consume. This is the measurement step people skip, and it is the one that turns tuning from guesswork into a short, ranked list of things to fix.

Snapshot versus differential, and the settings that matter

The biggest single lever is snapshot versus differential collection. A differential query returns only what changed since the last run, so a table of 400 processes that barely moves produces a handful of rows. A snapshot query returns the full result every time, all 400 rows, every interval. Snapshot mode has real uses, but reaching for it by habit multiplies both CPU and log volume for no benefit. Default to differential and use snapshot only when you truly need the complete current state each run.

Two more settings earn their keep. Schedule splay spreads query start times so a fleet does not run the same heavy query in lockstep and create a synchronized spike, and it defaults to a sensible ten percent. Intervals are the other one: match the interval to how fast the data actually changes. Installed packages do not need a 60 second query. Match it to reality:

{
  "schedule": {
    "process_events": { "query": "SELECT * FROM ...", "interval": 60 },
    "installed_packages": { "query": "SELECT name, version FROM deb_packages;", "interval": 86400, "snapshot": false }
  },
  "options": { "schedule_splay_percent": 10 }
}

Run the math on why this matters at scale. Suppose one query burns two seconds of CPU per run. At a 60 second interval that is 1,440 runs a day, roughly 48 minutes of CPU per host, from a single query. Move the same query to a 15 minute interval where the data allows it and you have cut that by a factor of fifteen without losing anything real. Across ten thousand hosts, that decision is the difference between a fleet nobody notices and a procurement conversation about why the security agent is on every performance report. Scaling these choices across a large deployment is the heart of osquery fleet management, and the split between the interactive shell and the scheduled daemon is covered in osqueryi vs osqueryd.

Why this matters more now

Osquery is under active development, with releases landing throughout 2025 and into 2026, and each version tends to add tables and refine collection behavior. New tables are tempting to schedule aggressively the day they ship. That is exactly when a performance review pays off, because an untuned new query on a short interval is the most common way a stable agent suddenly starts tripping the watchdog. The official performance safety documentation is worth rereading whenever you adopt a new build or a new table.

A tuning checklist

Keep the agent lightweight with a short, repeatable pass:

• Measure first. Query osquery_schedule and rank by memory, wall time, and output size before changing anything.
• Default to differential collection. Reserve snapshot mode for queries that truly need full state each run.
• Match every interval to how fast the underlying data changes. Inventory is daily, process and network activity is frequent.
• Keep splay on so the fleet does not spike in unison.
• Set the watchdog deliberately and then aim never to trip it. A tripped watchdog is a silent collection gap, not a win.
• Rerun the measurement after changes and after every osquery upgrade or new table you schedule.

The measurement and the settings are mechanical. The judgment call, which query is worth its cost given what it actually tells you, is the part that does not automate with a flag. That is a theme across our osquery packs guide and it is where Artemes AI focuses: pairing deep endpoint context with AI driven analysis that weighs what a signal is worth against what it costs to collect, so the fleet stays light and the telemetry stays useful. Osquery gives you the dials. Deciding where to set them is the real work.

Frequently asked questions

How much CPU and memory does osquery use?

By default the watchdog caps the worker at roughly 200 megabytes of memory and about 10 percent of a CPU core over a short window, with a restrictive profile at 100 megabytes and 5 percent. Actual use depends entirely on which queries you schedule and how often, which is why measuring your own schedule matters more than any general number.

What is the osquery watchdog and should I disable it?

The watchdog monitors the worker process and restarts it if a query exceeds the memory or CPU limit, denylisting the offending query for 24 hours. Do not disable it in production. If a query keeps tripping it, fix the query, because disabling the watchdog removes the protection that keeps a bad query from harming the host.

Why did one of my scheduled queries stop returning data?

It was likely denylisted by the watchdog for exceeding a resource limit. Check the osquery_schedule table and the daemon logs. The fix is to make the query cheaper through a tighter filter, a longer interval, or differential collection, then let it resume.

Should queries use snapshot or differential mode?

Differential by default. It returns only what changed, which cuts CPU and log volume dramatically for tables that are mostly stable. Use snapshot only when a use case truly needs the full current result set on every run.

The takeaway

Treat osquery performance as an operational discipline, not a one time setup. Measure your schedule with the osquery_schedule table, default to differential collection, match intervals to how fast data actually changes, keep splay on, and set the watchdog so it protects the host without ever needing to fire. The agent that survives in production is the one nobody notices, and staying unnoticed is a choice you make query by query. Get it right and osquery delivers the visibility with a footprint the rest of the organization never has to think about.