Storage & health

All evidence stays on the host. Recent evidence lives in a DuckDB file (the hot store). Older evidence moves to compressed Parquet files (the cold store). Hunt reads both. Alerts, cases, rules and settings live in a separate SQLite control database.

Where data lives

Item	Path (default)
Hot evidence database	%ProgramData%\FreeDR\data\evidence.duckdb (+ evidence.duckdb.wal)
DuckDB spill directory	%ProgramData%\FreeDR\data\evidence.tmp (leftover temp files are deleted at start)
Cold Parquet files	%ProgramData%\FreeDR\data\cold\
Control database	%ProgramData%\FreeDR\data\control.sqlite3
Retention setting	%ProgramData%\FreeDR\config\evidence-lifecycle.json

The DuckDB engine runs with 2 threads, extension auto-install off and external access off. It uses its own private Windows heap (ADR-0076). Each start logs duckdb.dll heap isolated in a private heap.

Evidence lifecycle

Stage	Rule	Setting
Hot	Rows stay in DuckDB until they are older than hot_retention, measured by written_at	hot_retention in the administrator TOML (seconds). Default and minimum: 2700 s (45 min). Env override FREEDR_HOT_RETENTION. Not editable in Settings.
Cold export	Every hot_retention interval, older hot rows are written to Parquet and then deleted from DuckDB	Same setting
Final expiry	Each export pass deletes whole cold files whose cutoff is older than retention_days. It also prunes artifact_path_state_v1 and expired control records (alerts, closed/merged cases, run audit; open cases are kept).	retention_days, default 90, range 1–3650. Editable in Settings. Lowering it asks for confirmation.

Notes:

• Retention is time-based only. There is no row-count or disk-size cap, and no early deletion when the disk fills. Free disk space is shown in storage diagnostics but nothing acts on it.
• Expiry runs even if the export failed. A file that cannot be deleted is retried on the next pass. Rows can outlive retention_days by up to one file's time span.
• Editing hot_retention from the console is not available.

Cold Parquet export

Format	Parquet, ZSTD level 12
Layout	Flat directory, one file per source per run: <hot_table>__<occurred_min_us>__<occurred_max_us>__<cutoff_us>.parquet. Progress is tracked in run.marker.
Safety	Write .parquet.tmp, verify row count, compare a full typed row hash, fsync, rename atomically, then delete exactly the exported hot rows. An interrupted run resumes with the same cutoff.
Dedupe	Rows already present in an older overlapping cold file are skipped. Reads prefer hot rows on duplicate event_id.
Schedule	First run one interval after first start. A failed run waits a full interval (no fast retry).
Health	Collector evidence-cold-export: attempts, successes, failures, hot_retention_seconds, next_due_at_us, last_success_at_us, last_failure_at_us. Degraded after a failed attempt.

What you can query

Who	Names	Tiers
Hunt (console, AI query tool, API /api/v1/hunt/query, /export)	events, and event_log (Windows Event Log fields) when the query names it	Hot + cold
Detection rules	Exactly one typed source in FROM: kernel_process_etw, registry_etw, file_etw, usn, artifact, windows_event_log. Subqueries may also use <source>_context, other typed sources and artifact_path_state. events is not allowed.	Hot only, within the rule timeframe (2, 8 or 15 min). Context reads look 60 s further.
Internal reads (process tree, enrichment, tuning)	Typed sources	Hot + cold

Physical table names (kernel_process_etw_v6, …) are rejected in Hunt.

Per-source tables

Source	Hot table	event_type values
Process	kernel_process_etw_v6	process.start, process.exit, image.load
Registry	registry_etw_v4	registry.key_created, registry.key_delete, registry.value_set, registry.value_delete, registry.key_security_set
File	file_etw_v5	file.create_open, file.write, file.rename, file.delete, file.set_info
USN	usn_v3	usn.change
Artifacts	artifact_v5 (+ artifact_path_state_v1)	artifact.content_version
Event Log / Sysmon	windows_event_log_v2	event_log.* (124 types)
(internal)	source_checkpoints_v1	Per-source durable cursors (USN volumes, Event Log bookmarks)

Hot tables have no indexes. Time and event_id predicates prune cold files.

The events view

events is a UNION ALL over all six sources with one shared column set. A source that lacks a column returns NULL for it.

Column group	Columns
Always present	event_id (lower-case UUID text), event_type, occurred_at, written_at
Process	pid, parent_pid, session_id, user_sid, user_name, image_path, command_line, image_base, exit_code, is_rundown, parent_image_path, parent_command_line, parent_start_at, grandparent_image_path
Actor (process, registry, file, event log)	actor_pid, actor_image_path, actor_command_line, actor_start_at, actor_user_sid, actor_user_name
Registry	key_path, value_name, value_type, value_data (hex)
File / USN	path, created, old_path, volume_serial, file_id, byte_offset, byte_count, journal_id, usn, reason
Artifact	file_size, created_at, modified_at, sha256, authenticode_status, signer, ads (JSON), pe_image_size, pe_checksum, pe_timestamp, pe_preferred_base, acquisition_reason
Event Log	task_name, task_content, task_action, task_result_code, script_block_text, event_payload

All *_at columns in events are epoch microseconds (integers).

Hunt limit	Value
Query text	16 KiB
Rows	1,000 interactive, 100,000 export
Output	16 MiB interactive, 64 MiB export, at most 64 columns

Ingestion

Stage	Behaviour
Writer	One ordered database writer for all sources. Queue: 1,024 slots (64 control, 960 evidence).
Batching	ETW turns of up to 2,048 events / 2 MiB, flushed every 250 ms. Store batches are capped at 8,192 events / 8 MiB. Checkpointed sources (USN, Event Log) use batches of at most 256 events.
Set-based append	Each batch is loaded into a temp stage with the DuckDB Appender, classified in one query and inserted with one INSERT … SELECT. The artifact path state is updated in the same transaction.
Checkpoints	Evidence and the source cursor commit in one transaction, with a revision check
Dedupe	Every source computes a deterministic event_id (UUIDv5 of the event content). A row already in hot storage, or repeated in the same batch, is counted as a duplicate replay and not inserted.
Durability	DuckDB CHECKPOINT every 60 s. A failed commit reopens the database (evidence_reopens).
Artifact requests	After a batch commits, inserted rows with eligible paths are queued for artifact acquisition

Collection-time reduction (first-only and merge windows, filters) is described on each source page.

ETW ring buffers

Kernel ETW callbacks copy raw records into per-lane byte rings. Decode workers drain them.

Lane	Source	Decode workers
lifecycle	Process	2
mutations, access	Registry	8
control, bulk	File	2

Rule	Value
Start / idle size	16 MiB per lane (80 MiB for all five)
Growth	When use reaches 2/3 of capacity, the ring doubles: 16 → 32 → 64 → 128 → 256 MiB
Maximum	256 MiB per lane (1.25 GiB if all five peak at once)
Shrink	After a full quiet minute, shrinks to the smallest step whose peak use in that minute was at most 25 % (never below 16 MiB). The old buffer is freed after admission reopens.
Ring full below 256 MiB	The producer waits up to 2 s for growth instead of dropping. After 2 s it is counted as full. During a buffer swap it waits rather than dropping.
Ring full at 256 MiB	Counted immediately (byte_full)

Lane counters use the form source_<id>_raw_lane_<lane>_*: admitted, byte_full, unavailable, malformed, oversized, overflow_*, size_*, wrap_markers, and gauges capacity_bytes, used_bytes, free_bytes, queued_records, high_water_bytes. Decoded evidence then passes through an evidence lane (Process 8 MiB, Registry 64 MiB, File 48 MiB) with evidence_backpressure_waits and evidence_lane_full.

Health

The Health page and GET /status report one entry per collector. Each entry has a state (starting, running, stopped, failed), a degraded flag, counters and gauges. /status is degraded while any collector is degraded.

Collector ID	UI label	Covers
unified-etw	Windows ETW	Process, registry and file ETW. Per-source counters are prefixed source_<id>_.
windows-event-log	Windows Event Log	Event Log and Sysmon
usn	NTFS change journal	USN
artifact-inventory	File artifact inventory	Artifacts
evidence-cold-export	Cold storage export	Parquet export and expiry
db-worker	Evidence database writer	Ingestion

What makes a collector degraded

Collector	Fault signal
unified-etw (per source)	Immediately on ring loss, an increase in the ETW controller's events_lost/buffers_lost, a worker failure or a rejection. Also on ≥ 1,000 malformed records in 30 s, or 30 s without progress while work is queued. The fault clears after 30 s clean.
windows-event-log	A channel that exists on the host is closed waiting to reopen, or a commit is being retried. Absent channels (for example no Sysmon) do not count.
usn	Worker failed, a faulted volume or failed worker, or growth in read_failures, decode_failures, checkpoint_failures or integration_failures. See USN.
artifact-inventory	Growth in capacity_drops, commit_drops, write_failures or lookup_failures. Per-file errors do not count.
db-worker	Growth in rejected, command_queue_full, acknowledgement_unavailable, invalid or write_errors
evidence-cold-export	A failed export attempt

Degraded hold and episodes

From ADR-0080:

• After a fault, a collector stays degraded for 30 minutes (DEGRADED_HOLD) after its last fault. Then it clears on its own.
• A fault after the previous period has cleared starts a new episode.
• Each collector reports a degraded_episodes counter and a degraded_last_fault_unix_s gauge.
• Episode history is kept in memory and resets when the agent restarts.

Common ETW counters

Group	Names (prefix source_<id>_)
Flow	routed, decoded, filtered, ingestion_accepted, ingestion_rejected
Raw admission loss	raw_lane_full, raw_byte_full, raw_unavailable, overflow_drops, raw_malformed, raw_oversized
Decode	unsupported, malformed, decode_failed, decode_oversized, rejected
Workers	shutdown_drops, worker_drops, worker_failures, queue_depth, queue_high_water
Persistence	persistence_inserted, persistence_duplicate_replay, persistence_retry, persistence_rejected, persistence_unavailable
Session-wide	events_lost, buffers_lost, restart_attempts, restart_failures, unexpected_exits, worker_panics
TDH schema	schema_not_found, required_property_missing, property_type_mismatch, malformed_or_truncated_payload, cache_capacity_exhausted, …

The Health page also lists, under "Other reported loss", every counter whose name contains drop, loss or lost and is above zero.

Storage diagnostics

GET /api/v1/storage/diagnostics and the Health page Storage section show: evidence schema version, rows per source table, hot and WAL bytes, cold bytes and file count, free disk space, and control database integrity.

The event flow chart (GET /api/v1/events/flow?source=&range=1h|24h) shows ingest volume per source.