Suricata (EVE) Analyst

Scope and Constraints

• •Expect Suricata EVE logs (JSON lines) under the current directory.
• •Logs may be larger than tool/file limits; avoid loading entire files into memory.
• •Prefer sampling (head) and targeted filtering (jq, rg) to understand schema and pick relevant event_type values.

Working Agreements

• •Do not delete scripts after creating them.
• •Before writing new code, check whether an existing script in the current directory already solves (or nearly solves) the task.
• •Keep intermediate outputs; do not "clean up" by deleting them.
  • •If you generate a throwaway output file, rename it to include a timestamp suffix like -YY-MM-DD_HH-MM.md (or similar) instead of deleting it.

Analyst Notes (Required)

• •For each analysis task, create a timestamped Markdown log capturing:
  • •What you did (commands/scripts and parameters)
  • •What you looked for / why
  • •Small, representative samples of the data (not exhaustive)
• •Naming convention: analyst_log-YY-MM-DD_HH-MM.md (prefer UTC).

Suricata (EVE) File Format

• •Treat the EVE file as JSON Lines (one JSON object per line).
• •Usually ignore event_type: "stats" for threat-hunting (it is operational telemetry, not traffic details).
• •Focus on public destination IPs (not RFC1918) when looking for suspicious egress.
• •High-signal event_type values for egress analysis often include:
  • •dns, tls, quic, http, flow, and alert (when present)

Coding Style

• •Use Python or jq to parse and analyze logs as needed.
• •Use orjson instead of the built-in json library.
• •Use polars to transform JSON -> Parquet when needed.
• •Use duckdb to query Parquet efficiently.
• •Use uv (not pip) for virtualenvs and dependency installs.
• •Do not hardcode filenames; take input paths from sys.argv (avoid argparse unless the user requests it).
• •Create a virtual environment with uv and maintain a requirements.txt file so you have all the libraries you need