Antivirus and EDR — signatures, heuristics, behavioral detection, and evasion

Antivirus (AV) and Endpoint Detection & Response (EDR) are host-level defense layers. Understanding how they work — and where they fail — allows you to configure them better and detect when they are being bypassed.

Classic antivirus — signature-based detection

Traditional AV maintains a database of hashes and byte patterns of known malware:

Detection process:
  1. File written to disk
  2. AV computes the hash (MD5/SHA-256)
  3. Compares against the signature database
  4. Match → quarantine or deletion

Limitation:
  Any modification to the file changes its hash
  A single changed byte = new hash = evasion

Heuristics — static behavioral analysis

Analyzes the file’s code without executing it, looking for suspicious patterns:

Heuristic indicators:
  - Base64-encoded strings inside the executable
  - Import of suspicious APIs: VirtualAlloc, WriteProcessMemory
  - Executable sections with high entropy (compressed/encrypted code)
  - Missing or invalid digital certificate
  - Packed with UPX or an unknown packer

EDR — runtime behavioral detection

EDR monitors process behavior during execution:

Telemetry captured by EDR:
  - System calls: CreateRemoteThread, OpenProcess, NtAllocateVirtualMemory
  - Network connections: destination, port, protocol
  - Registry modifications (Run Keys, services)
  - File creation and modification
  - Injection into other processes
  - Access to protected process memory (e.g., LSASS)

Example behavioral rule:
  Word.exe → spawns PowerShell → PowerShell downloads from URL
  → ALERT: malicious macro (real-world attack pattern)

Evasion techniques (defensive context)

Understanding evasion helps configure defenses better:

Signature evasion

- Code obfuscation (random variable names, encrypted strings)
- Custom packer: compresses and encrypts the payload, decrypts in memory
- Living off the Land (LOLBins): abuse legitimate OS binaries
  Examples: certutil.exe, mshta.exe, regsvr32.exe, rundll32.exe

EDR evasion

- Process hollowing: create a suspended legitimate process, replace its memory
- Direct syscalls: bypass EDR hooks by calling syscalls directly
- AMSI bypass: corrupt the Antimalware Scan Interface scan buffer
- Unhooking: restore original code in DLLs monitored by the EDR

Defense: modern EDR uses kernel callbacks and ETW (Event Tracing for Windows), which are much harder to bypass than user-mode hooks.

How to configure AV/EDR properly

1. Enable real-time protection — never disable it for "performance"
2. Update signatures automatically — every hour, not once a day
3. Enable AMSI integration — scans PowerShell, VBS, JS scripts
4. Configure alerts for high-risk events:
   - LSASS access
   - PowerShell encoded commands (-EncodedCommand)
   - Child process creation by Office/PDF readers
5. Integrate EDR with SIEM for correlation with other events

EDR vs AV: practical difference

AV:
  - Detects known threats by hash/signature
  - Response: automatic quarantine or deletion
  - Visibility limited to disk

EDR:
  - Detects anomalous runtime behavior
  - Response: endpoint isolation, process kill, remote forensics
  - Visibility: disk, memory, network, registry, processes
  - Retains historical telemetry for retrospective investigation

AV protects against the obvious. EDR is necessary for advanced threats.