garak

Teams deploying large language models (LLMs) do not always have a systematic way to check whether those models can be made to produce unwanted outputs. Garak addresses this need by automating the exploration of known LLM vulnerabilities, much like nmap or Metasploit automate network security auditing.

The tool sends series of crafted prompts designed to trigger problematic behaviors: jailbreaks, prompt injections, data leakage, hallucinations, toxic content generation, and misinformation. It combines static, dynamic, and adaptive probes, then produces a detailed report showing the failure rate for each attack type.

Garak is aimed at security teams, ML engineers, and developers who integrate LLMs into their products and want to assess their robustness before or during production deployment, particularly within an automated red-teaming pipeline.

The tool assumes a reasonable level of comfort with the command line and Python. It is not designed for non-technical users looking for a graphical interface or a turnkey service.