In the rapidly evolving landscape of cybersecurity, leveraging Large Language Models (LLMs) and AI agents to perform security tasks, and more importantly - understand security context, has become a game-changer. However, ensuring the reliability and quality of these AI-driven tools is crucial. In this talk we'll explore the built-in challenges with LLMs and AI agents for cyber security, and how we solved them to provide a product that is a security companion to the team.