In recent years, the importance of using secure private messaging applications for communication has come to the forefront of public attention. It's not uncommon to find news articles written almost weekly on user privacy concerns and what the more tech savvy can do to limit their exposure to interception and monitoring from external parties. One particularly interesting website is the Electronic Frontier Foundation (EFF) secure messaging scorecard, which aims to assist users in choosing “Which apps and tools actually keep your messages safeâ€. This type of score card drastically simplifies the problem domain, and leads one to question what the tradeoffs are when installing an application from the list. While the advocacy of privacy based communication is something we love to see reach a mainstream audience, we believe the scorecard misses many considerations and metrics that are critical to the discussion. We have been performing a review of the documentation and source code of a subset of applications in the EFF scorecard to understand their privacy versus security tradeoffs. This is a subtle and often overlooked difference - as passive monitoring may be disrupted with the use of encrypted communications, but attacks against software vulnerabilities can negate the advantage of using those IM applications in the first place. We believe sharing this perspective is important to assist users in deciding on the right balance between privacy and security. We will present an introduction, run through the assessments we have performed of various clients, then share our perspectives to takeaway.