We present TLS-Attacker, a novel framework for evaluating the security of TLS libraries. Using a simple interface, TLS-Attacker allows security engineers to create custom TLS message flows and arbitrarily modify TLS message contents in order to test the behavior of their TLS libraries. Based on TLS-Attacker, we first developed a two-stage TLS fuzzing approach. Our approach automatically searches for cryptographic failures and boundary violation vulnerabilities. It allowed us to find unusual padding oracle vulnerabilities and overflows/overreads in widely used TLS libraries, including OpenSSL, Botan, and MatrixSSL. Our findings encourage the use of comprehensive test suites for the evaluation of TLS libraries, including positive as well as negative tests. We used TLS-Attacker to create such a test suite framework, which finds further problems in TLS libraries. TLS-Attacker is an open source tool, and is currently being deployed for internal tests in Botan and MatrixSSL.