This session will discuss two approaches that adversaries use to "steal" information about proprietary black-box AI systems such as ChatGPT. Will first show how providing access to LM output probabilities via API can allow adversaries to reveal the hidden size of LLMs, and will then discuss how training imitation models via distillation can provide a mechanism for creating copycat models.