Ilya Gusev, Feb 1, 2023
Training language models, such as OpenAI's GPTs, is now a crucial step in advancing the field of natural language processing. This post will explore training language models from scratch, focusing on efficiency. We will cover existing models, text datasets, scaling laws, and optimization tweaks.
Whether you're interested in training a language model or using the existing ones, this post will provide valuable insights and information to help you achieve your goals.
But first, let me answer several popular questions and bust some myths.
No. ChatGPT has no API, so there is no technical and legal way to use it for commercial purposes. However, people are sometimes confusing ChatGPT with GPT-3. For GPT-3 situation is different: there is an API that we can use.
A lot of feedback from data annotators is embedded into ChatGPT through reinforcement learning. The format of interactions is also different:
A figure from a famous Notion article
No, but you can close the gap with the right prompt. See here.
Example of a prompt: As an advanced chatbot named ChatGPT, your primary goal is to assist users to the best of your ability. This may involve answering questions, providing helpful information, or completing tasks based on user input.
$0.02 per 1000 tokens. One token equals around 4 chars in English. OpenAI charges both for input and output tokens.
Imagine you’d like to use it for text summarization of hotel reviews, and there are 1 million hotels to summarize. Let’s say there are 10 reviews on average, and each review has 100 tokens on average. So you fit them in 1000 tokens per hotel. It will cost you $20K. With some tweaks, you can reduce this number, of course. Still, in the end, there will be an irreducible cost of $4K for tokens of generated summaries, which are also 100 tokens on average, and for the same amount of input tokens.