AI Tokens Explained - The TLDR Version

In our recent Immerso partnership announcement, we mentioned that Immerso enjoys access to 1 trillion (and growing) AI Data tokens.

But what does this mean? Are they the same as crypto tokens? And the answer is no.

Think of AI data tokens as language broken down into individual pieces of language that AI models can understand - something like the ‘’bytes’’ of AI text processing.

For example: "I love the metaverse" would be broken down by an AI model into  = "I", "love", "the", "metaverse" - each being its own token.

Furthermore, AI data tokens aren't just individual words. They can be parts of words, punctuation marks, or even spaces. In English, one word is typically 1-2 tokens, but this also varies by language and writing style.

So why do a trillion+ AI data tokens matter? Training an AI on a trillion tokens is like giving it a massive library of human knowledge to learn from. Take a read of just some of the benefits:

When an AI processes more tokens during training, it develops a deeper grasp of language patterns and context. With trillions of tokens, the AI encounters countless examples of how language is actually used in different contexts.

More tokens generally mean better performance. Think of it like this: the more examples an AI sees during training, the better it gets at generating accurate and helpful responses.

With a trillion tokens, an AI can learn from a vast range of topics and writing styles. This means it can handle everything from creative writing to technical analysis. It's like the difference between someone who's read a few books versus someone who's read an entire library.

So why is one trillion tokens mind-boggling? It is an astronomical amount of text, far more than any human could read in multiple lifetimes. 

• The average book contains roughly 90,000 words or about 120,000 tokens
• This means one trillion tokens is equivalent to reading about 8.3 million books
• If you read one book every day, it would take you over 22,800 years to read that much text
• A human reading at a fast pace (300 words per minute) would need 570 years of non-stop reading to get through one trillion tokens

Such an amount is enough to create incredibly capable AI systems, and although training on trillion-token datasets requires massive computing power and is incredibly expensive, the results speak for themselves in terms of the AI's capabilities. 

Immerso is creating systems that have consumed more text than a human could read in thousands of lifetimes - and that's what makes them such a powerful partner for Everdome.