Introduction

We all know that machine understands only low-level-language or machine readable language, but when it comes to AI models like ChatGPT, Google Gemini etc., an end user gives instructions or prompts in human redable language or high-level language. To us, words feel natural. But for a machine, text is nothing more than a long string of characters. Computers don’t “think” in words — they think in numbers.

So, how do we bridge this huge gap between human language and machine understanding?
That’s where tokenizers come in.