WhatEach token ID is swapped for a fixed list of numbers, a vector. In GPT-2 small, the demo model, it has 768 numbers.
HowThe lists are learned in training: tokens used in similar places end up with similar lists. A second list that encodes the position is added.
Why it mattersThe model cannot compute with letters, only with numbers. From here on everything is arithmetic on these lists.