Tokens and logits relation in LLM
A token is a piece of text the model understands. It may be: Now each token will have a specific numerical value assigned. Token Token ID I 40 love 3047 machine 7342 learning 7524 Above is take from gptforworks website. Each LLM has its own tokenizer and token IDs. Where Logits Come In, Logits = […]
