scanning pass

comment

 

tokenizer

read file, convert every character into a token

 
   

comments

  

strings

  

spaces

  

symbols

  

integers

  
   
   

token

  
 

  kind

 
 

  text

 
 

  line number

 
 

  position in line