Tokenize by Hand, Then by Machine

The point

Before you let a tool tokenize for you, tokenize by hand. Every disagreement between your answer and the machine's is a rule you had not noticed you were assuming — usually longest match, first match, or a phase boundary.

Do this

1. On paper, alone, five minutes. Write the token stream for the line below. One token per line: kind, then the lexeme that matched. Do not run anything yet.

   x=y<=z*2;  // done!

2. Compare with your partner. Where you differ, decide which of you is right and why before looking at any output. Write each disagreement down in one sentence.
3. Now run it. Put that line in a file and run a scanner over it. On day one you have two options: your instructor will run the reference lexer on the projector, or — if you brought a working scanner forward from CST-301 — use yours:

   cd student/topic-1-lexical-analysis/lexer
   make
   ./lexer yourfile.cm

4. Account for every difference. For each, name the rule that explains it: longest match, first match, or 'that is not the scanner's job'.
5. Then break it. Add a line containing a character the language does not have — x = y @ z; — and confirm the error names the right line AND the right column. Count the columns by hand to check.

Questions worth arguing about

• Is // done! a token? If not, what happened to it, and which phase would have to care if it were?
• How many tokens is <=? Which of the two conflict-resolution rules settles that, and could rule order change the answer?
• The line has no spaces around =. Did that matter? Should it? What in the theory says so?