Break the Scanner on Purpose · Principles of Compiler Design

Format pairsTime 45 minBring a working build of your scanner

The point

Five experiments, each breaking the scanner in one specific way. Predict the outcome first, then run it. A prediction you got wrong teaches more than three you got right, so write it down before running anything — otherwise you will remember having predicted correctly.

Work on a copy

Every experiment below breaks the scanner on purpose, so work on a copy and keep the original intact for comparison:

cp -r student/topic-1-lexical-analysis/lexer /tmp/broken
cd /tmp/broken && make

Use YOUR scanner, not a reference one. Project 1 is due tomorrow, so by now you should have enough of it working to break — and breaking your own code teaches considerably more.

The experiments

Move the identifier rule above the keyword rules. Predict what tests/03_keywords_vs_ids.cm will produce, then run it. Which conflict-resolution rule did you just exercise, and why did the other one not save you?
Delete the "<=" rule. Predict the exact token count for tests/02_longest_match.cm before you look. Then explain the result in terms of the combined NFA: what happened to the accepting state that used to record that pattern?
Move the "<" rule above "<=", with both present. Predict, then run. This one surprises people — explain in one sentence why nothing changed, naming the rule responsible.
Make the error action call exit(1) instead of counting and continuing. Run tests/05_lexical_errors.cm, which holds four bad characters. How many did you learn about? Now imagine that file is 4,000 lines long.
Delete the position-counting call from the block-comment rule, leaving the rule itself. Run tests/04_comments.cm and look at the line numbers. Which later phase does this break, and how would the resulting bug report look to someone who did not know?

One design question, no experiment

Our language has no string literals. Suppose you wanted to add them: print("total = ");

Write the regular definition for a string literal.
What should happen if the closing quote is missing before end of line? Which error-recovery strategy from §9 of the lecture notes would you choose, and what does it cost?
Which phase should reject "abc" + 1 — scanner, parser, or semantic analyzer? Defend it in one sentence, using what each phase can actually see.

A hint on the third question

Ask what information each phase has. The scanner sees one lexeme at a time and no context. The parser sees the shape of the expression but nothing about types — expr '+' expr matches perfectly well. Only one phase in this course is in a position to know that one operand is a string and the other an integer, and that is exactly what that phase is for.

Deliverable

Extend activity1.md, or start activity2.md, with:

for each of the five experiments: your written prediction, what actually happened, and one sentence reconciling them;
your answers to the three string-literal questions;
one sentence on which experiment surprised you most, and why.

This feeds Lab Questions 1 and 2 directly, and it is the substance of the 'theoretical principles and design practices' section your Project 1 video has to cover.