This flashcard is just one of a free flashcard set. See all flashcards!

Intent:
Analyze text at the lexical level.

(optional steps:)
- Chop input into “pieces”.
- Classify each piece.
- Process classified pieces in a stream.

Prosses:
1. Chopping input into pieces with java.util.Scanner

                scanner = new Scanner(new File(...));

2. Tokens = classifiers of pieces of input

               public enum Token {
               COMPANY,
               DEPARTMENT,
               MANAGER,
               CLOSE,
               STRING,
               FLOAT,
               ...
               }

3. Classify chopped pieces into keywords, floats, etc.

public static Token classify(String s) {
       if (keywords.containsKey(s))
           return keywords.get(s);
       else if (s.matches("\"[^\"]*\""))
           return STRING;
       else if (s.matches("\\d+(\\.\\d*)?"))
           return FLOAT;
       else
           throw new RecognitionException(...);
}

4. Process token stream to compute salary total

Summary:
{
* Declare an enum type for tokens.
* Set up instance of java.util.Scanner.
* Iterate over pieces (strings) returned by scanner.
* Classify pieces as tokens.
       - Use regular expression matching.
* Implement operations by iteration over pieces.
       - For example:
             - Total: aggregates floats
             - Cut: copy tokens, modify floats

A problem with the Chopper Pattern

Imput:
company “FooBar Inc.” { ...

Pieces:
‘company’, ‘“FooBar’, ‘Inc.”’, ‘{‘...

There is no general rule for chopping the input into pieces.