The parsers read each line of the source code and process tokens on each line. Tokens are discrete elements on a line of source code; they are language-dependent. For example, consider the following COBOL statement:

        MOVE 'SMITH' TO NAME.
 

Seven tokens appear in this example: MOVE, the two single quotation marks, SMITH, TO, NAME, and the period.

State variables are used to hold the current conditions and expectations created by the processing of prior tokens in order to process the current token. For example, if a single quote is found, the single quote state variable (state.single) is turned on. All tokens, regardless of multiple lines, are ignored until the matching single quote is found, or until the end of file is reached. In the COBOL and Assembler parsers, dependency names may be enclosed in quotes; all data after the dependency name is ignored until the matching quote is found. Dependency keywords (COPY or EXEC SQL INCLUDE) inside quotes are ignored. For example, consider the following COBOL statement:

        MOVE 'COPY B' TO ACTION.
 

B will not be placed into the dependency buffer because COPY will not be processed as a dependency keyword.