Lexical analyser and parser communication

My answer below is in reference to the Flex-Bison (or Lex-Yacc) model of compilation only. I have little knowledge of other models.\n\nI think of the lexer / parser combination as two co-operating modules in the same program.\nWhen you use Flex with Bison you see that the latter calls a function yylex() provided by the former (yylex() is equivalent to the getNextToken() function in your question). So it makes more sense to think of them as co-operating units in a single program rather than two different programs. Moreover, if the lexer and parser were 2 different programs, you'd have to deal with inter-process communication, shared memory, and related issues, further complicating the task at hand.\n\nTo answer your second question:\nI can think of one important issue that could arise from the parser coming into action after the lexer had finished reading all input: memory usage would be enormous for even moderate-sized programs, as you would have to store data structures for every token, in memory (think of tokens like , and = occupying multiple bytes in memory and you'll quickly see why it's not scalable).\n\nAs for error handling: if the lexer cannot match the input to any regular expression, then yylex() should return a -1 to the parser, using a flex rule as follows:\n\n. { return -1; }\n\n\n(Note the near-invisible period in the first column, which matches any input symbol except \\n)\n\n(NOTE: This rule should be the last rule to appear in your flex file, because the order of the rules determines priority: a token is matched by Flex using the first possible rule in the flex file.)\n\nA return value of -1 by the lexer indicates a tokenization error, and the Bison parser handles it automatically by calling yyerror(char *) (ideally defined by you); otherwise, if the input encounters a parsing error, the parser, again, calls yyerror(char *).\n\nAlso, if you want to display the erroneous piece of code when an error is encountered, you'd have to have a way to access the related source code given the defective token, which means the approach of reading the input entirely followed by parsing would not work at all, unless you store associated source code with each token while tokenizing, essentially making a memory behemoth of a compiler.",