Lexical Analysis, Syntax Analysis, and Semantic Analysis
As a qualified programmer, especially working in a low-level world of compilers, you have to understand what are lexical, syntax and semantic analysis and their differences. These are the very first steps of the compilation process. For a beginner, it might be a bit confusing about their conceptual meanings. So let's talk something about it.
Before everything, please read the answer from ChatGPT3.5(2023.08.01):
Lexical analysis, syntax analysis (parsing), and semantic analysis are fundamental stages in the process of compiling or interpreting programming languages. These stages help in understanding the structure and meaning of the source code and transforming it into an intermediate representation or machine code. Let's explain each of them:
- Lexical Analysis 词法分析: Lexical analysis is the first phase of the compilation process. It deals with breaking the source code into a sequence of tokens. Tokens are the smallest meaningful units in a programming language, such as keywords, identifiers, literals, operators, and punctuation symbols. The process of lexical analysis is performed by a component called a "lexer" or "scanner."
The lexer scans the entire source code and identifies each token by matching the input characters against a set of regular expressions or rules. It discards any unnecessary white spaces, comments, and formatting characters that do not impact the program's meaning. The output of lexical analysis is a stream of tokens that serves as input for the next phase, which is syntax analysis.
For example, consider the statement: int num = 42;
After lexical analysis, the tokens might be: int, num, =, 42, and ;.
- Syntax Analysis (Parsing) 语法分析（解析）: Syntax analysis, also known as parsing, is the second phase of the compilation process. It deals with analyzing the structure of the source code according to the formal grammar rules of the programming language. The purpose of syntax analysis is to ensure that the source code conforms to the language's syntax rules and to construct a hierarchical representation of the program, such as an Abstract Syntax Tree (AST) or a Parse Tree.
The parser takes the stream of tokens generated by the lexer as input and checks whether the sequence of tokens forms a valid statement or program according to the grammar rules. If the input adheres to the grammar, the parser constructs the hierarchical representation. If the input contains syntax errors, the parser reports them to the user.
For example, consider the statement: int num = 42;
The syntax analysis will confirm that the tokens can be arranged into a valid declaration statement: int -> identifier -> = -> literal -> ;
- Semantic Analysis 语义分析: Semantic analysis is the third phase of the compilation process, following lexical and syntax analysis. Its primary objective is to ensure the correctness and meaningfulness of the source code by examining the program's meaning and behavior. Unlike syntax analysis, which checks only the structure of the program, semantic analysis verifies the program's logical correctness.
The semantic analysis phase involves tasks such as:
- Type checking: Ensuring that the types of operands and expressions are compatible and consistent within the program.
- Scope resolution: Determining the validity and visibility of variables and other symbols within their respective scopes.
- Function overloading: Handling multiple functions with the same name but different parameters.
- Detecting semantic errors: Identifying issues that cannot be detected during lexical and syntax analysis but can lead to runtime errors.
For example, consider the statement: int num = 42;
Semantic analysis checks that the variable num is declared before being used, the variable has a valid type (int), and the literal 42 is compatible with the type int.
Once the source code passes the lexical, syntax, and semantic analysis phases, the compiler or interpreter moves on to the subsequent stages, such as intermediate code generation, optimization, and code generation (in the case of compilers) or executing the interpreted code directly (in the case of interpreters).
Here we are. I guess after reading the above-detailed explanation, you should have a much clearer picture of how they work. The only things you need to take away are:
- lexical analysis helps you to split the code into a bunch of tokens;
- syntax analysis helps you to check the grammar rules of the code;
- semantic analysis helps you to check the logical meaning of the code.
Easy to understand and remember, right?