So this is a demonstration of a lexical analyzer.

lexical analysis, parsing, semantic analysis, optimization, and code generation.

This process is called dun, dun, dun, dun lexical analysis.

We're going to see how to implement this in our lexical analyzer, but recall that our lexical analyzer was just based on regular expressions.

However, quite a few other things make use of lexical analysis.

It's the smallest unit of the output of a lexical analysis.

Lexical here has the same roots and lexicon like a dictionary.

This function call tells our lexical analysis library that we want to use all of the token definitions above to make a lexical analyzer, and break up strings.

So we just learned how to do that JavaScript token in lexical analysis.

Now, recall that the output of a lexical analyzer is a list of tokens.

Component technologies like word sense disambiguation deciding if we're talking about a rodent or a computer mouse when people talk about mouses in a search.

You're already familiar with all of the key concepts in making a lexical analysis library.

And we did Lexing or Lexical Analysis to break it up into a list of tokens.

For example, once I put these 3 rules together, they become a lexer, or lexical analyzer.

And then we'll have the terminal categories or lexical categories, categories that define classes of words.

By looking at this output, we can notice mistakes in our lexical analyzer and help fix them.

If you're making a lexical analyzer for HTML or JavaScript, ordering your token definitions is of prime importance.

And that task uses exactly the same sort of lexical analysis string matching that we've talked about here.

A lexical analyzer or lexer this is a term of art is just a collection of such tokens.

And in fact, suppose we passed to this lexical analyzer the input string 33 is less than 55.

And it turns out that there's effectively a standard, this notion of making lexical analyzers or making parsers.

It's telling Python where to find our lexical analyzer software or libraries that we're going to build upon.

These other numbers, after lexical analyzer token STRlNG, with token value, ' This ' are the line and character number information.

In JavaScript, we use curly braces and closing curly braces to denote this sort of lexical or syntactic scope.

The original tool for this was called Lex, a lexical analyzer generator, and the name in there, generator, is important.

Just like regular expressions were called RE to save space, a lexical analyzer is just called lex to save space.

It was called a lexical analyzer generator, but it was proprietary software at the time and it only worked for C.

That's exactly the same format that are used in the lexical analyzer and parser generator tools for all of the other languages.

There are a large number of great uses of regular expressions and lexical analysis beyond just building web browsers or writing language interpreters.

The goal of lexical analysis, then, is to divide the program text into its words, or what we call in compiler speak, the tokens.

So now I'm going to show you how to make a lexical analyzer which, recall is just a bunch of token definitions put together.

With the lexing and parsing, the lexical analysis and syntactical analysis that we'll cover in this class, do they carry over to parsing real world languages

I would like you to define a variable called (webpage) that holds a string that causes our lexical analyzer to produce exactly these 6 bits of output

That sort of readability metric can be easily computed using exactly the sort of break a string up into words kind of lexical analysis that we've been focusing on.

I hinted at this in the beginning of Unit 1, but things like phone numbers and credit card numbers are recognized and processed using regular expressions essentially using lexical analysis.

We're going to be focusing on lexical analysis, a process whereby we break down a string, like a sentence or an utterance or a webpage, into a list of tokens.

There was a very interesting and wide reaching question on the forums about the uses of lexical analysis and regular expressions and this general sort of string processing outside of what we're doing in this class.

We are going to start with your average web page, a string of HTML with some embedded Java script, and we wanted to break it down into tokens or words, and that process is called lexical analysis.

Well it consists five phases lexical analysis and parsing, which together take care of the syntactic aspects of the language, semantic analysis, which, of course, takes care of more semantic aspects, things like types and scope rules.

Both Lex and YACC were proprietary software, so the GNU Project made free versions of them called Flex, a fast lexical analyzer generator, and Bison, Bison being a pun on YACC, which were then very widely used.

Again, in real world industrial software development for a web browser, this sort of error recovery when you're doing lexical analysis or syntactic analysis is of critical importance because the vast majority of web pages are not standards compliant.

In this unit, we're going to put those concepts together to make a lexical analyzer a program that reads in a web page or a bit of JavaScript and breaks it down into words, just like I might break and English sentence down into words.

The idea was you would just write out some regular expressions, and this tool would automatically make the finite state machine, the lexical analyzer, for you, saving you a lot of time and grungy implementation work of converting regular expressions down to finite state machines.

To do this, we use 2 key techniques lexical analysis, which broke the string down into a stream of tokens, and syntactic analysis or parsing, which takes a string of tokens and checks to see if they adhere to, if they conform to a context free grammar.

And then the final thing I would mention is there are a lot of very high level readability metrics, both for software and also for natural language text, that are based on the sort of lexing or lexical analysis or token definition principles that we've come to know.