suspect-devices/circuitpython/py/lexer.h, branch 5.3.1

f-strings: Make optional, defaulting to !CIRCUITPY_MINIMAL_BUILD

2020-03-09T14:03:25+00:00

This should reclaim *most* code space added to handle f-strings. However, there may be some small code growth as parse_string_literal takes a new parameter (which will always be 0, so hopefully the optimizer eliminates it)

py: Implement partial PEP-498 (f-string) support

2020-03-09T13:16:07+00:00

This implements (most of) the PEP-498 spec for f-strings, with two exceptions: - raw f-strings (`fr` or `rf` prefixes) raise `NotImplementedError` - one special corner case does not function as specified in the PEP (more on that in a moment) This is implemented in the core as a syntax translation, brute-forcing all f-strings to run through `String.format`. For example, the statement `x='world'; print(f'hello {x}')` gets translated *at a syntax level* (injected into the lexer) to `x='world'; print('hello {}'.format(x))`. While this may lead to weird column results in tracebacks, it seemed like the fastest, most efficient, and *likely* most RAM-friendly option, despite being implemented under the hood with a completely separate `vstr_t`. Since [string concatenation of adjacent literals is implemented in the lexer](https://github.com/micropython/micropython/commit/534b7c368dc2af7720f3aaed0c936ef46d773957), two side effects emerge: - All strings with at least one f-string portion are concatenated into a single literal which *must* be run through `String.format()` wholesale, and: - Concatenation of a raw string with interpolation characters with an f-string will cause `IndexError`/`KeyError`, which is both different from CPython *and* different from the corner case mentioned in the PEP (which gave an example of the following:) ```python x = 10 y = 'hi' assert ('a' 'b' f'{x}' '{c}' f'str<{y:^4}>' 'd' 'e') == 'ab10{c}str< hi >de' ``` The above-linked commit detailed a pretty solid case for leaving string concatenation in the lexer rather than putting it in the parser, and undoing that decision would likely be disproportionately costly on resources for the sake of a probably-low-impact corner case. An alternative to become complaint with this corner case of the PEP would be to revert to string concatenation in the parser *only when an f-string is part of concatenation*, though I've done no investigation on the difficulty or costs of doing this. A decent set of tests is included. I've manually tested this on the `unix` port on Linux and on a Feather M4 Express (`atmel-samd`) and things seem sane.

all: Use the name MicroPython consistently in comments

2017-07-31T08:35:40+00:00

There were several different spellings of MicroPython present in comments, when there should be only one.

all: Unify header guard usage.

2017-07-18T01:57:39+00:00

The code conventions suggest using header guards, but do not define how those should look like and instead point to existing files. However, not all existing files follow the same scheme, sometimes omitting header guards altogether, sometimes using non-standard names, making it easy to accidentally pick a "wrong" example. This commit ensures that all header files of the MicroPython project (that were not simply copied from somewhere else) follow the same pattern, that was already present in the majority of files, especially in the py folder. The rules are as follows. Naming convention: * start with the words MICROPY_INCLUDED * contain the full path to the file * replace special characters with _ In addition, there are no empty lines before #ifndef, between #ifndef and one empty line before #endif. #endif is followed by a comment containing the name of the guard macro. py/grammar.h cannot use header guards by design, since it has to be included multiple times in a single C file. Several other files also do not need header guards as they are only used internally and guaranteed to be included only once: * MICROPY_MPHALPORT_H * mpconfigboard.h * mpconfigport.h * mpthreadport.h * pin_defs_*.h * qstrdefs*.h

py/lexer: Convert mp_uint_t to size_t where appropriate.

2017-02-17T01:44:24+00:00

py/lexer: Simplify handling of line-continuation error.

2017-02-17T00:30:14+00:00

Previous to this patch there was an explicit check for errors with line continuation (where backslash was not immediately followed by a newline). But this check is not necessary: if there is an error then the remaining logic of the tokeniser will reject the backslash and correctly produce a syntax error.

py/lexer: Use strcmp to make keyword searching more efficient.

2017-02-17T00:10:35+00:00

Since the table of keywords is sorted, we can use strcmp to do the search and stop part way through the search if the comparison is less-than. Because all tokens that are names are subject to this search, this optimisation will improve the overall speed of the lexer when processing a script. The change also decreases code size by a little bit because we now use strcmp instead of the custom str_strn_equal function.

py/lexer: Permanently disable the mp_lexer_show_token function.

2016-12-21T23:49:54+00:00

The lexer is very mature and this debug function is no longer used. If it's really needed one can uncomment it and recompile.

py/lexer: Make lexer use an mp_reader as its source.

2016-11-16T07:35:01+00:00

py: add async/await/async for/async with syntax

2016-04-13T14:26:38+00:00

They are sugar for marking function as generator, "yield from" and pep492 python "semantically equivalents" respectively. @dpgeorge was the original author of this patch, but @pohmelie made changes to implement `async for` and `async with`.