suspect-devices/circuitpython/py/lexer.c, branch 3.0.0-alpha.0CircuitPython - a Python implementation for teaching coding with microcontrollershttps://git.suspectdevices.com/suspect-devices/circuitpython/atom?h=3.0.0-alpha.02017-08-13T12:52:33+00:00all: Raise exceptions via mp_raise_XXX2017-08-13T12:52:33+00:00Javier Candeirajavier@candeira.com2017-08-09T04:40:45+00:00urn:sha1:35a1fea90b2cae9d5cc8e9eab62ba4c67e8786db
- Changed: ValueError, TypeError, NotImplementedError
- OSError invocations unchanged, because the corresponding utility
function takes ints, not strings like the long form invocation.
- OverflowError, IndexError and RuntimeError etc. not changed for now
until we decide whether to add new utility functions.
all: Use the name MicroPython consistently in comments2017-07-31T08:35:40+00:00Alexander Steffendevel.20.webmeister@spamgourmet.com2017-06-30T07:22:17+00:00urn:sha1:55f33240f3d7051d4213629e92437a36f1fac50e
There were several different spellings of MicroPython present in comments,
when there should be only one.
py,extmod: Some casts and minor refactors to quiet compiler warnings.2017-07-07T01:32:22+00:00Tom Collinstom.collins@digi.com2017-06-30T23:23:29+00:00urn:sha1:145796f037715e180b441b38c1ec1ba45ff77797py/lexer: Process CR earlier to allow newlines checks on chr1.2017-05-12T05:14:24+00:00Tom Collinstom.collins@digi.com2017-05-09T20:19:46+00:00urn:sha1:6f56412ec3b93bb93b109f0a01512ecaeebf4f79
Resolves an issue where lexer failed to accept CR after line continuation
character. It also simplifies the code.
py/lexer: Simplify lexer startup by using dummy bytes and next_char().2017-05-09T04:43:23+00:00Tom Collinstom.collins@digi.com2017-05-04T23:31:08+00:00urn:sha1:2998647c4e2c765cba72e390d467e4ba9dcf3262
Now consistently uses the EOL processing ("\r" and "\r\n" convert to "\n")
and EOF processing (ensure "\n" before EOF) provided by next_char().
In particular the lexer can now correctly handle input that starts with CR.
py/lexer: Simplify and reduce code size for operator tokenising.2017-03-28T23:56:52+00:00Damien Georgedamien.p.george@gmail.com2017-03-28T23:55:36+00:00urn:sha1:5010d1958feee84fe49f887ff6e4b1d37d211152
By removing the 'E' code from the operator token encoding mini-language the
tokenising can be simplified. The 'E' code was only used for the !=
operator which is now handled as a special case; the optimisations for the
general case more than make up for the addition of this single, special
case. Furthermore, the . and ... operators can be handled in the same way
as != which reduces the code size a little further.
This simplification also removes a "goto".
Changes in code size for this patch are (measured in bytes):
bare-arm: -48
minimal x86: -64
unix x86-64: -112
unix nanbox: -64
stmhal: -48
cc3200: -48
esp8266: -76
py/lexer: Remove obsolete comment, since lexer can now raise exceptions.2017-03-23T05:40:24+00:00Damien Georgedamien.p.george@gmail.com2017-03-23T05:40:00+00:00urn:sha1:f64a3e296e1c696b1c8a09bac68f7e075f419c8bpy: Allow lexer to raise exceptions during construction.2017-03-14T00:52:05+00:00Damien Georgedamien.p.george@gmail.com2017-03-14T00:16:31+00:00urn:sha1:1831034be13fef5344583c557ff089df31788251
This patch refactors the error handling in the lexer, to simplify it (ie
reduce code size).
A long time ago, when the lexer/parser/compiler were first written, the
lexer and parser were designed so they didn't use exceptions (ie nlr) to
report errors but rather returned an error code. Over time that has
gradually changed, the parser in particular has more and more ways of
raising exceptions. Also, the lexer never really handled all errors without
raising, eg there were some memory errors which could raise an exception
(and in these rare cases one would get a fatal nlr-not-handled fault).
This patch accepts the fact that the lexer can raise exceptions in some
cases and allows it to raise exceptions to handle all its errors, which are
for the most part just out-of-memory errors during construction of the
lexer. This makes the lexer a bit simpler, and also the persistent code
stuff is simplified.
What this means for users of the lexer is that calls to it must be wrapped
in a nlr handler. But all uses of the lexer already have such an nlr
handler for the parser (and compiler) so that doesn't put any extra burden
on the callers.
py/lexer: Convert mp_uint_t to size_t where appropriate.2017-02-17T01:44:24+00:00Damien Georgedamien.p.george@gmail.com2017-02-17T01:44:24+00:00urn:sha1:5124a940670dbb5c07f6681070b3d1580c71d697py: Do adjacent str/bytes literal concatenation in lexer, not compiler.2017-02-17T01:12:40+00:00Damien Georgedamien.p.george@gmail.com2017-02-17T01:12:40+00:00urn:sha1:534b7c368dc2af7720f3aaed0c936ef46d773957
It's much more efficient in RAM and code size to do implicit literal string
concatenation in the lexer, as opposed to the compiler.
RAM usage is reduced because the concatenation can be done right away in the
tokeniser by just accumulating the string/bytes literals into the lexer's
vstr. Prior to this patch adjacent strings/bytes would create a parse tree
(one node per string/bytes) and then in the compiler a whole new chunk of
memory was allocated to store the concatenated string, which used more than
double the memory compared to just accumulating in the lexer.
This patch also significantly reduces code size:
bare-arm: -204
minimal: -204
unix x64: -328
stmhal: -208
esp8266: -284
cc3200: -224