CircuitPython - a Python implementation for teaching coding with microcontrollers https://git.suspectdevices.com/suspect-devices/circuitpython/atom?h=v1.9.1 2017-05-12T05:14:24+00:00 2017-05-12T05:14:24+00:00 Tom Collins tom.collins@digi.com 2017-05-09T20:19:46+00:00 urn:sha1:6f56412ec3b93bb93b109f0a01512ecaeebf4f79 Resolves an issue where lexer failed to accept CR after line continuation character. It also simplifies the code. 2017-05-09T04:43:23+00:00 Tom Collins tom.collins@digi.com 2017-05-04T23:31:08+00:00 urn:sha1:2998647c4e2c765cba72e390d467e4ba9dcf3262 Now consistently uses the EOL processing ("\r" and "\r\n" convert to "\n") and EOF processing (ensure "\n" before EOF) provided by next_char(). In particular the lexer can now correctly handle input that starts with CR. 2017-03-28T23:56:52+00:00 Damien George damien.p.george@gmail.com 2017-03-28T23:55:36+00:00 urn:sha1:5010d1958feee84fe49f887ff6e4b1d37d211152 By removing the 'E' code from the operator token encoding mini-language the tokenising can be simplified. The 'E' code was only used for the != operator which is now handled as a special case; the optimisations for the general case more than make up for the addition of this single, special case. Furthermore, the . and ... operators can be handled in the same way as != which reduces the code size a little further. This simplification also removes a "goto". Changes in code size for this patch are (measured in bytes): bare-arm: -48 minimal x86: -64 unix x86-64: -112 unix nanbox: -64 stmhal: -48 cc3200: -48 esp8266: -76 2017-03-23T05:40:24+00:00 Damien George damien.p.george@gmail.com 2017-03-23T05:40:00+00:00 urn:sha1:f64a3e296e1c696b1c8a09bac68f7e075f419c8b 2017-03-14T00:52:05+00:00 Damien George damien.p.george@gmail.com 2017-03-14T00:16:31+00:00 urn:sha1:1831034be13fef5344583c557ff089df31788251 This patch refactors the error handling in the lexer, to simplify it (ie reduce code size). A long time ago, when the lexer/parser/compiler were first written, the lexer and parser were designed so they didn't use exceptions (ie nlr) to report errors but rather returned an error code. Over time that has gradually changed, the parser in particular has more and more ways of raising exceptions. Also, the lexer never really handled all errors without raising, eg there were some memory errors which could raise an exception (and in these rare cases one would get a fatal nlr-not-handled fault). This patch accepts the fact that the lexer can raise exceptions in some cases and allows it to raise exceptions to handle all its errors, which are for the most part just out-of-memory errors during construction of the lexer. This makes the lexer a bit simpler, and also the persistent code stuff is simplified. What this means for users of the lexer is that calls to it must be wrapped in a nlr handler. But all uses of the lexer already have such an nlr handler for the parser (and compiler) so that doesn't put any extra burden on the callers. 2017-02-17T01:44:24+00:00 Damien George damien.p.george@gmail.com 2017-02-17T01:44:24+00:00 urn:sha1:5124a940670dbb5c07f6681070b3d1580c71d697 2017-02-17T01:12:40+00:00 Damien George damien.p.george@gmail.com 2017-02-17T01:12:40+00:00 urn:sha1:534b7c368dc2af7720f3aaed0c936ef46d773957 It's much more efficient in RAM and code size to do implicit literal string concatenation in the lexer, as opposed to the compiler. RAM usage is reduced because the concatenation can be done right away in the tokeniser by just accumulating the string/bytes literals into the lexer's vstr. Prior to this patch adjacent strings/bytes would create a parse tree (one node per string/bytes) and then in the compiler a whole new chunk of memory was allocated to store the concatenated string, which used more than double the memory compared to just accumulating in the lexer. This patch also significantly reduces code size: bare-arm: -204 minimal: -204 unix x64: -328 stmhal: -208 esp8266: -284 cc3200: -224 2017-02-17T00:30:14+00:00 Damien George damien.p.george@gmail.com 2017-02-17T00:30:14+00:00 urn:sha1:773278ec3030ea9ed809c5a248fde2278ce4b557 Previous to this patch there was an explicit check for errors with line continuation (where backslash was not immediately followed by a newline). But this check is not necessary: if there is an error then the remaining logic of the tokeniser will reject the backslash and correctly produce a syntax error. 2017-02-17T00:10:35+00:00 Damien George damien.p.george@gmail.com 2017-02-17T00:10:35+00:00 urn:sha1:ae436797927c3c9f7ccdc25dd78af3dd279ca7ff Since the table of keywords is sorted, we can use strcmp to do the search and stop part way through the search if the comparison is less-than. Because all tokens that are names are subject to this search, this optimisation will improve the overall speed of the lexer when processing a script. The change also decreases code size by a little bit because we now use strcmp instead of the custom str_strn_equal function. 2017-02-16T23:59:57+00:00 Damien George damien.p.george@gmail.com 2017-02-16T23:59:57+00:00 urn:sha1:a68c75468812859633814e90207faec1a361778e Keywords only needs to be searched for if the token is a MP_TOKEN_NAME, so we can move the seach to the part of the code that does the tokenising for MP_TOKEN_NAME.