CircuitPython - a Python implementation for teaching coding with microcontrollers https://git.suspectdevices.com/suspect-devices/circuitpython/atom?h=main 2021-03-15T13:57:36+00:00 2021-03-15T13:57:36+00:00 microDev 70126934+microDev1@users.noreply.github.com 2021-03-15T13:57:36+00:00 urn:sha1:a52eb88031620a81521b937f2a0651dbac2bb350 2020-09-21T15:03:07+00:00 Jeff Epler jepler@gmail.com 2020-09-21T15:02:27+00:00 urn:sha1:0318eb359fbeb91c6b37ed2050e57711ec2740bc Discord user Folknology encountered a problem building with Python 3.6.9, `TypeError: ord() expected a character, but string of length 0 found`. I was able to reproduce the problem using Python3.5*, and discovered that the meaning of the regular expression `"|."` had changed in 3.7. Before, ``` >>> [m.group(0) for m in re.finditer("|.", "hello")] ['', '', '', '', '', ''] ``` After: ``` >>> [m.group(0) for m in re.finditer("|.", "hello")] ['', 'h', '', 'e', '', 'l', '', 'l', '', 'o', ''] ``` Check if `words` is empty and if so use `"."` as the regular expression instead. This gives the same result on both versions: ``` ['h', 'e', 'l', 'l', 'o'] ``` and fixes the generation of the huffman dictionary. Folknology verified that this fix worked for them. * I could easily install 3.5 but not 3.6. 3.5 reproduced the same problem 2020-09-19T15:16:13+00:00 Jeff Epler jepler@gmail.com 2020-09-19T15:16:13+00:00 urn:sha1:bfbbbd6c5ce58ba1bcb57323566209902e1000b0 This construct (which I added without sufficient testing, apparently) is only supported in Python 3.7 and newer. Make it optional so that this script works on other Python versions. This means that if you have a system with non-UTF-8 encoding you will need to use Python 3.7. In particular, this affects a problem building circuitpython in github's ubuntu-18.04 virtual environment when Python 3.7 is not explicitly installed. cookie-cuttered libraries call for Python 3.6: ``` - name: Set up Python 3.6 uses: actions/setup-python@v1 with: python-version: 3.6 ``` Since CircuitPython's own build calls for 3.8, this problem was not detected. This problem was also encountered by discord user mdroberts1243. The failure I encountered was here: https://github.com/jepler/Jepler_CircuitPython_udecimal/runs/1138045020?check_suite_focus=true .. while my step of "clone and build circuitpython unix port" is unusual, I think the same problem would have affected "build assets" if that step had been reached. 2020-09-16T13:28:15+00:00 Jeff Epler jepler@gmail.com 2020-09-16T12:58:55+00:00 urn:sha1:a8e98cda83119b8716533ce24af6e3d691f7d2ca 2020-09-13T16:50:01+00:00 Taku Fukada naninunenor@gmail.com 2020-09-13T16:25:13+00:00 urn:sha1:d18d79ac4709364c2371e2eaefd8f96bb20802e7 2020-09-13T00:43:08+00:00 Jeff Epler jepler@gmail.com 2020-09-13T00:39:35+00:00 urn:sha1:15964a4750d9eb1a34317c166d9bde5f40c9878d Most users and the CI system are running in configurations where Python configures stdout and stderr in UTF-8 mode. However, Windows is different, setting values like CP1252. This led to a build failure on Windows, because makeqstrdata printed Unicode strings to its stdout, expecting them to be encoded as UTF-8. This script is writing (stdout) to a compiler input file and potentially printing messages (stderr) to a log or console. Explicitly configure stdout to use utf-8 to get consistent behavior on all platforms, and configure stderr so that if any log/diagnostic messages are printed that cannot be displayed correctly, they are still displayed instead of creating an error while trying to print the diagnostic information. I considered setting the encodings both to ascii, but this would just be occasionally inconvenient to developers like me who want to show diagnostic info on stderr and in comments while working with the compression code. Closes: #3408 2020-09-12T15:10:45+00:00 Jeff Epler jepler@gmail.com 2020-09-12T15:10:18+00:00 urn:sha1:40ab5c6b21ff93ad11ca51dc66d3613dcd77e5ef Massive savings. Thanks so much @ciscorn for providing the initial code for choosing the dictionary. This adds a bit of time to the build, both to find the dictionary but also because (for reasons I don't fully understand), the binary search in the compress() function no longer worked and had to be replaced with a linear search. I think this is because the intended invariant is that for codebook entries that encode to the same number of bits, the entries are ordered in ascending value. However, I mis-placed the transition from "words" to "byte/char values" so the codebook entries for words are in word-order rather than their code order. Because this price is only paid at build time, I didn't care to determine exactly where the correct fix was. I also commented out a line to produce the "estimated total memory size" -- at least on the unix build with TRANSLATION=ja, this led to a build time KeyError trying to compute the codebook size for all the strings. I think this occurs because some single unicode code point ('ァ') is no longer present as itself in the compressed strings, due to always being replaced by a word. As promised, this seems to save hundreds of bytes in the German translation on the trinket m0. Testing performed: - built trinket_m0 in several languages - built and ran unix port in several languages (en, de_DE, ja) and ran simple error-producing codes like ./micropython -c '1/0' 2020-09-09T00:07:53+00:00 Jeff Epler jepler@gmail.com 2020-09-09T00:07:53+00:00 urn:sha1:bdb07adfccaf25576a8f1e074db7f8b7e48890b2 Now this gets filled in with values e.g., 128 (0x80) and 159 (0x9f). 2020-09-03T00:09:23+00:00 Jeff Epler jepler@gmail.com 2020-09-03T00:09:23+00:00 urn:sha1:cbfd38d1ce8839e11e828b0e8742d5d983446313 2020-09-02T20:52:02+00:00 Jeff Epler jepler@gmail.com 2020-09-02T20:52:02+00:00 urn:sha1:c34cb82ecb26ac2d5d85394ec535928194cd9e5e The previous range was unintentionally big and overlaps some characters we'd like to use (and also 0xa0, which we don't intentionally use)