CircuitPython - a Python implementation for teaching coding with microcontrollers https://git.suspectdevices.com/suspect-devices/circuitpython/atom?h=6.0.0-alpha.3 2020-08-18T14:21:14+00:00 2020-08-18T14:21:14+00:00 Taku Fukada naninunenor@gmail.com 2020-08-18T14:21:14+00:00 urn:sha1:79a3796b1cd74b376e091e63c1b3a2426ec3187c 2020-08-17T01:50:48+00:00 Jeff Epler jepler@gmail.com 2020-08-17T01:38:05+00:00 urn:sha1:08ed09acc6c73bae5475735e6e39b10ccd951d36 This check as implemented is misleading, because it compares the compressed size in bytes (including the length indication) with the source string length in Unicode code points. For English this is approximately fair, but for Japanese this is quite unfair and produces an excess of "increased length" messages. This message might have existed for one of two reasons: * to alert to an improperly function huffman compression * to call attention to a need for a "string is stored uncompressed" case We know by now that the huffman compression is functioning as designed and effective in general. Just to be on the safe side, I did some back-of-the-envelope estimates. I considered these three replacements for "the true source string size, in bytes": + decompressed_len_utf8 = len(decompressed.encode('utf-8')) + decompressed_len_utf16 = len(decompressed.encode('utf-16be')) + decompressed_len_bitsize = ((1+len(decompressed)) * math.ceil(math.log(1+len(values), 2)) + 7) // 8 The third counts how many bits each character requires (fewer than 128 characters in the source character set = 7, fewer than 256 = 8, fewer than 512 = 9, etc, adding a string-terminating value) and is in some way representative of the best way we would be able to store "uncompressed strings". The Japanese translation (largest as of writing) has just a few strings which increase by this metric. However, the amount of loss due to expansion in those cases is outweighed by the cost of adding 1 bit per string to indicate whether it's compressed or not. For instance, in the BOARD=trinket_m0 TRANSLATION=ja build the loss is 47 bytes over 300 strings. Adding 1 bit to each of 300 strings will cost about 37 bytes, leaving just 5 Thumb instructions to implement the code to check and decode "uncompressed" strings in order to break even. 2020-05-28T16:30:46+00:00 Jeff Epler jepler@gmail.com 2020-05-28T16:29:28+00:00 urn:sha1:d0f9b5901e37a751a0bf5770c2f26a5c6c039dd4 2020-05-28T13:36:08+00:00 Jeff Epler jepler@gmail.com 2020-05-28T12:40:56+00:00 urn:sha1:fe3e8d1589e54de999cccc775f269a39443c82d6 Length was stored as a 16-bit number always. Most translations have a max length far less. For example, US English translation lengths always fit in just 8 bits. probably all languages fit in 9 bits. This also has the side effect of reducing the alignment of compressed_string_t from 2 bytes to 1. testing performed: ran in german and english on pyruler, printed messages looked right. Firmware size, en_US Before: 3044 bytes free in flash After: 3408 bytes free in flash Firmware size, de_DE (with #2967 merged to restore translations) Before: 1236 bytes free in flash After: 1600 bytes free in flash 2019-12-02T20:49:23+00:00 Jeff Epler jepler@gmail.com 2019-12-02T20:49:23+00:00 urn:sha1:1a0dcb5caaaa2d9f0dc25d87735b76f1bf5f2c24 If a translation only has unicode code points 255 and below, the "values" array can be 8 bits instead of 16 bits. This reclaims some code size, e.g., in a local build, trinket_m0 / en_US reclaimed 112 bytes and de_DE reclaimed 104 bytes. However, languages like zh_Latn_pinyin, which use code points above 255, did not benefit. 2019-12-02T16:18:48+00:00 Jeff Epler jepler@gmail.com 2019-12-02T16:18:48+00:00 urn:sha1:879e1041c9bb0067bc06337870f2515922469cf0 2019-12-02T15:46:46+00:00 Jeff Epler jepler@gmail.com 2019-12-02T14:41:03+00:00 urn:sha1:e06a3bbceb548432969bfcfa1076dc4363478151 By treating each unicode code-point as a single entity for huffman compression, the overall compression rate can be somewhat improved without changing the algorithm. On the decompression side, when compressed values above 127 are encountered, they need to be converted from a 16-bit Unicode code point into a UTF-8 byte sequence. Doing this returns approximately 1.5kB of flash storage with the zh_Latn_pinyin translation. (292 -> 1768 bytes remaining in my build of trinket_m0) Other "more ASCII" translations benefit less, and in fact zh_Latn_pinyin is no longer the most constrained translation! (de_DE 1156 -> 1384 bytes free in flash, I didn't check others before pushing for CI) English is slightly pessimized, 2840 -> 2788 bytes, probably mostly because the "values" array was changed from uint8_t to uint16_t, which is strictly not required for an all-ASCII translation. This could probably be avoided in this case, but as English is not the most constrained translation it doesn't really matter. Testing performed: built for feather nRF52840 express and trinket m0 in English and zh_Latn_pinyin; ran and verified the localized messages such as Àn xià rènhé jiàn jìnrù REPL. Shǐyòng CTRL-D chóngxīn jiāzài. and Press any key to enter the REPL. Use CTRL-D to reload. were properly displayed. 2019-08-06T12:39:09+00:00 Jeff Epler jepler@gmail.com 2019-08-06T12:38:49+00:00 urn:sha1:c4f3a02b3bd3709fd0e07ed07b372b3d5ec3815d It is possible for this routine to expand some inputs, and in fact it does for certan strings in the proposed Korean translation of CircuitPython (#1858). I did not determine what the maximum expansion is -- it's probably modest, like len()/7+2 bytes or something -- so I tried to just make enc[] an adequate over-allocation, and then ensured that all the strings in the proposed ko.po now worked. The worst actual expansion seems to be a string that goes from 65 UTF-8-encoded bytes to 68 compressed bytes (+4.6%). Only a few out of all strings are reported as non-compressed. 2018-11-10T00:41:08+00:00 Scott Shawcroft scott@tannewt.org 2018-11-10T00:41:08+00:00 urn:sha1:355abc835ea75715b4dea38604ef64f8c4eeaa0f 2018-08-17T00:40:57+00:00 Scott Shawcroft scott@tannewt.org 2018-08-16T07:27:01+00:00 urn:sha1:137a30ad75803255615a422c15ebcd35fbd31130