CircuitPython - a Python implementation for teaching coding with microcontrollers https://git.suspectdevices.com/suspect-devices/circuitpython/atom?h=5.1.x 2019-12-02T20:49:23+00:00 2019-12-02T20:49:23+00:00 Jeff Epler jepler@gmail.com 2019-12-02T20:49:23+00:00 urn:sha1:1a0dcb5caaaa2d9f0dc25d87735b76f1bf5f2c24 If a translation only has unicode code points 255 and below, the "values" array can be 8 bits instead of 16 bits. This reclaims some code size, e.g., in a local build, trinket_m0 / en_US reclaimed 112 bytes and de_DE reclaimed 104 bytes. However, languages like zh_Latn_pinyin, which use code points above 255, did not benefit. 2019-12-02T16:18:48+00:00 Jeff Epler jepler@gmail.com 2019-12-02T16:18:48+00:00 urn:sha1:879e1041c9bb0067bc06337870f2515922469cf0 2019-12-02T15:46:46+00:00 Jeff Epler jepler@gmail.com 2019-12-02T14:41:03+00:00 urn:sha1:e06a3bbceb548432969bfcfa1076dc4363478151 By treating each unicode code-point as a single entity for huffman compression, the overall compression rate can be somewhat improved without changing the algorithm. On the decompression side, when compressed values above 127 are encountered, they need to be converted from a 16-bit Unicode code point into a UTF-8 byte sequence. Doing this returns approximately 1.5kB of flash storage with the zh_Latn_pinyin translation. (292 -> 1768 bytes remaining in my build of trinket_m0) Other "more ASCII" translations benefit less, and in fact zh_Latn_pinyin is no longer the most constrained translation! (de_DE 1156 -> 1384 bytes free in flash, I didn't check others before pushing for CI) English is slightly pessimized, 2840 -> 2788 bytes, probably mostly because the "values" array was changed from uint8_t to uint16_t, which is strictly not required for an all-ASCII translation. This could probably be avoided in this case, but as English is not the most constrained translation it doesn't really matter. Testing performed: built for feather nRF52840 express and trinket m0 in English and zh_Latn_pinyin; ran and verified the localized messages such as Àn xià rènhé jiàn jìnrù REPL. Shǐyòng CTRL-D chóngxīn jiāzài. and Press any key to enter the REPL. Use CTRL-D to reload. were properly displayed. 2019-08-06T12:39:09+00:00 Jeff Epler jepler@gmail.com 2019-08-06T12:38:49+00:00 urn:sha1:c4f3a02b3bd3709fd0e07ed07b372b3d5ec3815d It is possible for this routine to expand some inputs, and in fact it does for certan strings in the proposed Korean translation of CircuitPython (#1858). I did not determine what the maximum expansion is -- it's probably modest, like len()/7+2 bytes or something -- so I tried to just make enc[] an adequate over-allocation, and then ensured that all the strings in the proposed ko.po now worked. The worst actual expansion seems to be a string that goes from 65 UTF-8-encoded bytes to 68 compressed bytes (+4.6%). Only a few out of all strings are reported as non-compressed. 2018-11-10T00:41:08+00:00 Scott Shawcroft scott@tannewt.org 2018-11-10T00:41:08+00:00 urn:sha1:355abc835ea75715b4dea38604ef64f8c4eeaa0f 2018-08-17T00:40:57+00:00 Scott Shawcroft scott@tannewt.org 2018-08-16T07:27:01+00:00 urn:sha1:137a30ad75803255615a422c15ebcd35fbd31130 2018-08-17T00:40:57+00:00 Scott Shawcroft scott@tannewt.org 2018-08-16T01:32:37+00:00 urn:sha1:de5a9d72dcdaacdd5048195cd5bab007f4b2baef This saves code space in builds which use link-time optimization. The optimization drops the untranslated strings and replaces them with a compressed_string_t struct. It can then be decompressed to a c string. Builds without LTO work as well but include both untranslated strings and compressed strings. This work could be expanded to include QSTRs and loaded strings if a compress method is added to C. Its tracked in #531. 2018-08-10T23:17:03+00:00 Scott Shawcroft scott@adafruit.com 2018-08-10T23:17:03+00:00 urn:sha1:4513bd6ea3773ebaae70bd2fa0a2b35c46eae2ff Escape table was incorrect 2018-08-09T22:58:45+00:00 Scott Shawcroft scott@tannewt.org 2018-08-09T22:58:45+00:00 urn:sha1:24e53ad5911f42efb6508b4a8dcf4a196a81494a 2018-08-09T20:29:30+00:00 Scott Shawcroft scott@tannewt.org 2018-08-09T01:24:49+00:00 urn:sha1:96ebf5bc3fb455162286b9bc31e763f3c4f1c457 * Fix finding translations with escaped characters. * Add back \r to translations since its needed by screen.