CircuitPython - a Python implementation for teaching coding with microcontrollers https://git.suspectdevices.com/suspect-devices/circuitpython/atom?h=5.0.0 2020-01-08T22:36:43+00:00 2020-01-08T22:36:43+00:00 Jeff Epler jepler@gmail.com 2020-01-08T22:36:43+00:00 urn:sha1:f1c2dee1c05227d63e3833cb10d6fe99b9bd5092 2020-01-08T15:42:44+00:00 Jeff Epler jepler@gmail.com 2020-01-08T15:42:44+00:00 urn:sha1:5baaac55ce2b9fb89c1f395e92f67abb5276ce6f In this unusual case, (len + 1) is zero, the allocation in vstr_init succeeds (allocating 1 byte), and then the caller is likely to erroneously access outside the allocated region, for instance with a memset(). This could be triggered with os.urandom(-1) after it was converted to use mp_obj_new_bytes_of_zeros. 2017-09-21T10:29:41+00:00 Damien George damien.p.george@gmail.com 2017-08-31T22:05:24+00:00 urn:sha1:ede8a0235b47333aaaa8d03a3b9e20b014be3808 Current users of fixed vstr buffers (building file paths) assume that there is no overflow and do not check for overflow after building the vstr. This has the potential to lead to NULL pointer dereferences (when vstr_null_terminated_str returns NULL because it can't allocate RAM for the terminating byte) and stat'ing and loading invalid path names (due to the path being truncated). The safest and simplest thing to do in these cases is just raise an exception if a write goes beyond the end of a fixed vstr buffer, which is what this patch does. It also simplifies the vstr code. 2017-07-31T08:35:40+00:00 Alexander Steffen devel.20.webmeister@spamgourmet.com 2017-06-30T07:22:17+00:00 urn:sha1:55f33240f3d7051d4213629e92437a36f1fac50e There were several different spellings of MicroPython present in comments, when there should be only one. 2017-07-07T01:32:22+00:00 Tom Collins tom.collins@digi.com 2017-06-30T23:23:29+00:00 urn:sha1:145796f037715e180b441b38c1ec1ba45ff77797 2016-10-14T05:46:34+00:00 Damien George damien.p.george@gmail.com 2016-10-14T05:46:34+00:00 urn:sha1:824f5c5a32d740acad50d23b7ab1d69660dcf3ad Now there is just one function to allocate a new vstr, namely vstr_new (in addition to vstr_init etc). The caller of this function should know what initial size to allocate for the buffer, or at least have some policy or config option, instead of leaving it to a default (as it was before). 2016-09-19T02:28:55+00:00 Damien George damien.p.george@gmail.com 2016-09-19T01:17:02+00:00 urn:sha1:5da0d29d3cefa6a3cac52e0db96e9ede820d6a51 The vstr.had_error flag was a relic from the very early days which assumed that the malloc functions (eg m_new, m_renew) returned NULL if they failed to allocate. But that's no longer the case: these functions will raise an exception if they fail. Since it was impossible for had_error to be set, this patch introduces no change in behaviour. An alternative option would be to change the malloc calls to the _maybe variants, which return NULL instead of raising, but then a lot of code will need to explicitly check if the vstr had an error and raise if it did. The code-size savings for this patch are, in bytes: bare-arm:188, minimal:456, unix(NDEBUG,x86-64):368, stmhal:228, esp8266:360. 2016-05-09T21:56:51+00:00 Paul Sokolovsky pfalcon@users.sourceforge.net 2016-05-09T21:56:51+00:00 urn:sha1:6f34e138f1ecf197212dc1392cbffeea41b44a93 Effect measured on esp8266 port: Before: >>> pystone_lowmem.main(10000) Pystone(1.2) time for 10000 passes = 44214 ms This machine benchmarks at 226 pystones/second >>> pystone_lowmem.main(10000) Pystone(1.2) time for 10000 passes = 44246 ms This machine benchmarks at 226 pystones/second After: >>> pystone_lowmem.main(10000) Pystone(1.2) time for 10000 passes = 44343ms This machine benchmarks at 225 pystones/second >>> pystone_lowmem.main(10000) Pystone(1.2) time for 10000 passes = 44376ms This machine benchmarks at 225 pystones/second 2016-05-09T19:39:57+00:00 Paul Sokolovsky pfalcon@users.sourceforge.net 2016-05-09T19:39:57+00:00 urn:sha1:a1f2245a81cb9e99cc620797e54a003bf0907b16 vstr_null_terminated_str is almost certainly a vstr finalization operation, so it should add the requested NUL byte, and not try to pre-allocate more. The previous implementation could actually allocate double of the buffer size. 2015-07-06T16:29:27+00:00 Dave Hylands dhylands@gmail.com 2015-05-18T20:25:36+00:00 urn:sha1:9f76dcd682389d4f3aee847da723aad17a3e45ad I checked the entire codebase, and every place that vstr_init_len was called, there was a call to mp_obj_new_str_from_vstr after it. mp_obj_new_str_from_vstr always tries to reallocate a new buffer 1 byte larger than the original to store the terminating null character. In many cases, if we allocated the initial buffer to be 1 byte longer, we can prevent this extra allocation, and just reuse the originally allocated buffer. Asking to read 256 bytes and only getting 100 will still cause the extra allocation, but if you ask to read 256 and get 256 then the extra allocation will be optimized away. Yes - the reallocation is optimized in the heap to try and reuse the buffer if it can, but it takes quite a few cycles to figure this out. Note by Damien: vstr_init_len should now be considered as a string-init convenience function and used only when creating null-terminated objects.