diff options
Diffstat (limited to 'docs/reference/packages.rst')
| -rw-r--r-- | docs/reference/packages.rst | 312 |
1 files changed, 312 insertions, 0 deletions
diff --git a/docs/reference/packages.rst b/docs/reference/packages.rst new file mode 100644 index 000000000..8be2461c2 --- /dev/null +++ b/docs/reference/packages.rst @@ -0,0 +1,312 @@ +Distribution packages, package management, and deploying applications +===================================================================== + +Just as the "big" Python, MicroPython supports creation of "third party" +packages, distributing them, and easily installing them in each user's +environment. This chapter discusses how these actions are achieved. +Some familiarity with Python packaging is recommended. + +Overview +-------- + +Steps below represent a high-level workflow when creating and consuming +packages: + +1. Python modules and packages are turned into distribution package + archives, and published at the Python Package Index (PyPI). +2. `upip` package manager can be used to install a distribution package + on a `MicroPython port` with networking capabilities (for example, + on the Unix port). +3. For ports without networking capabilities, an "installation image" + can be prepared on the Unix port, and transferred to a device by + suitable means. +4. For low-memory ports, the installation image can be frozen as the + bytecode into MicroPython executable, thus minimizing the memory + storage overheads. + +The sections below describe this process in details. + +Distribution packages +--------------------- + +Python modules and packages can be packaged into archives suitable for +transfer between systems, storing at the well-known location (PyPI), +and downloading on demand for deployment. These archives are known as +*distribution packages* (to differentiate them from Python packages +(means to organize Python source code)). + +The MicroPython distribution package format is a well-known tar.gz +format, with some adaptations however. The Gzip compressor, used as +an external wrapper for TAR archives, by default uses 32KB dictionary +size, which means that to uncompress a compressed stream, 32KB of +contguous memory needs to be allocated. This requirement may be not +satisfiable on low-memory devices, which may have total memory available +less than that amount, and even if not, a contiguous block like that +may be hard to allocate due to memory fragmentation. To accommodate +these constraints, MicroPython distribution packages use Gzip compression +with the dictionary size of 4K, which should be a suitable compromise +with still achieving some compression while being able to uncompressed +even by the smallest devices. + +Besides the small compression dictionary size, MicroPython distribution +packages also have other optimizations, like removing any files from +the archive which aren't used by the installation process. In particular, +`upip` package manager doesn't execute ``setup.py`` during installation +(see below), and thus that file is not included in the archive. + +At the same time, these optimizations make MicroPython distribution +packages not compatible with `CPython`'s package manager, ``pip``. +This isn't considered a big problem, because: + +1. Packages can be installed with `upip`, and then can be used with + CPython (if they are compatible with it). +2. In the other direction, majority of CPython packages would be + incompatible with MicroPython by various reasons, first of all, + the reliance on features not implemented by MicroPython. + +Summing up, the MicroPython distribution package archives are highly +optimized for MicroPython's target environments, which are highly +resource constrained devices. + + +``upip`` package manager +------------------------ + +MicroPython distribution packages are intended to be installed using +the `upip` package manager. `upip` is a Python application which is +usually distributed (as frozen bytecode) with network-enabled +`MicroPython ports <MicroPython port>`. At the very least, +`upip` is available in the `MicroPython Unix port`. + +On any `MicroPython port` providing `upip`, it can be accessed as +following:: + + import upip + upip.help() + upip.install(package_or_package_list, [path]) + +Where *package_or_package_list* is the name of a distribution +package to install, or a list of such names to install multiple +packages. Optional *path* parameter specifies filesystem +location to install under and defaults to the standard library +location (see below). + +An example of installing a specific package and then using it:: + + >>> import upip + >>> upip.install("micropython-pystone_lowmem") + [...] + >>> import pystone_lowmem + >>> pystone_lowmem.main() + +Note that the name of Python package and the name of distribution +package for it in general don't have to match, and oftentimes they +don't. This is because PyPI provides a central package repository +for all different Python implementations and versions, and thus +distribution package names may need to be namespaced for a particular +implementation. For example, all packages from `micropython-lib` +follow this naming convention: for a Python module or package named +``foo``, the distribution package name is ``micropython-foo``. + +For the ports which run MicroPython executable from the OS command +prompts (like the Unix port), `upip` can be (and indeed, usually is) +run from the command line instead of MicroPython's own REPL. The +commands which corresponds to the example above are:: + + micropython -m upip -h + micropython -m upip install [-p <path>] <packages>... + micropython -m upip install micropython-pystone_lowmem + +[TODO: Describe installation path.] + + +Cross-installing packages +------------------------- + +For `MicroPython ports <MicroPython port>` without native networking +capabilities, the recommend process is "cross-installing" them into a +"directory image" using the `MicroPython Unix port`, and then +transferring this image to a device by suitable means. + +Installing to a directory image involves using ``-p`` switch to `upip`:: + + micropython -m upip install -p install_dir micropython-pystone_lowmem + +After this command, the package content (and contents of every depenency +packages) will be available in the ``install_dir/`` subdirectory. You +would need to transfer contents of this directory (without the +``install_dir/`` prefix) to the device, at the suitable location, where +it can be found by the Python ``import`` statement (see discussion of +the `upip` installation path above). + + +Cross-installing packages with freezing +--------------------------------------- + +For the low-memory `MicroPython ports <MicroPython port>`, the process +described in the previous section does not provide the most efficient +resource usage,because the packages are installed in the source form, +so need to be compiled to the bytecome on each import. This compilation +requires RAM, and the resulting bytecode is also stored in RAM, reducing +its amount available for storing application data. Moreover, the process +above requires presence of the filesystem on a device, and the most +resource-constrained devices may not even have it. + +The bytecode freezing is a process which resolves all the issues +mentioned above: + +* The source code is pre-compiled into bytecode and store as such. +* The bytecode is stored in ROM, not RAM. +* Filesystem is not required for frozen packages. + +Using frozen bytecode requires building the executable (firmware) +for a given `MicroPython port` from the C source code. Consequently, +the process is: + +1. Follow the instructions for a particular port on setting up a + toolchain and building the port. For example, for ESP8266 port, + study instructions in ``ports/esp8266/README.md`` and follow them. + Make sure you can build the port and deploy the resulting + executable/firmware successfully before proceeding to the next steps. +2. Build `MicroPython Unix port` and make sure it is in your PATH and + you can execute ``micropython``. +3. Change to port's directory (e.g. ``ports/esp8266/`` for ESP8266). +4. Run ``make clean-frozen``. This step cleans up any previous + modules which were installed for freezing (consequently, you need + to skip this step to add additional modules, instead of starting + from scratch). +5. Run ``micropython -m upip install -p modules <packages>...`` to + install packages you want to freeze. +6. Run ``make clean``. +7. Run ``make``. + +After this, you should have the executable/firmware with modules as +the bytecode inside, which you can deploy the usual way. + +Few notes: + +1. Step 5 in the sequence above assumes that the distribution package + is available from PyPI. If that is not the case, you would need + to copy Python source files manually to ``modules/`` subdirectory + of the port port directory. (Note that upip does not support + installing from e.g. version control repositories). +2. The firmware for baremetal devices usually has size restrictions, + so adding too many frozen modules may overflow it. Usually, you + would get a linking error if this happens. However, in some cases, + an image may be produced, which is not runnable on a device. Such + cases are in general bugs, and should be reported and further + investigated. If you face such a situation, as an initial step, + you may want to decrease the amount of frozen modules included. + + +Creating distribution packages +------------------------------ + +Distribution packages for MicroPython are created in the same manner +as for CPython or any other Python implementation, see references at +the end of chapter. Setuptools (instead of distutils) should be used, +because distutils do not support dependencies and other features. "Source +distribution" (``sdist``) format is used for packaging. The post-processing +discussed above, (and pre-processing discussed in the following section) +is achieved by using custom ``sdist`` command for setuptools. Thus, packaging +steps remain the same as for the standard setuptools, the user just +needs to override ``sdist`` command implementation by passing the +appropriate argument to ``setup()`` call:: + + from setuptools import setup + import sdist_upip + + setup( + ..., + cmdclass={'sdist': sdist_upip.sdist} + ) + +The sdist_upip.py module as referenced above can be found in +`micropython-lib`: +https://github.com/micropython/micropython-lib/blob/master/sdist_upip.py + + +Application resources +--------------------- + +A complete application, besides the source code, oftentimes also consists +of data files, e.g. web page templates, game images, etc. It's clear how +to deal with those when application is installed manually - you just put +those data files in the filesystem at some location and use the normal +file access functions. + +The situation is different when deploying applications from packages - this +is more advanced, streamlined and flexible way, but also requires more +advanced approach to accessing data files. This approach is treating +the data files as "resources", and abstracting away access to them. + +Python supports resource access using its "setuptools" library, using +``pkg_resources`` module. MicroPython, following its usual approach, +implements subset of the functionality of that module, specifically +``pkg_resources.resource_stream(package, resource)`` function. +The idea is that an application calls this function, passing a +resource identifier, which is a relative path to data file within +the specified package (usually top-level application package). It +returns a stream object which can be used to access resource contents. +Thus, the ``resource_stream()`` emulates interface of the standard +`open()` function. + +Implementation-wise, ``resource_stream()`` uses file operations +underlyingly, if distribution package is install in the filesystem. +However, it also supports functioning without the underlying filesystem, +e.g. if the package is frozen as the bytecode. This however requires +an extra intermediate step when packaging application - creation of +"Python resource module". + +The idea of this module is to convert binary data to a Python bytes +object, and put it into the dictionary, indexed by the resource name. +This conversion is done automatically using overridden ``sdist`` command +described in the previous section. + +Let's trace the complete process using the following example. Suppose +your application has the following structure:: + + my_app/ + __main__.py + utils.py + data/ + page.html + image.png + +``__main__.py`` and ``utils.py`` should access resources using the +following calls:: + + import pkg_resources + + pkg_resources.resource_stream(__name__, "data/page.html") + pkg_resources.resource_stream(__name__, "data/image.png") + +You can develop and debug using the `MicroPython Unix port` as usual. +When time comes to make a distribution package out of it, just use +overridden "sdist" command from sdist_upip.py module as described in +the previous section. + +This will create a Python resource module named ``R.py``, based on the +files declared in ``MANIFEST`` or ``MANIFEST.in`` files (any non-``.py`` +file will be considered a resource and added to ``R.py``) - before +proceeding with the normal packaging steps. + +Prepared like this, your application will work both when deployed to +filesystem and as frozen bytecode. + +If you would like to debug ``R.py`` creation, you can run:: + + python3 setup.py sdist --manifest-only + +Alternatively, you can use tools/mpy_bin2res.py script from the +MicroPython distribution, in which can you will need to pass paths +to all resource files:: + + mpy_bin2res.py data/page.html data/image.png + +References +---------- + +* Python Packaging User Guide: https://packaging.python.org/ +* Setuptools documentation: https://setuptools.readthedocs.io/ +* Distutils documentation: https://docs.python.org/3/library/distutils.html |