Create, build and ship a Python3 pip module in 5 minutes
If you use Python, you likely encountered pip on your path. Pip is the most used package manager for Python modules.
Pip is a recursive acronym for “Pip Installs Packages” [Wikipedia]
Last year was the 10th birthday of pip, which makes it a good software — a rarity these days. I remember the time when pip was considered a liability. Pip had this label of being a second rebellious captain on a ship, known to conflict with the distribution’s primary packaging system, bringing up all sorts of security and stability related concerns.
Today, with containers — such as Docker — the worries are gone. Application code and their dependencies run in a controlled, isolated environment. Testing for vulnerabilities is easier, and no more need to fight the sys-admin.
Python’s use and popularity are skyrocketing as well — thanks to machine learning for that. This makes the world hungry for more Python modules.
Let’s ship some modules.
Shipping pip packages — the automated way
Prerequisites
- PyPi account. Register here and save your credentials in a safe place.
- Docker. Install instructions: Ubuntu or Mac.
You also need the ability to run make, have a POSIX compliant shell and a code editor — nearly every Linux-, Unix- or Mac system will work.
PyShipper
PyShipper is an automation part, doing the grunt-work of setting up a module structure, adds some boilerplate code, and provides a pipeline to build, test and ship a module — saving time when you update or create a module.
The relevant code is mostly in Makefile, complemented with some Docker configuration and a slightly modified setup.py. For the curious readers, this previous Docker & Makefile article explains the base. PyShipper is a version that does the job of Python module delivery.
Steps
- Think of a name
- Create a new module directory
- Configure variables
- Run and edit code
- Make module
- Publish
I will go through each step, and explain where and how PyShipper kicks in to automate a few things along the way.
1. Think of a name
If you find it difficult to come up with a good name, you are not alone:
There are only two hard things in Computer Science: cache invalidation and naming things. — Phil Karlton
I always struggle with naming things. After hours in slow thinking mode, I usually end up with a name that describes what problem is solved — hence the name PyShipper. If a name is taken, pre- or postfixing the name with “Py”, “Script”, “Tool”, or something similar usually works out.
Naming is one of these things I have not yet figured out how to automate. If there are any machine learning related ideas, please share!
2. Create a new module structure
This documentation on Python packages describes the required package structure. Even as we automated this step, it goes without saying the documentation is still essential reading material — useful for debugging.
Let’s go to the one-liners.
# get a copy of PyShipper and change into it
git clone https://github.com/LINKIT-Group/pyshipper.git
cd pyshipper# fork (copy/ paste) the contents to a new directory
make fork dest=~/${YOUR_NEW_MODULE_NAME}
This forks a stripped version of PyShipper to a new directory called ~/${YOUR_NEW_MODULE_NAME}. Files like LICENSE and README.md are not copied. After all, you own the new module, and thus need to add a license and documentation to it — no license strings attached.
For the purpose of review, this is the Makefile fork target:
Safety note: files are only copied if there is either no directory or an empty directory.
3. Configure variables
Our next step is to change to the new directory and edit the /variables file:
# switch to the new module
cd ~/${YOUR_NEW_MODULE_NAME}# edit the /variables file
{replace_with_your_editor} variables
The NAME variable in /variables is picked up by Makefile, and all variables are exported to the environment of a Docker container, used by setup.py during container execution.
The Makefile line took me a few hours to work, stripping of strong-quotes properly is nearly always painful. The problem became easy when I stumbled on this piece of documentation — number 47 it is, for GNU AWK at least.
While the single Makefile line was hard work, getting the variables passed to setup.py was much easier. I only needed a way to pass all variables to the container. A short ~/.profile file in the container space does that job — note, for this to work, the /variables file must also be present via a link or mount.
To finalize this section, this is the /setup.py file that loads the variables in the container environment. The small lambda ensures an empty string is inserted if a variable is not present.
4. Run and edit code
PyShipper ships with a /module directory containing boilerplate code for a module. There is also a coding pattern, with a minimal “hello-world”-like function, that can be called both CLI- and import-style.
Of course all Python3 — as Python2 goes EOL Januari 2020!
# enter the container runtime
make shell# test run the module in CLI
python3 -m module --name "PyShipper"# start Python3, import the module and test
python3>>> import module
>>> module.main(name="PyShipper")
This runs the code under /module. Instead of using the name “module”, you can also replace it with the name of your own module. A symlink is created to make both work — in the shipped version, only your own name can be used to reference the module.
If you are a Python developer, I am sure you need to know what do next. All module code to edit is in the /module — have fun tinkering!
5. Make module
When you have a minimal working version it is time to build the package. One thing you may want to do first, is to check the VERSION in /variables and ensure it is updated — I just made a note to myself to automate that in a next version, who likes keeping track of versions? — right ;).
# build the module -- this runs setup.py in the container
make module# better practice version of the former
# includes pylint code quality testing
make pylint module
This is the Makefile configuration connected to the one-liners above:
6. Publish
The output of the build process is a gzipped tar archive, present in the /dist directory. By uploading this file to PyPi —Python Package Index — the module is published, and install-able through Python pip by everyone.
# upload the module
make uploadThe command above prompts for a username and password. You need to insert your PyPi credentials—see Prerequisites at the beginning of this chapter.
The Makefile snippet for the upload target is:
After you created the module, published it on PyPi, you can install it on any capable system and use it like any other Python3 pip module.
# install the module on system
sudo python3 -m pip install ${YOUR_NEW_MODULE_NAME}Final thoughts
I already use this automation myself and I am quite happy with it. I regularly create small Python modules for specific tasks; it’s incredibly easy to import them in containers or server-less environments afterwards.
Now with this piece of automation more repetitive work is cut — that means more time to spend on other innovations.
I hope you find this equally useful. I’d be delighted if this helps, at least to some of you, to contribute to the Python eco-system.
Happy Python module writing!
