You are not an advanced programmer if you have never faced the Works on my machine syndrome (and a liar if you claim you never made that statement to others). If you work in big teams, you will always come across a problem where a certain code (written by you, or a critical third party component) only works in certain versions.
This scenario becomes a nightmare when system libraries and tools (like compilers) are thrown into the Only works with X version mix. One Python project I knew only worked with a five year old version of MySql, and a specific version of Perl. I never understood why they needed Perl for a Python project.
Some general tips
Every time you install a library, or use one provided by either the system or the compiler / IDE /etc, make sure you note its version. Even better, put it in source control. At one place I worked, we used a proprietary compiler (because we used a certain embedded processor). The company often updated libraries, without caring about backward compatibility. To make it worse, these libraries would be silently updated when you upgraded the IDE (which you had to do to use newer chips). We ended up adding every single library we used to source control.
Make sure all libraries are being called explicitly, so you know there are no hidden dependencies.
There is no simple solution, unfortunately. However, today we will only talk about isolating Python code. There are several ways to isolate your Python code, and you must be aware of all of them, just so that you can choose the best tool for the task.
Isolating Python code
Python stores its libraries in a central location, so you can’t really back them up.
Instead, you must create isolated instances of Python, which is possible as you can have multiple Python installs (including multiple versions) on the same machines. Here are a few ways to isolate your Python code:
Virtualenv is the most popular and standard method of creating an isolated Python instance. Virtualenv installs Python in a local directory, and allows you to use that version of Python. That way, you can control which versions of libraries you are using.
Virtualenvs are activated using an activate command, which changes the path so that you end up using your local Python instead of the system Python.
Tthough on Linux you can just give the full path to your own Python version, like:
--> Your code goes here
Virtuaenv is the most common way, so most people think there is no problems with it. Many Python programmers are primarily web programmers, and that is the only world they know. In that world, virtualenv is good enough.
The problems starts when you want to use Python for engineering/science. Many scientific libraries like Numpy need to be compiled from source, and pip doesn’t handle that quite well. On Linux, you usually have all compilers and tools, but these are rarely present on Windows.
For people on Windows I usually recommend Anaconda. It solves most of the install problems for libraries like Scipy/Numpy.
Anaconda is based on Conda, which like Virtualenv, allows you to create boxed environments for running Python code. Unfortunately, while it works great on Linux, I couldn’t get it to work on Windows.
3. Virtual Machine
Virtual machines have several problems. They are several gigabytes in size (which make them hard to distribute). Not only that, many VMs come with a lot of decisions made for you-versions of important tools, databases etc. Virtual Machines weren’t really that easy to use.
Till Vagrant came along. That completely changed the game. Vagrant’s images are tiny, just a few hundred megabytes. You can choose to install the libraries you want, and even better, put the install scripts in version control. That means you can easily share your VM with others.
Vagrant also supports Puppet/Chef/Salt/Ansible, which means you can then deploy the code to the server without much effort either.
Vagrant is now my recommended choice. For small, personal projects, it doesn’t matter so much, but if you work in a team and want to share your code, there is nothing better than Vagrant.
If you are using Vagrant, make sure that you give a complete Vagrant file (ie, contains the scripts to install any libraries /tools you need), so that all the end user has to do is type vagrant up. Don’t make the user struggle with installing libraries, as that beats the purpose of using Vagrant.
Docker provides another layer of isolation. Unlike a VM, it uses a lightweight container that runs on top on the operating system.
While I understand the theory behind Docker, I’m not really sure when/why you would use it. If you have used Docker and have a killer usecase for it, feel free to share it in the comments.
If I have missed any other way to isolate your code, please share that as well, and I’ll update the blog.
PS: Want a free 1+ hour video course, Introduction to Web Scraping and Data Analysis? Also get a free mini-book, Python: From Apprentice to Master.
I will never spam you. Unsubscribe anytime.