Stop struggling with Python on Windows


The title of the page isn’t a hyperbole. If you have every tried to use Python beyond toy problems on Windows, you will have struggled with library clashes, 32/64 bit versions, ghost libraries (that seemed to be installed, but can’t be called).

As someone who uninstalled and reinstalled Python eight times in the last four weeks, let me tell you, getting Python working perfectly on Windows isn’t easy.

“But Mr T,” you say, “what’s the problem? You go to python.org and just download it. What’s the big deal?”

If you plan to just stick to solving Project Euler or other such toy problems, then the official site is good enough.

(Note: Lots of people have taken the previous sentence as an insult. It is not, and I apologise if it came out that way. All I meant was, in this day and age, it’s a shame there is no good packager for Python on Windows).

But I found that as soon as you try to do anything more complicated, like install libraries which need to be compiled (like numpy or scipy), the official build starts falling down. For my upcoming course on Python, I needed a lot of third party libraries, and I found that Python’s packaging system is really broken.

Here are some of the few Python versions I’ve tried:

1. The official from Python.org: Especially on Windows, this is good only for learning. I didn’t find it very stable, and even installing basic things like Virtualenv gave me problems.

2. The version from ActiveState: ActiveState have a version of Python, and it’s a lot more stable than the official version. It also makes it easy to install packages, and for a long time, this is what I used. But then I tried to install OpenCv, and got problems with versions. I decided to re-install Python, and that’s when I hit problems.

For some stupid reason, ActiveState install libraries in the C:\users\<name>\roaming\ region. These libraries take precedence over the local files (maybe due it being higher on the path hierarchy). Anyway, even after installing a different Python build (from Conda, see below), I kept getting library clashes which I couldn’t trace. I blamed Conda and spent weeks investigating obtuse error messages. I finally realised that ActiveState was to blame. The path to Python libraries must have been put in the registry or something, because even when I deleted the folders, it still looked for them in the Roaming folders.

I had to reinstall ActiveState Python, remove the libraries, and then uninstall it again.
Why am I telling you this? To show you how hard getting Python running can be on Windows. Recently, I discovered Anaconda, and that’s what I recommend now.

3. Anaconda
Based on the open source Conda packaging system, Anaconda is the best Python distribution I have found till now. For one, it comes with all the scientific libraries like numpy, scipy etc preinstalled, so you don’t have to worry about messing with compiling them yourself.

Second, it does not put important stuff in hidden folders (as far as I can see). Uninstalling it is easy and works.

Conda isn’t perfect, as I couldn’t get its environment system to work. But it’s a lot better than anything out there, and the only one I would recommend, at least on Windows

To install new packages, first try to use Conda install. If there is no Conda package, try pip, and then compiling from source (but never use easy_install).

In conclusion

Stop struggling with Python on Windows. Avoid distributions with fancy install methods (I’m looking at you, ActiveState). Install the Anaconda build, preferably 32 bit (as some libraries aren’t available for 64 bit, and you will get library clashes).

Interested in knowing more? Subscribers will exclusive content I won’t post here.
* indicates required
I will never spam you. Unsubscribe anytime.

Leave a comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

22 thoughts on “Stop struggling with Python on Windows

  • Matthew Blott

    You describe my experience about 9 months ago! I’ve become so frustrated I am now getting myself a Mac – I have issues with Ruby, PHP and Node on Windows as well. The main program installers work without much bother but as soon as you start playing with third party libraries you will get issues.

    I’ve not tried Anaconda but I’ve just had a look and it has an impressive set of libraries out the box, I’ll try it out. You might want to have a look at this site (if you haven’t already):

    http://www.lfd.uci.edu/~gohlke/pythonlibs/

    It’s an impressive up to date list of Windows binaries for Python packages and I’ve installed a few with good results!

    Regards,

    Matt ;-)

    • Shantnu Post author

      Thanks Matt. Many people struggle, but there is a macho attitude in the community “Just be a big boy and get on with it”, which is why I decided to write this post.

      It’s not just a Python issue- as you mention, other frameworks, most notably Ruby have had problems as well. I was looking at a job posting by a famous company for Ruby developers, and the advert basically said “You must be good with Linux, as Ruby doesn’t work on Windows” (their words, not mine).

      Yes, I’ve seen that link, even tried a few libraries from it. While they work, my problem is the same: I’d rather have a centralised package manager than having to install/uninstall libraries from a dozen locations. That’s what I like about Conda. It makes it easier to manage once you have more than a dozen libraries.

    • Robert

      Just a innocent question, (at least I’m trying not to make it loaded :) ) why not install a virtual machine in windows for your use case, or dual boot to Linux, instead of buying another machine?

      • Shantnu Post author

        Robert, that’s a good point. For my course, I will be using a VM running Linux, but I’ve been asked by many people for help on Windows, so I thought I’d give it a try. I thought, how hard can it be? Plenty hard, it seems.

      • Matthew Blott

        I did try Linux but I find you have to do a fair bit of heavy lifting with it. For example, Node has a Mac installer (and Windows) but you have to build it with Linux. That’s fine for a production server but when you want to tinker around with stuff (as I do) it gets a bit inconvenient. I use C# a lot and building that on Linux is a mission – on a Mac it just works out the box. Because Windows and Mac aren’t free means they get targeted for the professional dev tools. I know there are some lovers of Linux out there and I’m not knocking you!

  • Andrey

    Have you been developing for Windows for 1—2 weeks? For 7 years of Python programming I can assure you you should have no problems installing virtualenv, numpy, scikits or even PYAudio. Windows version of Python have the same stability as any other versions.

    Don’t blame the tools.

    • Shantnu Post author

      Andrey, obviously, I’m pretty incompetent. I should stop showing my face in public. In fact, I’m going to do cover my face with a paper bag right now. :)

    • Stu

      If you dont have vs installed already it is a pain to work out which sdk to install to get compiling working.

      Last time I did numpy on windows in a virtualenv I had to unpack the installer and use easy install on the parts inside, which wasnt intuitive.

    • Matthew Blott

      I didn’t blame any “tools” I said you’ll find a lot of third party libraries don’t run on Windows. Do you dispute this? And since I said I had problems months ago your question of whether I’ve only been using Windows for a couple of weeks I will have to put down to impertinence.

  • richard fernandez

    I use Anaconda on Linux and it works great! Python is not really suited for Windows but that’s why there are VMs for those who can’t get around needing windows on their machine.

  • Gavin Scott

    Anaconda might be the most powerful (and free!) software package for general end-user computing that there is. It just works, and it gives you everything you need on all three major platforms to do virtually anything out of the box.

    I too recommend: http://www.lfd.uci.edu/~gohlke/pythonlibs/ for those Windows things you might need that are not available as conda packages (like Postgres and MySQL database drives to go with SQL Alchemy). The installers from there integrate well with anaconda (it will find all Python environments registered on the machine, let you pick one, and then install into the correct places).

    anaconda installs in a single directory and makes it easy to have multiple versions and multiple python versions installed. It comes with the Spyder IDE, and all the iPython stuff, including what’s probably the most impressive component of the Python eco-system these days, the iPython Notebook interface. I now do basically all my calculation and ad-hoc computing in support of my development work inside Notebooks.

    Honestly everyone who does any form of “computing” using their computer needs to have the anaconda system, and Python is the #1 language/tool to learn because you can do and drive basically anything and everything with it.

    If you’re an iOS user, there are a couple great Python environments there too. Pythonista (http://omz-software.com/pythonista/) for the iPhone and iPad and now also the new Computable (http://computableapp.com/) which brings the iPython Notebook interface plus numpy, sympy, pandas, etc. to the iPad.

    G.

  • Stu

    If you want a proper setup – virtualenvs, and lots oc libraries that need a compiler it is a pain.

    Having wasted many hours myself I always use linux in a vm if I have a windows box and advise others to do the same.

  • Steve

    An interesting article and I recognise some of the challenges of Python on Windows but have to say that it doesn’t accord with my experience. I’ve found the ActiveState 32bit Python 2.7 to be stable and with pypm it is trivial to install 95% of what I need – matplotlib, numpy, scipy, ipython, pandas, nmap, etc, etc. OpenCV was easy to install too – the install instructions on the website work perfectly – you do read the instructions, don’t you? ;-) If you really want a command line package manager for the OS you can always install Chocolatey https://chocolatey.org/.

    I’ve spent many years working with various Linux distros too and could write a similar article on the challenges posed by that OS. The time I’ve spent getting CUPS to work with my printer easily outweighs any time spent fiddling with Windows to get Python to work nicely.

    I think that it’s quite realistic to develop with Python on Windows or *nix but you should expect in either OS that you sometimes have to get under the hood in order to get things to work!

    • Shantnu Post author

      ActiveState Python is stable, I just dont like its packaging manager.

      And you are right that Linux has its own problems. But since a large number of people (majority?) use Windows, I thought I’d write on that

    • Matthew Blott

      Generally developing Python on Windows isn’t too bad it’s just there are restrictions I’d rather do without.

      I tried Chocolatey but got rid of it. Windows uninstalls have a habit of leaving turds in your file system and registry so I only install stuff with Revo Installer now, much safer :-)

      • Shantnu Post author

        Windows uninstalls have a habit of leaving turds in your file system and registry

        True. We need to start running Windows in a VM (instead of Linux, which is what people do), and every six months wipe everything and start clean :)