Multiple Versions of Python on Windows

A lot of testers like to use Python. A lot of testers are on Windows machines. There is currently a large gap in the Python world between the 2.x and 3.x branch. (See more details on this.) This often puts testers in the position of having to make Python solutions that run on both versions. But, on Windows at least, this has been tricky in terms of having two Python versions installed and easily moving between them. Here I’ll talk about how the Python development team has eased this with the Windows launcher.

The official solution for coexistence is the Python Launcher for Windows, talked about in PEP 397, which was included in Python 3.3.0. The documentation on this is sometimes tricky to wade through so I want to present what I hope is a simplified view here.

Installing Multiple Versions

Let’s say you want to have the latest 3.x branch and 2.x branch installed at the same time. At the time I write this, that would be 3.4.1 and 2.7.7. So here’s what you do:

  1. Go to the Python download page.
  2. Download the 3.4.1 installer package.
  3. Install it and accept the defaults. (This is important. The defaults include not adding python.exe to the path.)
  4. Download the 2.7.7 installer package.
  5. Install it and, this time, do NOT accept the defaults. Specifically, remove the “Register Extensions” option from installing.

Installing the 3.4.1 package will have put py.eye and pyw.exe launchers into %SYSTEMROOT% (on the Windows system PATH) and then associate those with .py and .pyw scripts, respectively. That last part is important: the installation will have registered them as the applications to execute when Python is executed. This is specifically why I had you remove the option to Register Extensions for the 2.7 install. Doing that would have overridden the launcher being the default. Note that these py.exe and pyw.exe files are different from the normal python.exe executable that each package you installed will have placed in c:\Python34 and c:\Python27, respectively.

Note: It’s possible the next iteration of the 2.7.x launcher will remove the fact that Register Extensions are installed by default. The main thing is just to pay attention to it and make sure the setting is disabled.

But how does all this work? First, understand that these new py.exe and pyw.exe files:

  • Open Python source and byte-code files launched by icon clicks or filename commands.
  • Do not require a directory path or PATH settings when used at the command line.
  • Accept Python version numbers to be passed in as command-line arguments.
  • Accept Unix-style #! comment lines at the top of scripts to determine the Python version to use.

Now let’s see how you actually make all that happen.

Using the Launcher

The new launcher mechanism means you can now select versions explicitly on either a per-file or per-command basis. Or, in fact, both at the same time. You can also specify versions in either partial or full form in both of those contexts. This raft of options is nice, I suppose, but it can be confusing. So, for example, if you want to select the Python version to run per file, you can use the Unix-style “shebang” comments like these:

#! python2
#! python3
#! /usr/bin/python2.7
#! /usr/bin/env python3.4
#! c:\Python34\python.exe

Traditionally those shebang lines have been ignored on Windows, but Python versions 3.3 and up will parse this line on Windows as well. There is a caveat to this, which I’ll take about in a bit.

If you want to select versions per command, you can use command line options like these:

py -2 my_file.py
py -3 my_file.py
py -2.7 my_file.py
py -3.4 my_file.py

The launcher also applies some logic to select a specific Python version when version information is missing or only partly described. For example, in both of my example sets above, you can see I use generic references to “python2” and “python3” (or “py -2” and “py -3”). How this works is that the latest 2.x Python version is run when only a 2 is specified. So if you have multiple 2.x versions installed, it’s the most recent that will be used if you specify a generic 2. The same applies for a generic 3: the latest Python version in the 3.x branch will be used.

You can, however, change this behavior if you want by setting valus for PY_PYTHON2 and PY_PYTHON3 variables. For example, if you have Python 2.5, 2.6 and 2.7 installed but you want a generic line like #! python2 to use 2.6, then you would simply set the PY_PYTHON2 variable to 2.6. This works similarly for using PY_PYTHON3 when there are multiple Python 3.x versions installed.

What’s more interesting to some people is that a 2.x version is preferred for files that do not name a version in a #! line when launched by icon click or generic command line calls. That’s an important point: by default, a 2.x version (if available) will be used by the launchers as the default Python even if a 3.x version is available. You can, however, change this behavior if you set a PY_PYTHON environment variable and set it to 3.

So just to be clear, let’s say you have Python 3.4.1 and 2.7.7 installed. You have a Python script (my_file.py) with the following shebang line: #!/usr/bin/python. In this case, that script will run against Python 2.7.7. You could change this operation by:

  • Changing the shebang line to: #!/usr/bin/python3
  • Specify the version at the command line: py -3 my_file.py
  • Change the value of PY_PYTHON to 3.

This probably seems odd behavior but I think the rationale here is two-fold. One is that many people using a Python ecosystem are still stuck in the 2.x world since many libraries are still in process of being ported to Python 3. So a default of Python 2 may make sense. Second is that Unix-based versions tend to rely on symbolic links from python to a specific version and that tends to be a 2.x version on most distributions. So I think the Windows launcher is trying to emulate the most likely use case for Python developers, but this can come at the expense of newcomers to Python.

A Few Caveats

Earlier, in particular with shebang lines, I mentioned there was a caveat that I would mention. That particular caveat is that the new Windows launcher recognizes Unix #! lines that begin with #!/usr/bin/env python but not other forms that are potentially used, like #!/bin/env python. Scripts with these shebang lines will work just fine on Unix-based machines but will fail on Windows with an error something like this: Unable to create process using '/bin/env python3.4. You’ll find the same occurs with a shebang line like this #! /bin/python. It seems the launcher requires you to follow one of the following patterns:

  • /usr/bin/env python*
  • /usr/bin/python*
  • /usr/local/bin/python*
  • python*

Another caveat to be aware of is that if there are both shebang lines in the Python script and a version number switch is provided in the command line used to execute that script, the command line version will override that specified in the shebang line.

Note that both launchers should have automatically disabled setting the PATH to a location of python.exe. This is important as well. In the new “Windows Launcher” world, if you start scripts with py instead of python, then having python.exe on the PATH is irrelevant at best, and confusing at worst, because the whole point is that py subsumes python and calls the relevant Python executable based on context.

Another caveat, although a potential useful one, is that the launcher will also let you distinguish between 32-bit and 64-bit versions of Python that are installed. So I can have Python 2.7.7 installed as both 32-bit and 64-bit as well as having 3.4.1 installed as both 32-bit and 64-bit. Using the launcher would allow me to distinguish between all of those.

I mentioned earlier how the “latest” Python 2.x or Python 3.x is used if a generic 2 or 3 is specified. I hope the above was clear what that meant but just to make it crystal clear: the “latest” does not refer to the “latest installed.” It refers to the “latest version number.” In the past, installing an older version of Python on a machine that already had a newer version would mean the “latest installed” would be used over the “latest version”. The launcher turns that around and always attempts to use the latest version that it can find on the machine.

If you request a version of Python that is not installed, you will get an error message along the lines of Requested Python version (2.7) is not installed.

Beyond that, I have not found too many other caveats worthy of mention. I’ve been using multiple versions of Python for quite some time and have had no issues. If Python is your language of choice, but you are worried about the 2.x/3.x divide that currently exists: fear not. You can easily develop with both.

Share

About Jeff Nyman

Anything I put here is an approximation of the truth. You’re getting a particular view of myself … and it’s the view I’m choosing to present to you. If you’ve never met me before in person, please realize I’m not the same in person as I am in writing. That’s because I can only put part of myself down into words.

If you have met me before in person then I’d ask you to consider that the view you’ve formed that way and the view you come to by reading what I say here may, in fact, both be true. I’d advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.

This entry was posted in Python. Bookmark the permalink.

9 Responses to Multiple Versions of Python on Windows

  1. Adam Connor says:

    How do you deal with python executables that install in the scripts directory? You can install something like pip with py -2 get-pip.py, but “pip” itself isn’t in the path.

  2. Adam Connor says:

    Eh, I guess you can run “py -2 -m pip <command and args>”, although I find it pretty verbose. Time for some new bat files…

    • Jeff Nyman says:

      Yeah, that’s a good point. You could make both “Scripts” directories available on your PATH. Then you could call pip2.exe and pip3.exe, respectively. Doing that, I can do this:

      C:\Users\Jeff
      $ pip2 –version
      pip 1.5.6 from C:\Python27\lib\site-packages (python 2.7)

      C:\Users\Jeff
      $ pip3 –version
      pip 6.0.8 from C:\Python34\lib\site-packages (python 3.4)

  3. Adam Connor says:

    I think that is workable for well-behaved packages, but there seem to be some out there that install their scripts into the main Python directory (virtualenvwrapper-win, for example). While they should probably be fixed, their scripts are inaccessible (without work) using this approach.

    • Jeff Nyman says:

      Indeed, that’s another very good point. I find myself getting back into the Python world, and likely on Windows for the time being, so I’ll explore this a bit more and see what I can learn and update with any findings.

  4. Adam Markon says:

    I’m currently following the instructions you gave, but with one difference which may be causing problems: I’m trying to dual install Python 2.7.6 and 2.7.9. There are major SSL differences between the two releases which is why I need the dual installation. As such, my steps to complete this task have been as follows:

    Install Python 2.7.6 to the default directory (C:\Python27\) with default settings
    Install Python 2.7.9 to a custom directory (C:\Python279\) with custom settings (don’t update registry, don’t add python.exe to path)

    After following these steps, both \Python27\python.exe and \Python279\python.exe open a Python 2.7.9 interpreter. PATH is only set to look in the Python27 directory, so I know that’s not the issue. Any ideas?

    • Jeff Nyman says:

      Interesting. When I try this setup, the installer for 2.7.9 tells me (in red text) that it will replace 2.7.6 … and that’s even when I tell it I’m installing in a new directory and not updating the registry or path. I’ll try a few more variations of this to see what’s going on.

    • Jeff Nyman says:

      Huh. I honestly don’t know. If I install Python 2.7.9 after 2.7.6, then it looks like Python tries to entirely overwrite the installation of the previous 2.x version.

      If, however, I install Python 2.7.9 first, I do get my two directories and the registry looks intact — but, as you state, any call to Python — even from within the 2.7.6 directory — calls up Python 2.7.9.

      I then tried using the Python launcher (introduced in Python 3.x). So basically using the instructions I provided here but then installing the two Python 2.x versions. Perhaps not surprisingly, the same problem occurred.

      It feels like it has to be something in the registry. But I’m at a loss for what.

  5. Ed says:

    I have decided to uninstall my various python versions, and tried what you have proposed, it works like a charm,.  The method you have proposed is so simple than many of the articles I came across to get multiple versions of python to co-exist.  Less headaches than what I was doing when writing scripts in multiple versions.

    BTW, how do you handle products like Eclipse, iPython, etc?  Is there an easy way like this?

    Thank you for writing something that is most helpful.

Leave a Reply

Your email address will not be published. Required fields are marked *