Creating this stream since finding time to sit down to right full coherent blog posts seems unlikely and people might like to hear a bit about our progress :)
First starting point, my PyCon AU 2013 talk Nobody Expects the Python Packaging Authority
If you don't want to listen to me babble on for ~35 minues and then answer a few questions for another 10, here's the slide source markup with speaker notes
Corrections related to pip and archiving downloads:
--download-cache option)I didn't quite understand Jacob's question about better alerting when packages were outdated. pip already provides a pip list --outdated option, but Jacob's interested in getting better notification of security issues at that kind of thing (he clarified this in his OWASP top 10 talk which followed mine).
We're hopeful we'll eventually be able to do something along those lines using the metadata extensions feature in PEP 426.
Discussions around PEP 439 (pip bootstrapping) have continued following my talk the PyCon AU sprints, in particular the thorny aspect of how to deal with system package managers (potentially including even MSI on Windows if pip is bootstrapped by the CPython installers) and pip getting into fights over who controls pip upgrades in the main Python installation.
Closely related are the current discussions regarding the apparently circular instructions for getting pip and virtualenv installed onto a system (where the recommended solution for getting pip is to use a virtualenv created virtual environment, which always includes pip, while the recommended solution for getting virtualenv is to install it using pip).
You can break the cycle with get-pip.py, but that gets you into the situation where you're in direct conflict with the system package manager on Linux, so its a poor solution to recommend to beginners (while using pip to install into the system site-packages on Linux can be useful if you know what you're doing, it can also get your system into a bad state if you make a mistake).
The view we seem to be edging towards is that it may be most appropriate to treat the "pip in the system Python" use case as an advanced use case, because the failure modes when the system package manager and pip get into an argument can be a bit ugly, and resolving them can be very platform specific. Even though things probably won't go wrong in a lot of cases, when they do go wrong, they're likely to wrong in a way that is hard to understand and fix if you're not an expert in the vagaries of Python's import system.
It's also where most of the complexity in PEP 439 lies: how do we keep the bootstrapped pip up to date? Do users have to run "pip install --upgrade pip"? Does it happen automatically when a maintenance release of CPython is installed? Do we bundle a particular version of pip with the installer binaries, or just offer the option to grab the latest from the internet as part of the installation process?
Thus, the consensus on the pip installer doc update issue is that the updated installation instructions will focus on bootstrapping virtualenv into the system Python, rather than bootstrapping pip directly.
The forward direction for Python 3.4 would then be to give the native pyvenv command (added in 3.3) an option that instructed it to add pip to the created venv, rather than attempting to make pip work outside an activated virtual environment.
Keep in mind this is all still a bit up in the air, though. Our main near term deadline is Python 3.4 beta 1 in November (see PEP 429)
This draft road map I posted to distutils-sig will likely be cleaned up and replace my previous Future of Python Packaging essay.
Although keep in mind that the draft road map is already missing things (see the distutils-sig thread), and if the pip docs switch to recommended virtualenv as the thing to bootstrap (rather than pip), that will affect the PEP that replaces PEP 439.
We eventually came to the view that pip-in-the-system Python is only a potential problem for Linux distros. It's the right thing to do everywhere else (including custom Python source builds on Linux).
So we plan to recommend that approach, and work with the Linux distros to ensure pip is available by default when Python is installed.
Donald Stufft has a draft PEP to that effect, but it's not quite ready for publication yet.
The odd (and broken from a security perspective) DNS based mirror autodiscovery protocol is also likely going away early next year - PEP 449 has the details.
Marcus Smith has updated the pip installation docs to avoid the appearance of depending on virtualenv: http://www.pip-installer.org/en/latest/installing.html
This gives a clear bootstrapping sequence for existing versions (setuptools -> pip -> everything else, including virtualenv), which should match the default experience in 3.4
Donald's explicit bootstrapping PEP was published as PEP 453 and has been through one round of edits based on the initial review. Ideally I'd like to see that in alpha 3 (Sep 29), but alpha 4 (Oct 20) is probably a more realistic target.
At the moment, I'm checking with some especially suitable core developers to see if they're interested in being BDFL-Delegate before I post it to python-dev, otherwise I'll make a general call when I send it (earlier discussions have been on distutils-sig). I'd prefer not to decide when this is one is "done" myself (although I will if necessary), and Guido has his hands pretty full with PEP 3156 (the standard async event loop API).
The draft road map has been tidied up and posted to the Python Packaging User Guide site in place of my personal essay from back in March.
The full text of that essay has been restored on my personal site.
Donald tells me that the whole of PyPI actually redirects to HTTPS now, including the parts used by easy_install and pip.
python-dev review resulted in PEP 453 being updated to include only docs changes for Python 2.7 and 3.3 and limit the functional changes to Python 3.4. The updated version has been sent to python-dev and is awaiting MvL's review (it's a long PEP, so it may take some time).
Donald has also started working on MSI installers for pip (not ready for prime-time yet, but kicking the tires of the experimental ones is encouraged).
Donald's project to update PyPI to a fully documented web application with a properly specified API also proceeds apace
PEP 453 has been accepted! Now, to get it implemented and documented for the first beta :)
Initial tracker issues for integration
Marcus has done some fine work tidying up the Python Packaging User Guide, including switching to the same theme as the Python 3 documentation and adding "Page Status" and "Last Reviewed" markers at the top of each page.
I now plan to refer to this from the CPython docs, rather than referring directly to pip-installer.org
The initial ensurepip patch for CPython 3.4 is basically complete, and just undergoing a final review prior to merge.
A very early version of the PyPI replacement project (Warehouse) is available at https://preview-pypi.python.org/
The core difference between Warehouse and the legacy PyPI infrastructure is that Warehouse actually has automated regression tests, and thus can be modified in the future without fear of breaking the entire Python packaging ecosystem. Building it alongside the existing PyPI service means it can also be tested for backwards compatibility issues before migrating the main site over to it.
Initial ensurepip implementation committed to CPython! Don't click this link, since hgweb displays the bundled wheel files in the commit output...
Good progress on the PEP 453 integration into CPython.
Just the Mac OS X installers still to go to be ready for beta 1 this weekend :)
CPython 3.4b1 shipped with the pip bundling included.
It's off by default for the moment on Windows, since we decided we wanted to solve the uninstallation problem after all. That should be fixed for beta 2.
I posted a collaboration/division of responsibility proposal for pip/virtualenv/venv and conda.
Divide and conquer!
The TUF folks posted an end-to-end security proposal for PyPI. We'll almost certainly adopt it in one form or another, the only question is when.
Given the challenges of working on the legacy PyPI code base (it has next to no automated tests, making it incredibly fragile), it will be some time after Warehouse reaches a sufficient level of maturity that the PyPI maintainers swap it in as the production PyPI implementation.
Oops, mega fail at keeping this up to date...
All the various issues with the pip bundling appear to have been worked out. pip 1.5.4 installed by default in the Windows and Mac OS X binary installers now, and we haven't had any new problems reported since rc3, so it all looks good for the final 3.4 release.
Donald Stufft has a new role at Rackspace that includes spending a substantial portion of his time on Warehouse development, so that is proceeding apace. The preview site is now integrated with the Fastly CDN and development and testing work is proceeding to ensure all the legacy APIs are implemented in order to make it possible to switch over from the legacy system.
I have also rewritten the Installing and Distribution guides that are part of the core Python documentation. They should now help to guide new users towards key components of the broader Python ecosystem, especially the Python Packaging User Guide.