Exploring A Python Module Path Problem

 tech  No Responses »
Jun 152018
 

I recently encountered a surprising problem while setting up an AWS environment: pylint fails to run in a python 3.6 virtualenv on Amazon Linux 2018.3!

(pylint-test) [root@myhost ~]# pylint
Traceback (most recent call last):
File "/root/pylint-test/bin/pylint", line 11, in
sys.exit(run_pylint())
File "/root/pylint-test/local/lib/python3.6/dist-packages/pylint/__init__.py", line 15, in run_pylint
from pylint.lint import Run
File "/root/pylint-test/local/lib/python3.6/dist-packages/pylint/lint.py", line 64, in
import astroid
File "/root/pylint-test/local/lib/python3.6/dist-packages/astroid/__init__.py", line 54, in
from astroid.exceptions import *
File "/root/pylint-test/local/lib/python3.6/dist-packages/astroid/exceptions.py", line 11, in
from astroid import util
File "/root/pylint-test/local/lib/python3.6/dist-packages/astroid/util.py", line 11, in
import lazy_object_proxy
ModuleNotFoundError: No module named 'lazy_object_proxy'
(pylint-test) [root@myhost ~]#

This post is about hunting down the problem, finding a workaround, and identifying next steps toward a root cause fix.

Initial Troubleshooting

A glance at the stack trace suggests this is simply a matter of a missing module. Basically, that the pylint dependency chain wasn’t correctly installed.

This seemed unlikely, though – I’d simply used pip to install pylint in an uncomplicated virtualenv, and pip had successfully identified and installed the needed dependencies. If this was the problem, then there was a more fundamental problem with either pip or one of the modules it was installing.

I went through my install/setup steps again, just to make sure this wasn’t the result of simple user error.

  • Set up a fresh virtualenv with the desired python version:
    [root@myhost ~]# rpm -qf `which python3`
    python36-3.6.5-1.9.amzn1.x86_64
    [root@myhost ~]# virtualenv-3.6 -p /usr/bin/python3 pylint-test
    Running virtualenv with interpreter /usr/bin/python3
    Using base prefix '/usr'
    New python executable in /root/pylint-test/bin/python3
    Also creating executable in /root/pylint-test/bin/python
    Installing setuptools, pip, wheel...done.
  • Activate the virtualenv:
    [root@myhost ~]# source pylint-test/bin/activate
  • Check that the expected pip is being used in the virtualenv:
    (pylint-test) [root@myhost ~]# pip --version
    pip 10.0.1 from /root/pylint-test/local/lib/python3.6/dist-packages/pip (python 3.6)
  • Install pylint via pip (ignoring anything pip has cached, just in case):
    (pylint-test) [root@myhost ~]# pip --no-cache-dir install pylint Collecting pylint Downloading https://files.pythonhosted.org/packages/f2/95/0ca03c818ba3cd14f2dd4e95df5b7fa232424b7fc6ea1748d27f293bc007/pylint-1.9.2-py2.py3-none-any.whl (690kB) 100% || 696kB 7.7MB/s Collecting isort>=4.2.5 (from pylint)
    Downloading https://files.pythonhosted.org/packages/1f/2c/22eee714d7199ae0464beda6ad5fedec8fee6a2f7ffd1e8f1840928fe318/isort-4.3.4-py3-none-any.whl (45kB)
    100% || 51kB 9.7MB/s
    Collecting astroid<2.0,>=1.6 (from pylint)
    Downloading https://files.pythonhosted.org/packages/0e/9b/18b08991c8c6aaa827faf394f4468b8fee41db1f73aa5157f9f5fb2e69c3/astroid-1.6.5-py2.py3-none-any.whl (293kB)
    100% || 296kB 8.1MB/s
    Collecting mccabe (from pylint)
    Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
    Collecting six (from pylint)
    Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
    Collecting lazy-object-proxy (from astroid<2.0,>=1.6->pylint)
    Downloading https://files.pythonhosted.org/packages/65/1f/2043ec33066e779905ed7e6580384425fdc7dc2ac64d6931060c75b0c5a3/lazy_object_proxy-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (55kB)
    100% || 61kB 16.9MB/s
    Collecting wrapt (from astroid<2.0,>=1.6->pylint)
    Downloading https://files.pythonhosted.org/packages/a0/47/66897906448185fcb77fc3c2b1bc20ed0ecca81a0f2f88eda3fc5a34fc3d/wrapt-1.10.11.tar.gz
    Installing collected packages: isort, lazy-object-proxy, wrapt, six, astroid, mccabe, pylint
    Running setup.py install for wrapt ... done
    Successfully installed astroid-1.6.5 isort-4.3.4 lazy-object-proxy-1.3.1 mccabe-0.6.1 pylint-1.9.2 six-1.11.0 wrapt-1.10.11

That all seems as expected. But still, no joy getting pylint to run:

(pylint-test) [root@myhost ~]# pylint
Traceback (most recent call last):
File "/root/pylint-test/bin/pylint", line 11, in
sys.exit(run_pylint())
File "/root/pylint-test/local/lib/python3.6/dist-packages/pylint/__init__.py", line 15, in run_pylint
from pylint.lint import Run
File "/root/pylint-test/local/lib/python3.6/dist-packages/pylint/lint.py", line 64, in
import astroid
File "/root/pylint-test/local/lib/python3.6/dist-packages/astroid/__init__.py", line 54, in
from astroid.exceptions import *
File "/root/pylint-test/local/lib/python3.6/dist-packages/astroid/exceptions.py", line 11, in
from astroid import util
File "/root/pylint-test/local/lib/python3.6/dist-packages/astroid/util.py", line 11, in
import lazy_object_proxy
ModuleNotFoundError: No module named 'lazy_object_proxy'
(pylint-test) [root@myhost ~]#

Isolating the Potential Problem Space

With basic user error ruled out, I set out to narrow down the scope of the problem. It seems extremely unlikely that pylint or its package are just broken – it’s too commonly used a tool for that. For my own assurance, though, I checked a few other similar environments I had handy and confirmed that:

  • python 3.6.5 and pylint worked on my mac
  • setting up a similar virtualenv on a RHEL 7.5 EC2 instance also worked (“worked” meaning pylint ran as expected)
  • pylint in  a python 3.6 virtualenv on my Qubes (linux) desktop also worked

OK, so this is something specific to this particular environment. The OS distro (Amazon Linux 2018.03) was the most obvious difference between the non-working and working environments. But by itself that’s not much of a clue – the OS should have almost nothing to do with the behavior of pip or a module inside a virtualenv.

Since the apparent issue was with the lazy_object_proxy module, I thought I’d try to install just that module and see what happened:

(pylint-test) [root@myhost ~]# pip --no-cache-dir install lazy-object-proxy
Collecting lazy-object-proxy
Downloading https://files.pythonhosted.org/packages/65/1f/2043ec33066e779905ed7e6580384425fdc7dc2ac64d6931060c75b0c5a3/lazy_object_proxy-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (55kB)
100% |████████████████████████████████| 61kB 2.6MB/s
Installing collected packages: lazy-object-proxy
Successfully installed lazy-object-proxy-1.3.1
(pylint-test) [root@myhost ~]# echo $?
0
(pylint-test) [root@myhost ~]# pip list | grep lazy
(pylint-test) [root@myhost ~]#

Huh? It says it installs successfully, but immediately thereafter isn’t on the list of installed modules? Fishy. It seems like understanding what pip is doing with this module is the appropriate next step troubleshooting.

Digging in to pip and lazy_object_proxy

So if pip thinks it’s installing lazy_object_proxy, what’s it doing on the filesystem when it does so? Let’s look:

[root@myhost pylint-test]# find . -name lazy\*
./lib64/python3.6/dist-packages/lazy_object_proxy
./lib64/python3.6/dist-packages/lazy_object_proxy-1.3.1.dist-info

Hrm. So it is installing the module. How does this compare to the equivalent virtualenv on the RHEL 7.5 system where pylint works?

(pylint-test) [root@otherhost pylint-test]# find . -name lazy\*
./lib/python3.6/site-packages/lazy_object_proxy-1.3.1.dist-info
./lib/python3.6/site-packages/lazy_object_proxy

Aha – there’s a difference! On the (working) RHEL-7.5 system, lazy_object_proxy ends up in lib/python3.6/site-packages/, and on the Amazon Linux system with the non-functional pylint it’s in lib64/python3.6/dist-packages/. For reference, this blog post by Lee Mendelowitz does a nice job explaining some fundamentals of package loading, and of site-packages vs. dist-packages in general.

Just because we’ve found a difference doesn’t mean it’s a meaningful difference. In fact, it looks like all of the modules installed on Amazon Linux are in dist-packages rather than site-packages. (I find this surprising, since my understanding is that Amazon Linux is more or less RedHat derived, but that’s a different topic.)

And indeed, packages in dist-packages generally work fine on the Amazon Linux system. For example, the ‘six’ module:

(pylint-test) [root@myhost pylint-test]# pip list | grep six
six 1.11.0
(pylint-test) [root@myhost pylint-test]# python -i
Python 3.6.5 (default, Apr 26 2018, 00:14:31)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.util
>>> importlib.util.find_spec('six')
ModuleSpec(name='six', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f1dbed43c88>, origin='/root/pylint-test/local/lib/python3.6/dist-packages/six.py')
>>> import six
>>>

This makes sense, because pylint-test/local/lib/python3.6/dist-packages is in the pythonpath (pylint-test/local/lib is a symlink to pylint-test/lib/):

>>> import sys
>>> print('\n'.join(sys.path))
/root/pylint-test/local/lib64/python3.6/site-packages
/root/pylint-test/local/lib/python3.6/site-packages
/root/pylint-test/lib64/python3.6
/root/pylint-test/lib/python3.6
/root/pylint-test/lib64/python3.6/site-packages
/root/pylint-test/lib/python3.6/site-packages
/root/pylint-test/lib64/python3.6/lib-dynload
/root/pylint-test/local/lib/python3.6/dist-packages
/usr/lib64/python3.6
/usr/lib/python3.6
>>>

Problem Identified; Workaround

The problem with the lazy_object_proxy module is clear at this point: the module is being installed to a directory not in the module search path (sys.path). For some reason, pip installs it to ./lib64/python3.6/dist-packages/lazy_object_proxy, but neither lib64 nor local/lib64 (which symlinks to the former) is in the module path.

This suggests an easy workaround – manually adding the appropriate directory to the pythonpath should make the module loadable:

(pylint-test) [root@ip-172-31-44-121 pylint-test]# python -i
Python 3.6.5 (default, Apr 26 2018, 00:14:31)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lazy_object_proxy
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'lazy_object_proxy'
>>>
(pylint-test) [root@ip-172-31-44-121 pylint-test]# PYTHONPATH="./lib64/python3.6/dist-packages/" python -i
Python 3.6.5 (default, Apr 26 2018, 00:14:31)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lazy_object_proxy
>>>

Speculation Re: Root Cause & Real Solution

While manually modifying the environment (PYTHONPATH) might be an effective workaround, it’s not really a solution. As I see it, a key question remains: why is pip installing a package outside of the module search path? Or perhaps pip is doing the right thing, and the question is why isn’t lib64/python3.6/dist-packages in the default module search path?

Is this a pip bug? An error in Amazon Linux’s python packaging? Something subtly wrong in the lazy_object_proxy packaging?

The pip docs don’t seem to address this question, or for that matter explain what determines the path a given module will end up in. They do suggest other possible workarounds (i.e. manually configuring pip install destinations), but those don’t seem more satisfying to me than the PYTHONPATH workaround, and don’t shed light on the remaining mystery.

I have a gut suspicion that the core of this issue is likely related to something specific to Amazon Linux – perhaps the logic in site.py that sets the default module search path. This is very much just a hunch, but the use of dist-packages (rather than site-packages) seems odd. It’s also strange that the out-of-the-box sys.path in Amazon Linux includes /root/pylint-test/local/lib/python3.6/dist-packages but not its lib64 equivalent. I started a thread on the AWS forum to see if anybody there can has a relevant insight.

 Posted by jrh at 9:51 am  Tagged with: , ,
© 2021 layer8 Suffusion theme by Sayontan Sinha