{"id":217,"date":"2018-06-15T09:51:30","date_gmt":"2018-06-15T15:51:30","guid":{"rendered":"https:\/\/www.netfluvia.org\/layer8\/?p=217"},"modified":"2018-06-15T09:51:30","modified_gmt":"2018-06-15T15:51:30","slug":"exploring-a-python-module-path-problem","status":"publish","type":"post","link":"https:\/\/www.netfluvia.org\/layer8\/?p=217","title":{"rendered":"Exploring A Python Module Path Problem"},"content":{"rendered":"<p>I recently encountered a surprising problem while setting up an AWS environment: <code>pylint<\/code>\u00c2\u00a0fails to run in a python 3.6 virtualenv on Amazon Linux 2018.3!<\/p>\n<p style=\"padding-left: 30px;\"><code>(pylint-test) [root@myhost ~]# pylint<br \/>\nTraceback (most recent call last):<br \/>\nFile \"\/root\/pylint-test\/bin\/pylint\", line 11, in<br \/>\nsys.exit(run_pylint())<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/pylint\/__init__.py\", line 15, in run_pylint<br \/>\nfrom pylint.lint import Run<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/pylint\/lint.py\", line 64, in<br \/>\nimport astroid<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/astroid\/__init__.py\", line 54, in<br \/>\nfrom astroid.exceptions import *<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/astroid\/exceptions.py\", line 11, in<br \/>\nfrom astroid import util<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/astroid\/util.py\", line 11, in<br \/>\nimport lazy_object_proxy<br \/>\nModuleNotFoundError: No module named 'lazy_object_proxy'<br \/>\n(pylint-test) [root@myhost ~]#<\/code><\/p>\n<p>This post is about hunting down the problem, finding a workaround, and identifying next steps toward a root cause fix.<\/p>\n<h2>Initial Troubleshooting<\/h2>\n<p>A glance at the stack trace suggests this is simply a matter of a missing module. Basically, that the <code>pylint<\/code> dependency chain wasn&#8217;t correctly installed.<\/p>\n<p>This seemed unlikely, though &#8211; I&#8217;d simply used pip to install pylint in an uncomplicated virtualenv, and pip had successfully identified and installed the needed dependencies. If this was the problem, then there was a more fundamental problem with either pip or one of the modules it was installing.<\/p>\n<p>I went through my install\/setup steps again, just to make sure this wasn&#8217;t the result of simple user error.<\/p>\n<ul>\n<li><strong>Set up a fresh virtualenv with the desired python version:<\/strong><br \/>\n<code>[root@myhost ~]# rpm -qf `which python3`<br \/>\npython36-3.6.5-1.9.amzn1.x86_64<br \/>\n[root@myhost ~]# virtualenv-3.6 -p \/usr\/bin\/python3 pylint-test<br \/>\nRunning virtualenv with interpreter \/usr\/bin\/python3<br \/>\nUsing base prefix '\/usr'<br \/>\nNew python executable in \/root\/pylint-test\/bin\/python3<br \/>\nAlso creating executable in \/root\/pylint-test\/bin\/python<br \/>\nInstalling setuptools, pip, wheel...done.<br \/>\n<\/code><\/li>\n<li><strong>Activate the virtualenv:<\/strong><br \/>\n<code>[root@myhost ~]# source pylint-test\/bin\/activate<br \/>\n<\/code><\/li>\n<li><strong>Check that the expected pip is being used in the virtualenv:<\/strong><br \/>\n<code>(pylint-test) [root@myhost ~]# pip --version<br \/>\npip 10.0.1 from \/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/pip (python 3.6)<br \/>\n<\/code><\/li>\n<li><strong>Install pylint via pip (ignoring anything pip has cached, just in case):<\/strong><br \/>\n<code>(pylint-test) [root@myhost ~]# pip --no-cache-dir install pylint Collecting pylint Downloading https:\/\/files.pythonhosted.org\/packages\/f2\/95\/0ca03c818ba3cd14f2dd4e95df5b7fa232424b7fc6ea1748d27f293bc007\/pylint-1.9.2-py2.py3-none-any.whl (690kB) 100% || 696kB 7.7MB\/s Collecting isort&gt;=4.2.5 (from pylint)<br \/>\nDownloading https:\/\/files.pythonhosted.org\/packages\/1f\/2c\/22eee714d7199ae0464beda6ad5fedec8fee6a2f7ffd1e8f1840928fe318\/isort-4.3.4-py3-none-any.whl (45kB)<br \/>\n100% || 51kB 9.7MB\/s<br \/>\nCollecting astroid&lt;2.0,&gt;=1.6 (from pylint)<br \/>\nDownloading https:\/\/files.pythonhosted.org\/packages\/0e\/9b\/18b08991c8c6aaa827faf394f4468b8fee41db1f73aa5157f9f5fb2e69c3\/astroid-1.6.5-py2.py3-none-any.whl (293kB)<br \/>\n100% || 296kB 8.1MB\/s<br \/>\nCollecting mccabe (from pylint)<br \/>\nDownloading https:\/\/files.pythonhosted.org\/packages\/87\/89\/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57\/mccabe-0.6.1-py2.py3-none-any.whl<br \/>\nCollecting six (from pylint)<br \/>\nDownloading https:\/\/files.pythonhosted.org\/packages\/67\/4b\/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a\/six-1.11.0-py2.py3-none-any.whl<br \/>\nCollecting lazy-object-proxy (from astroid&lt;2.0,&gt;=1.6-&gt;pylint)<br \/>\nDownloading https:\/\/files.pythonhosted.org\/packages\/65\/1f\/2043ec33066e779905ed7e6580384425fdc7dc2ac64d6931060c75b0c5a3\/lazy_object_proxy-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (55kB)<br \/>\n100% || 61kB 16.9MB\/s<br \/>\nCollecting wrapt (from astroid&lt;2.0,&gt;=1.6-&gt;pylint)<br \/>\nDownloading https:\/\/files.pythonhosted.org\/packages\/a0\/47\/66897906448185fcb77fc3c2b1bc20ed0ecca81a0f2f88eda3fc5a34fc3d\/wrapt-1.10.11.tar.gz<br \/>\nInstalling collected packages: isort, lazy-object-proxy, wrapt, six, astroid, mccabe, pylint<br \/>\nRunning setup.py install for wrapt ... done<br \/>\nSuccessfully installed astroid-1.6.5 isort-4.3.4 lazy-object-proxy-1.3.1 mccabe-0.6.1 pylint-1.9.2 six-1.11.0 wrapt-1.10.11<br \/>\n<\/code><\/li>\n<\/ul>\n<p>That all seems as expected. But still, no joy getting <code>pylint<\/code> to run:<\/p>\n<p style=\"padding-left: 30px;\"><code>(pylint-test) [root@myhost ~]# pylint<br \/>\nTraceback (most recent call last):<br \/>\nFile \"\/root\/pylint-test\/bin\/pylint\", line 11, in<br \/>\nsys.exit(run_pylint())<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/pylint\/__init__.py\", line 15, in run_pylint<br \/>\nfrom pylint.lint import Run<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/pylint\/lint.py\", line 64, in<br \/>\nimport astroid<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/astroid\/__init__.py\", line 54, in<br \/>\nfrom astroid.exceptions import *<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/astroid\/exceptions.py\", line 11, in<br \/>\nfrom astroid import util<br \/>\nFile \"\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/astroid\/util.py\", line 11, in<br \/>\nimport lazy_object_proxy<br \/>\nModuleNotFoundError: No module named 'lazy_object_proxy'<br \/>\n(pylint-test) [root@myhost ~]#<\/code><\/p>\n<h2>Isolating the Potential Problem Space<\/h2>\n<p>With basic user error ruled out, I set out to narrow down the scope of the problem. It seems extremely unlikely that pylint or its package are just broken &#8211; it&#8217;s too commonly used a tool for that. For my own assurance, though, I checked a few other similar environments I had handy and confirmed that:<\/p>\n<ul>\n<li>python 3.6.5 and pylint worked on my mac<\/li>\n<li>setting up a similar virtualenv on a RHEL 7.5 EC2 instance also worked (&#8220;worked&#8221; meaning pylint ran as expected)<\/li>\n<li>pylint in\u00c2\u00a0 a python 3.6 virtualenv on my Qubes (linux) desktop also worked<\/li>\n<\/ul>\n<p>OK, so this is something specific to this particular environment. The OS distro (Amazon Linux 2018.03) was the most obvious difference between the non-working and working environments. But by itself that&#8217;s not much of a clue &#8211; the OS should have almost nothing to do with the behavior of pip or a module inside a virtualenv.<\/p>\n<p>Since the apparent issue was with the lazy_object_proxy module, I thought I&#8217;d try to install just that module and see what happened:<\/p>\n<p style=\"padding-left: 30px;\"><code>(pylint-test) [root@myhost ~]# pip --no-cache-dir install lazy-object-proxy<br \/>\nCollecting lazy-object-proxy<br \/>\nDownloading https:\/\/files.pythonhosted.org\/packages\/65\/1f\/2043ec33066e779905ed7e6580384425fdc7dc2ac64d6931060c75b0c5a3\/lazy_object_proxy-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (55kB)<br \/>\n100% |\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6\u00e2\u2013\u02c6| 61kB 2.6MB\/s<br \/>\nInstalling collected packages: lazy-object-proxy<br \/>\nSuccessfully installed lazy-object-proxy-1.3.1<br \/>\n(pylint-test) [root@myhost ~]# echo $?<br \/>\n0<br \/>\n(pylint-test) [root@myhost ~]# pip list | grep lazy<br \/>\n(pylint-test) [root@myhost ~]# <\/code><\/p>\n<p>Huh? It says it installs successfully, but immediately thereafter isn&#8217;t on the list of installed modules? Fishy. It seems like <strong>understanding what pip is doing with this module is the appropriate next step troubleshooting.<\/strong><\/p>\n<h2>Digging in to pip and lazy_object_proxy<\/h2>\n<p>So if pip\u00c2\u00a0<em>thinks<\/em> it&#8217;s installing lazy_object_proxy, what&#8217;s it doing on the filesystem when it does so? Let&#8217;s look:<\/p>\n<p style=\"padding-left: 30px;\"><code>[root@myhost pylint-test]# find . -name lazy\\*<br \/>\n.\/lib64\/python3.6\/dist-packages\/lazy_object_proxy<br \/>\n.\/lib64\/python3.6\/dist-packages\/lazy_object_proxy-1.3.1.dist-info<\/code><\/p>\n<p>Hrm. So it\u00c2\u00a0<em>is<\/em> installing the module. How does this compare to the equivalent virtualenv on the RHEL 7.5 system where pylint works?<\/p>\n<p style=\"padding-left: 30px;\"><code>(pylint-test) [root@otherhost pylint-test]# find . -name lazy\\*<br \/>\n.\/lib\/python3.6\/site-packages\/lazy_object_proxy-1.3.1.dist-info<br \/>\n.\/lib\/python3.6\/site-packages\/lazy_object_proxy<\/code><\/p>\n<p>Aha &#8211; there&#8217;s a difference! On the (working) RHEL-7.5 system, lazy_object_proxy ends up in lib\/python3.6\/site-packages\/, and on the Amazon Linux system with the non-functional pylint it&#8217;s in lib64\/python3.6\/dist-packages\/. For reference, <a href=\"http:\/\/(pylint-test) [root@ip-172-31-26-164 pylint-test]# find . -name lazy\\* .\/lib\/python3.6\/site-packages\/lazy_object_proxy-1.3.1.dist-info .\/lib\/python3.6\/site-packages\/lazy_object_proxy (pylint-test) [root@ip-172-31-26-164 pylint-test]#\">this blog post by Lee Mendelowitz<\/a> does a nice job explaining some fundamentals of package loading, and of site-packages vs. dist-packages in general.<\/p>\n<p>Just because we&#8217;ve found a difference doesn&#8217;t mean it&#8217;s a <em>meaningful<\/em> difference. In fact, it looks like all of the modules installed on Amazon Linux are in dist-packages rather than site-packages. (I find this surprising, since my understanding is that Amazon Linux is more or less RedHat derived, but that&#8217;s a different topic.)<\/p>\n<p>And indeed, packages in dist-packages generally work fine on the Amazon Linux system. For example, the &#8216;six&#8217; module:<\/p>\n<p style=\"padding-left: 30px;\"><code>(pylint-test) [root@myhost pylint-test]# pip list | grep six<br \/>\nsix        1.11.0<br \/>\n(pylint-test) [root@myhost pylint-test]# python -i<br \/>\nPython 3.6.5 (default, Apr 26 2018, 00:14:31)<br \/>\n[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux<br \/>\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.<br \/>\n&gt;&gt;&gt; import importlib.util<br \/>\n&gt;&gt;&gt; importlib.util.find_spec('six')<br \/>\nModuleSpec(name='six', loader=&lt;_frozen_importlib_external.SourceFileLoader object at 0x7f1dbed43c88&gt;, origin='\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/six.py')<br \/>\n&gt;&gt;&gt; import six<br \/>\n&gt;&gt;&gt; <\/code><\/p>\n<p>This makes sense, because pylint-test\/local\/lib\/python3.6\/dist-packages is in the pythonpath (pylint-test\/local\/lib is a symlink to pylint-test\/lib\/):<\/p>\n<p style=\"padding-left: 30px;\"><code>&gt;&gt;&gt; import sys<br \/>\n&gt;&gt;&gt; print('\\n'.join(sys.path))<br \/>\n\/root\/pylint-test\/local\/lib64\/python3.6\/site-packages<br \/>\n\/root\/pylint-test\/local\/lib\/python3.6\/site-packages<br \/>\n\/root\/pylint-test\/lib64\/python3.6<br \/>\n\/root\/pylint-test\/lib\/python3.6<br \/>\n\/root\/pylint-test\/lib64\/python3.6\/site-packages<br \/>\n\/root\/pylint-test\/lib\/python3.6\/site-packages<br \/>\n\/root\/pylint-test\/lib64\/python3.6\/lib-dynload<br \/>\n\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages<br \/>\n\/usr\/lib64\/python3.6<br \/>\n\/usr\/lib\/python3.6<br \/>\n&gt;&gt;&gt;<\/code><\/p>\n<h2>Problem Identified; Workaround<\/h2>\n<p>The problem with the lazy_object_proxy module is clear at this point: the module is being installed to a directory not in the module search path (sys.path). For some reason, pip installs it to\u00c2\u00a0.\/lib64\/python3.6\/dist-packages\/lazy_object_proxy, but neither lib64 nor local\/lib64 (which symlinks to the former) is in the module path.<\/p>\n<p>This suggests <strong>an easy workaround<\/strong> &#8211; manually adding the appropriate directory to the pythonpath should make the module loadable:<\/p>\n<p style=\"padding-left: 30px;\"><code>(pylint-test) [root@ip-172-31-44-121 pylint-test]# python -i<br \/>\nPython 3.6.5 (default, Apr 26 2018, 00:14:31)<br \/>\n[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux<br \/>\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.<br \/>\n&gt;&gt;&gt; import lazy_object_proxy<br \/>\nTraceback (most recent call last):<br \/>\nFile \"\", line 1, in<br \/>\nModuleNotFoundError: No module named 'lazy_object_proxy'<br \/>\n&gt;&gt;&gt;<br \/>\n(pylint-test) [root@ip-172-31-44-121 pylint-test]# PYTHONPATH=\".\/lib64\/python3.6\/dist-packages\/\" python -i<br \/>\nPython 3.6.5 (default, Apr 26 2018, 00:14:31)<br \/>\n[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux<br \/>\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.<br \/>\n&gt;&gt;&gt; import lazy_object_proxy<br \/>\n&gt;&gt;&gt; <\/code><\/p>\n<h2>Speculation Re: Root Cause &amp; Real Solution<\/h2>\n<p>While manually modifying the environment (PYTHONPATH) might be an effective workaround, it&#8217;s not really a\u00c2\u00a0<em>solution<\/em>. As I see it, a key question remains:\u00c2\u00a0<em>why is pip installing a package outside of the module search path<\/em>? Or perhaps pip is doing the right thing, and the question is\u00c2\u00a0<em>why isn&#8217;t lib64\/python3.6\/dist-packages<\/em> <em>in the default module search path<\/em>?<\/p>\n<p>Is this a pip bug? An error in Amazon Linux&#8217;s python packaging? Something subtly wrong in the lazy_object_proxy packaging?<\/p>\n<p>The <a href=\"https:\/\/pip.pypa.io\/en\/stable\/user_guide\/\">pip docs<\/a> don&#8217;t seem to address this question, or for that matter explain what determines the path a given module will end up in. They do suggest other possible workarounds (i.e. manually configuring pip install destinations), but those don&#8217;t seem more satisfying to me than the PYTHONPATH workaround, and don&#8217;t shed light on the remaining mystery.<\/p>\n<p>I have a gut suspicion that the core of this issue is likely related to something specific to Amazon Linux &#8211; perhaps the logic in <code>site.py<\/code> that sets the default module search path. This is very much just a hunch, but the use of dist-packages (rather than site-packages) seems odd. It&#8217;s also strange that the out-of-the-box sys.path in Amazon Linux includes\u00c2\u00a0<code>\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages<\/code> but not its <code>lib64<\/code> equivalent. I started <a href=\"https:\/\/forums.aws.amazon.com\/thread.jspa?threadID=284124\">a thread on the AWS forum<\/a> to see if anybody there can has a relevant insight.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I recently encountered a surprising problem while setting up an AWS environment: pylint\u00c2\u00a0fails to run in a python 3.6 virtualenv on Amazon Linux 2018.3! (pylint-test) [root@myhost ~]# pylint Traceback (most recent call last): File &#8220;\/root\/pylint-test\/bin\/pylint&#8221;, line 11, in sys.exit(run_pylint()) File &#8220;\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/pylint\/__init__.py&#8221;, line 15, in run_pylint from pylint.lint import Run File &#8220;\/root\/pylint-test\/local\/lib\/python3.6\/dist-packages\/pylint\/lint.py&#8221;, line 64, in import <a href='https:\/\/www.netfluvia.org\/layer8\/?p=217'>[&#8230;]<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[30,28,29],"class_list":["post-217","post","type-post","status-publish","format-standard","hentry","category-tech","tag-aws","tag-python","tag-troubleshooting","category-6-id","post-seq-1","post-parity-odd"],"_links":{"self":[{"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=\/wp\/v2\/posts\/217","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=217"}],"version-history":[{"count":1,"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=\/wp\/v2\/posts\/217\/revisions"}],"predecessor-version":[{"id":220,"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=\/wp\/v2\/posts\/217\/revisions\/220"}],"wp:attachment":[{"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=217"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=217"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.netfluvia.org\/layer8\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=217"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}