ImportError: No module named [name].items

Question

I am attempting to run my own scrapy project. I thought I resolved a related issue in a thread I posted here:[urlparse: ModuleNotFoundError, presumably in Python2.7 and under conda

I did a complete system image restore and simply installed Python 2.7 and Miniconda. However, Atom Editor is still flagging/underlining 'import urlparse'.

The code is based off a well written book and the author provides a great VM playground to run scripts exampled in the book. In the VM the code works fine.

However, in an attempt to practice on my own, I now receive the following error:

(p2env) C:\Users\User-1\Desktop\scrapy_projects\dictionary>scrapy crawl basic Traceback (most recent call last): File "C:\Users\User-1\Miniconda2\envs\p2env\Scripts\scrapy-script.py", line 5, in <module> sys.exit(scrapy.cmdline.execute()) File "C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\cmdline.py", line 148, in execute cmd.crawler_process = CrawlerProcess(settings) File "C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\crawler.py", line 243, in __init__ super(CrawlerProcess, self).__init__(settings) File "C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\crawler.py", line 134, in __init__ self.spider_loader = _get_spider_loader(settings) File "C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\crawler.py", line 330, in _get_spider_loader return loader_cls.from_settings(settings.frozencopy()) File "C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\spiderloader.py", line 61, in from_settings return cls(settings) File "C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\spiderloader.py", line 25, in __init__ self._load_all_spiders() File "C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\spiderloader.py", line 47, in _load_all_spiders for module in walk_modules(name): File "C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\utils\misc.py", line 71, in walk_modules submod = import_module(fullpath) File "C:\Users\User-1\Miniconda2\envs\p2env\lib\importlib\__init__.py", line 37, in import_module __import__(name) File "C:\Users\User-1\Desktop\scrapy_projects\dictionary\dictionary\spiders\basic.py", line 11, in <module> from terms.items import TermsItem ImportError: No module named terms.items

My folder hierarchy is as follows:

└───dictionary
│   scrapy.cfg
│
└───dictionary
    │   items.py
    │   middlewares.py
    │   pipelines.py
    │   settings.py
    │   settings.pyc
    │   __init__.py
    │   __init__.pyc
    │
    └───spiders
            basic.py
            basic.pyc
            __init__.py
            __init__.pyc

My items.py code is as follows:

# -*- coding: utf-8 -*-

from scrapy.item import Item, Field


class TermsItem(Item):
    # Primary fields
    title = Field()
    definition = Field()
    # Housekeeping fields
    url = Field()
    project = Field()
    spider = Field()
    server = Field()
    date = Field()

My spider.py is as follows:

# -*- coding: utf-8 -*-

import datetime
import urlparse
import socket
import scrapy

from scrapy.loader.processors import MapCompose, Join
from scrapy.loader import ItemLoader

from terms.items import TermsItem


class BasicSpider(scrapy.Spider):
    name = "basic"
    allowed_domains = ["web"]

    # Start on a property page
    start_urls = (
        'http://dictionary.com/browse/there',
    )

    def parse(self, response):
        # Create the loader using the response
        l = ItemLoader(item=TermsItem(), response=response)

        # Load fields using XPath expressions
        l.add_xpath('title', '//h1[@class="head-entry"][1] / text()',
                    MapCompose(unicode.strip, unicode.title))
        l.add_xpath('definition', '//*[@class="def-list"][1]/text()',
                    MapCompose(unicode.strip, unicode.title))

        # Housekeeping fields
        l.add_value('url', response.url)
        l.add_value('project', self.settings.get('BOT_NAME'))
        l.add_value('spider', self.name)
        l.add_value('server', socket.gethostname())
        l.add_value('date', datetime.datetime.now())

        return l.load_item()

Atom Editor still flags 'import urlparse' and 'from scrapy.loader.processors import MapCompose, Join'

Based on this stackoverflow question: Scrapy ImportError: No module named Item where coders are instructed to '**execute the Scrapy command from inside the top level directory of your project. – alecxe'** has me wondering if the conda environment I am using is causing the error? No module named items stack question has a similar point '**What is doing the import? And what is the working directory/the contents of sys.path. You can't find Project_L if the parent directory isn't the working directory and doesn't appear in sys.path. – ShadowRanger May 11 at 22:24'** However, to the best of my knowledge I am structuring the project correctly and the corresponding hierarchy is correct.

Any help would be greatly appreciated. Apologies for the lengthy post, I just wanted to be as comprehensive as possible and make sure that people appreciate the difference between this question and the similar ones I have linked to.

Regards,

Answer 1

The backtrace indicates the problem is in a specific import. You can verify this from the command line. For example on my machine I get

$ python -c 'from terms.items import TermsItem'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named terms.items

Looking at your folder hierarchy I see no called "terms" module so that's probably what you're missing but since you indicate the code is working in the author's VM what I would do is try running the following command in that VM:

$ python -v -c 'from terms.items import TermsItem'

The -v option will cause python to show you all the paths being imported.
e.g.

$ python -v -c 'from terms.items import TermsItem'
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /usr/local/var/pyenv/versions/2.7.12/lib/python2.7/site.pyc matches /usr/local/var/pyenv/versions/2.7.12/lib/python2.7/site.py
import site # precompiled from /usr/local/var/pyenv/versions/2.7.12/lib/python2.7/site.pyc
...
import encodings.ascii # precompiled from /usr/local/var/pyenv/versions/2.7.12/lib/python2.7/encodings/ascii.pyc
Python 2.7.12 (default, Nov 29 2016, 14:57:54) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named terms.items
# clear __builtin__._
# clear sys.path
...
# cleanup ints: 20 unfreed ints
# cleanup floats

If you do that where the code is working then somewhere in that output will be a successful import. From that you may be able to work out the name of the missing module on your system and install it accordingly.

EDIT: looking closer at your post I notice you mention your "items.py" contains

class TermsItem(Item):
    # Primary fields
    ...

so I suspect your problem is that your import should be

from items import TermsItem

Answer 2

Succinctly, I was able to produce a solution and bypass the error by replacing:

from terms.items import TermsItem

with

from ..items import TermsItem

This solution and its variations has already been provided in a link I initially enclosed above:

Scrapy ImportError: No module named Item

I am certain that I tried it initially, however I suspect when I attempted to re-run the code I must have forgotten to save, and thus was pulling from the prior .pyc

It Should be noted that the book in which the code is based off of, and the aforementioned link Scrapy ImportError: No module named Item indicate that my original code

from terms.items import TermsItem

should be working. I strongly recommend to those who are encountering the same error, who have already tried the solutions provided in the related stackflow questions I linked, and have made sure to differentiate the parent directory name from the spider's name, to review the comments made in this thread by jq170727. The individual provides great instruction on how to use the -v option to better identify the issue and has included a link to a very detailed error tracing method he has documented which makes use of the PYTHONVERBOSE command. I am a total newb, and thus was unable at this point to fully appreciate what the implications are, however, I will post what I believe to be the issue traced from jq170727's method:

import scrapy.loader.processors # precompiled from C:\Users\User-1\Miniconda2\envs\p2env\lib\site-packages\scrapy\loader\processors.pyc
Traceback (most recent call last):

I believe this is arising due to the use of a conda enviroment that I am using to currently house the Python2.7 and Scrapy that I am using. Again, in the full thread I link and quote other stackflow contributor's related points which I think may allow someone to fully correct their problem (as I am still very green at this).