Debugging and troubleshooting singer-tap development for a SQL source using the Meltano SDK

Question

Does anyone have experience developing SQL singer-taps using the Meltano SDK?

Problem

My biggest problem is troubleshooting the exceptions I am experiencing when testing my tap.

Context

• I am developing in an Ubuntu environment running on a windows machine using WSL2.
• I have successfully established connection to the source using the custom sqlalchemy dialect I found for Progress Openedge (view here) on the ubuntu machine with a url created using this code:

pyodbc_connstr = urllib.parse.quote_plus(r"""
    DRIVER={DataDirect 7.1 OpenEdge Wire Protocol};
    HOST=xxxx;
    PORT=xxxx;
    DB=xxxxx;
    UID=sysprogress;
    PWD=;
    DEFAULTSCHEMA=PUB;
""")                                         

sa_str = "progress+pyodbc:///?odbc_connect={}".format(pyodbc_connstr)

• I am referring to this meltano article as guidance for setting up the project.
• I've set up the project with the following responses to the cookie cutter prompts:
  • Stream Type = 3 - SQL
  • Auth Type = 6 - Custom or N/A
• I've installed the progress_sa SQLAlchemy dialect into the project from the local directory on the Ubuntu machine using poetry add <local path to package>
• I've created a .env file for testing the tap.
• All developer TODO areas of the SDK are complete.

Where I'm stuck

When running poetry run tap-<my tap> --config= ENV I get the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 493, in invoke
    tap = cls(
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 625, in __init__
    super().__init__(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 102, in __init__
    self.setup_mapper()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 172, in setup_mapper
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 165, in catalog
    self._catalog = self.input_catalog or self._singer_catalog
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 318, in _singer_catalog
    for stream in self.streams.values()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 127, in streams
    for stream in self.load_streams():
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 351, in load_streams
    for stream in self.discover_streams():
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 655, in discover_streams
    for catalog_entry in self.catalog_dict["streams"]:
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 643, in catalog_dict
    result["streams"].extend(connector.discover_catalog_entries())
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 497, in discover_catalog_entries
    engine = self._engine
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 305, in _engine
    self._cached_engine = self.create_engine()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 322, in create_engine
    return sqlalchemy.create_engine(
  File "<string>", line 2, in create_engine
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 680, in create_engine
    raise TypeError(
TypeError: Invalid argument(s) 'json_serializer','json_deserializer' sent to create_engine(), using configuration Progress_pyodbc/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components. 

I admittedly am still a developing python programmer, and so I am still learning how to effectively troubleshoot.

It is hard to tell if the issue is caused by something that I've done in setting up the tap or not.

Answer 1

This was addressed by the v0.32.0 release of the Singer SDK. In the particular by https://github.com/meltano/sdk/pull/1949.