I have code to connect using sftp with private key. From the same host where i run this code I am able to connect manually to sftp nevertheless getting error when running from code. What could be a problem here?
import pysftp
def run(**kwargs):
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
try:
sftp = pysftp.Connection(host="xxxx", username="xxx", private_key="/opt/airflow/.ssh/id_rsa", cnopts=cnopts)
except:
raise
run()
Error: (note that i replaced confidentials information like username and host by xxx)
[2023-12-07, 12:27:52 UTC] {warnings.py:109} WARNING - /home/***/.local/lib/python3.10/site-packages/pysftp/__init__.py:61: UserWarning: Failed to load HostKeys from /home/***/.ssh/known_hosts. You will need to explicitly load HostKeys (cnopts.hostkeys.load(filename)) or disableHostKey checking (cnopts.hostkeys = None).
warnings.warn(wmsg, UserWarning)
[2023-12-07, 12:27:53 UTC] {transport.py:1893} INFO - Connected (version 2.0, client OpenSSH_6.7)
[2023-12-07, 12:27:54 UTC] {transport.py:1893} INFO - Authentication (publickey) failed.
[2023-12-07, 12:27:54 UTC] {logging_mixin.py:150} INFO - 2023-12-07 12:27:54.521613: ('Traceback (most recent call last):\n'
' File "/opt/***/dags/customer_dag.py", line 161, in customer_name\n'
' sftp = pysftp.Connection(host="xxx", '
'username="xxx", private_key="/opt/***/.ssh/id_rsa", '
'cnopts=cnopts)\n'
' File '
'"/home/***/.local/lib/python3.10/site-packages/pysftp/__init__.py", line '
'143, in __init__\n'
' self._transport.connect(**self._tconnect)\n'
' File '
'"/home/***/.local/lib/python3.10/site-packages/paramiko/transport.py", '
'line 1411, in connect\n'
' self.auth_publickey(username, pkey)\n'
' File '
'"/home/***/.local/lib/python3.10/site-packages/paramiko/transport.py", '
'line 1658, in auth_publickey\n'
' return self.auth_handler.wait_for_response(my_event)\n'
' File '
'"/home/***/.local/lib/python3.10/site-packages/paramiko/auth_handler.py", '
'line 263, in wait_for_response\n'
' raise e\n'
'paramiko.ssh_exception.AuthenticationException: Authentication failed.\n')
[2023-12-07, 12:27:54 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 181, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 198, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/customer_dag.py", line 161, in customer_name
sftp = pysftp.Connection(host="xxx", username="xxx", private_key="/opt/airflow/.ssh/id_rsa", cnopts=cnopts)
File "/home/airflow/.local/lib/python3.10/site-packages/pysftp/__init__.py", line 143, in __init__
self._transport.connect(**self._tconnect)
File "/home/airflow/.local/lib/python3.10/site-packages/paramiko/transport.py", line 1411, in connect
self.auth_publickey(username, pkey)
File "/home/airflow/.local/lib/python3.10/site-packages/paramiko/transport.py", line 1658, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "/home/airflow/.local/lib/python3.10/site-packages/paramiko/auth_handler.py", line 263, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
[2023-12-07, 12:27:54 UTC] {taskinstance.py:1345} INFO - Marking task as FAILED. dag_id=customer_name, task_id=task_customer_name, execution_date=20231207T122750, start_date=20231207T122752, end_date=20231207T122754
[2023-12-07, 12:27:54 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 206 for task task_customer_name (Authentication failed.; 96)
[2023-12-07, 12:27:54 UTC] {local_task_job_runner.py:225} INFO - Task exited with return code 1
[2023-12-07, 12:27:54 UTC] {taskinstance.py:2653} INFO - 0 downstream tasks scheduled from follow-on schedule check
Log from: (cnopts.log = True)
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - starting thread (client mode): 0xce8f2620
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - Local version/idstring: SSH-2.0-paramiko_3.2.0
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - Remote version/idstring: SSH-2.0-OpenSSH_6.7
[2023-12-07, 13:27:43 UTC] {transport.py:1893} INFO - Connected (version 2.0, client OpenSSH_6.7)
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - === Key exchange possibilities ===
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - kex algos: [email protected], ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group14-sha1
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - server key: ssh-rsa, ssh-dss
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - client encrypt: aes128-ctr, aes192-ctr, aes256-ctr, [email protected], [email protected], [email protected]
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - server encrypt: aes128-ctr, aes192-ctr, aes256-ctr, [email protected], [email protected], [email protected]
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - client mac: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], hmac-sha2-256, hmac-sha2-512, hmac-sha1
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - server mac: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], hmac-sha2-256, hmac-sha2-512, hmac-sha1
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - client compress: none, [email protected]
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - server compress: none, [email protected]
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - client lang: <none>
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - server lang: <none>
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - kex follows: False
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - === Key exchange agreements ===
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - Kex: [email protected]
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - HostKey: ssh-rsa
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - Cipher: aes128-ctr
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - MAC: hmac-sha2-256
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - Compression: none
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - === End of kex handshake ===
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - Switch to new keys ...
[2023-12-07, 13:27:43 UTC] {transport.py:1893} DEBUG - Attempting public-key auth...
[2023-12-07, 13:27:44 UTC] {transport.py:1893} DEBUG - userauth is OK
[2023-12-07, 13:27:44 UTC] {transport.py:1893} DEBUG - Finalizing pubkey algorithm for key of type 'ssh-rsa'
[2023-12-07, 13:27:44 UTC] {transport.py:1893} DEBUG - Our pubkey algorithm list: ['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa']
[2023-12-07, 13:27:44 UTC] {transport.py:1893} DEBUG - Server did not send a server-sig-algs list; defaulting to our first preferred algo ('rsa-sha2-512')
[2023-12-07, 13:27:44 UTC] {transport.py:1893} DEBUG - NOTE: you may use the 'disabled_algorithms' SSHClient/Transport init kwarg to disable that or other algorithms if your server does not support them!
[2023-12-07, 13:27:44 UTC] {transport.py:1893} INFO - Authentication (publickey) failed.
[2023-12-07, 13:27:44 UTC] {logging_mixin.py:150} INFO - 2023-12-07 13:27:44.766480: ('Traceback (most recent call last):\n'
' File "/opt/***/dags/customer_dag.py", line 162, in customer_name\n'
' sftp = pysftp.Connection(host="xxx", '
'username="mnsgsd", private_key="/opt/***/.ssh/id_rsa", '
'cnopts=cnopts)\n'
' File '
'"/home/***/.local/lib/python3.10/site-packages/pysftp/__init__.py", line '
'143, in __init__\n'
' self._transport.connect(**self._tconnect)\n'
' File '
'"/home/***/.local/lib/python3.10/site-packages/paramiko/transport.py", '
'line 1411, in connect\n'
' self.auth_publickey(username, pkey)\n'
' File '
'"/home/***/.local/lib/python3.10/site-packages/paramiko/transport.py", '
'line 1658, in auth_publickey\n'
' return self.auth_handler.wait_for_response(my_event)\n'
' File '
'"/home/***/.local/lib/python3.10/site-packages/paramiko/auth_handler.py", '
'line 263, in wait_for_response\n'
' raise e\n'
'paramiko.ssh_exception.AuthenticationException: Authentication failed.\n')
[2023-12-07, 13:27:44 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 181, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 198, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/customer_dag.py", line 162, in customer_name
sftp = pysftp.Connection(host="xxx", username="mnsgsd", private_key="/opt/airflow/.ssh/id_rsa", cnopts=cnopts)
File "/home/airflow/.local/lib/python3.10/site-packages/pysftp/__init__.py", line 143, in __init__
self._transport.connect(**self._tconnect)
File "/home/airflow/.local/lib/python3.10/site-packages/paramiko/transport.py", line 1411, in connect
self.auth_publickey(username, pkey)
File "/home/airflow/.local/lib/python3.10/site-packages/paramiko/transport.py", line 1658, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "/home/airflow/.local/lib/python3.10/site-packages/paramiko/auth_handler.py", line 263, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
[2023-12-07, 13:27:44 UTC] {taskinstance.py:1345} INFO - Marking task as FAILED. dag_id=customer_name, task_id=task_customer_name, execution_date=20231207T132738, start_date=20231207T132742, end_date=20231207T132744
[2023-12-07, 13:27:44 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 210 for task task_customer_name (Authentication failed.; 91)
[2023-12-07, 13:27:44 UTC] {local_task_job_runner.py:225} INFO - Task exited with return code 1
[2023-12-07, 13:27:44 UTC] {taskinstance.py:2653} INFO - 0 downstream tasks scheduled from follow-on schedule check