yubihsm2 signatures are invalid when signing ETH transactions

Question

I am trying to figure out how to get yubihsm2 to work with signing eth transactions. I have been using the python lib and so far i have had some basic setup. Below is an abbreviation of what I have

    web3_endpoint = ''

web3 = Web3(HTTPProvider(web3_endpoint))

hsm = YubiHsm.connect("http://localhost:12345")
session = hsm.create_session_derived(1, "password")
key = session.get_object(1,OBJECT.ASYMMETRIC_KEY)

#key = AsymmetricKey.generate(session, 1, "EC Key", 1, CAPABILITY.SIGN_ECDSA, ALGORITHM.EC_K256)
pub_key = key.get_public_key()
#raw_pub = pub_key.public_bytes(
#        encoding=serialization.Encoding.DER,
#        format=serialization.PublicFormat.SubjectPublicKeyInfo
#    )
raw_pub = pub_key.public_bytes(
        encoding=serialization.Encoding.X962,
        format=serialization.PublicFormat.UncompressedPoint
        )
print ("Public key (Uncompressed):\n",binascii.b2a_hex(raw_pub))
unindexPub = raw_pub[1:]
public_key_hash = Web3.keccak(unindexPub)
address_bytes = public_key_hash[-20:]
address = address_bytes.hex()
print(address)

Now so far i can consistently get the same public key and it looks correct. I then get the same public key each time. When i say correct, the formatting looks correct and is the correct number of bytes.

1. Should I be using the commented out public key formatting or the uncompressed X962 encoding that i have above.

From there, this is where things get a bit weird

transaction = {
        'to': Web3.toChecksumAddress('0x785AB1daE1b0Ee3f2412aCF55e4153A9517b07e1'),
        'gas': 21000,
        'gasPrice': Web3.toWei(5, 'gwei'),
        'value': 1,
        'nonce': 1,
        'chainId': 4,
}
serializable_transaction = serializable_unsigned_transaction_from_dict(transaction)
transaction_hash = serializable_transaction.hash()
print(transaction_hash.hex())
# sign the transaction hash and calculate v value
signature = key.sign_ecdsa(transaction_hash,hashes.SHA3_256())
r, s = ecdsa.util.sigdecode_der(signature, ecdsa.SECP256k1.generator.order())
print("r: "+str(r)+"\ns: "+str(s))

v = 28

# encode the transaction along with the full signature and send it
encoded_transaction = encode_transaction(serializable_transaction, vrs=(v, r, s))
web3.eth.sendRawTransaction(encoded_transaction)

I am settings v to 28.. I also test it with 27.. I could use the correct amount with the chainid.. but it's not necessary right from the perspective of trying to get a valid signature (recoverable to get the same public key each time).

Sometimes I am getting the error "invalid sender" and other times I am getting the error "insufficient gas."

If I take the signature output and use a javascript lib to try to find the public key, each time I am getting a different public key. But I keep consistently generating the same public key from the yubihsm2 in this python app.

I have also commented out in sign_ecdsa the hashing function as I am passing in the data already hashed (in order to use keccak256).

Is there something I am missing? Why are these transactions not signing correctly for eth?

I am getting some of those serialization helpers from Yubico/python-yubihsm and eth-account/eth_account/_utils/legacy_transactions.py