Kubernetes ingress redirects to 504

Question

I'm trying to learn kubernetes with a couple of rpi's at home. I'm trying to run pihole in the cluster, which has worked, now the issue i'm facing is a redirect issue with ingress.

my ingress.yaml file output:

## pihole.ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: pihole
  name: pihole-ingress
  annotations:
    # use the shared ingress-nginx
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: / 
spec:
  rules:
  - host: pihole.192.168.1.230.nip.io
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: pihole-web
            port:
              number: 80

output of kubectl describe ingress:

Name:             pihole-ingress
Namespace:        pihole
Address:          192.168.1.230
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                         Path  Backends
  ----                         ----  --------
  pihole.192.168.1.230.nip.io  
                               /   pihole-web:80 (10.42.2.7:80)
Annotations:                   kubernetes.io/ingress.class: nginx
                               nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason  Age                 From                      Message
  ----    ------  ----                ----                      -------
  Normal  Sync    18s (x12 over 11h)  nginx-ingress-controller  Scheduled for sync

Output of get svc -n ingress-nginx

NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller-admission   ClusterIP      10.43.240.186   <none>          443/TCP                      22h
ingress-nginx-controller             LoadBalancer   10.43.64.54     192.168.1.230   80:31093/TCP,443:30179/TCP   22h

I'm able to get into the pod and curl the cluster ip to get the output i expect, but when i try to visit pihole.192.168.1.230, i get a 504 error. Hoping anyone can assist with my ingress to redirect to the pihole-web service. Please let me know if there's any additional information i can provide.

EDIT:

kubectl get po -n pihole -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP          NODE         NOMINATED NODE   READINESS GATES
pihole-7d4dc6b8d8-vclxz   1/1     Running   0          9h    10.42.2.8   node02.iad   <none>           <none>

kubectl get svc -n pihole
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
pihole-web       ClusterIP   10.43.102.198   <none>        80/TCP,443/TCP   9h
pihole-dhcp      NodePort    10.43.191.110   <none>        67:32021/UDP     9h
pihole-dns-udp   NodePort    10.43.214.15    <none>        53:31153/UDP     9h
pihole-dns-tcp   NodePort    10.43.168.6     <none>        53:32754/TCP     9h

another edit: since this question was originally posted, and the above edit was made, pihole pod ip was changed from 10.42.2.7 to 10.42.2.8

I checked the logs for the ingress controller and saw the following. Hoping someone can help me decipher this:

2021/09/03 17:52:35 [error] 1938#1938: *3132346 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.42.1.1, server: pihole.192.168.1.230.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.42.2.8:80/", host: "pihole.192.168.1.230.nip.io", referrer: "http://pihole.192.168.1.230.nip.io/"