DatePicker with additional buttons to modify date - incorrect display format

Question

I'm trying to create a View to allow me to tap a back arrow to go back a day, tap a forward arrow to go forward a day, and tap a date picker to select a specific date more quickly. The UI looks like this:

This works, but has a strange side effect. Occasionally (sometimes every other date, but there are large blocks that don't have the issue), the date format shown will change to this:

I have no idea what could cause this, but I imagine I'm not altering the date correctly in some way. The code for the View is below:

import SwiftUI

struct DateSelector: View {
    @State var date: Date

    var body: some View {
        VStack {
            Rectangle()
                .fill(.gray)
                .frame(maxWidth: .infinity, minHeight: 1, idealHeight: 1, maxHeight: 1)
        
            HStack {
                Spacer()
            
                Button {
                    let calendar = Calendar.current
                    date = calendar.date(byAdding: .day, value: -1, to: date)!
                } label: {
                    Image(systemName: "arrowtriangle.backward.fill")
                }
            
                Group {
                    Spacer()
                    Spacer()
                    Spacer()
                }
            
                DatePicker("Selected Date", selection: $date, displayedComponents: [.date])
                    .labelsHidden()
            
                Group {
                    Spacer()
                    Spacer()
                    Spacer()
                }
            
                Button {
                    let calendar = Calendar.current
                    date = calendar.date(byAdding: .day, value: 1, to: date)!
                } label: {
                    Image(systemName: "arrowtriangle.forward.fill")
                }
            
                Spacer()
            }
            .padding(0.1)
        
            Rectangle()
                .fill(.gray)
                .frame(maxWidth: .infinity, minHeight: 1, idealHeight: 1, maxHeight: 1)
        }
    }
}