I have two documents, a .docx and an .xlsx both are in the same folder (Document Library) on the MOSS site and are linked in a content page. The .docx downloads as expected. When you click on the .xlsx file it prompts for a username/password.
The documents are both "Published" so that wasn't the issue.
Any ideas?
Comment the xlsx extension from file serverfilesExcelServer.xml in location:
C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\TEMPLATE\XML
solution found here: open email link to xlsx (excel 2007) file in MOSS 2007 document library.