My website is a static html photo gallery which has more than 100,000 html pages with no database, just simply html (old website, not a adult site)
I used to create sitemap and find broken links using Xenu application and I did skipped images (*.jpg) when creating a sitemap before but now (after some 4 years) its not working or I forget what I added in exclude options.
I used below formats to exclude/skip images to add in generating list/checking.
*.jpg*
*jpg*
*.jpeg*
*jpeg*
But its not working.
I really need to skip images because if there is 100k html pages, there is 100k HQ photos with 100k thumbnails :( and all this will make bigger sitemap and I don't want my direct photo links to index in search engine.
Please help. Any possible ways to skip "image/jpeg" ?
Thank you very much.
I just had a similar requirement, and it turns out that Xenu have a "wildcard edition" that works with regex. It is available on their website.