EDIT1: intershop.urlrewrite.CheckSource is turned off already
We are recently having quite big problems with URL rewrite rules not being loaded in test and production multi-node environments. The problem started happening after introducing another organization and it's related application onto the servers. From then on we have tried multiple changes and debugging methods to try to figure it out, but without any result. Also the major problem is that it doesn't happen all the time and server restart can fix it but not always.
Here are the details so far how the problem manifests (this has been going on for more than a month now on our production system):
- Most of the time it starts happening after the deploy of new code and starting up the server
- Then multiple people from multiple computers and locations try opening the website and some open it and others get either 404 or "URL invalid" page, so its 50/50.
- On the PC where someone successfully opens page, if you try again in incognito mode then you may get again 404(probably connects to another node/appserver).
- Usually the problem is resolved either by server restart or by restarting a single node(no code or configuration changes) although this is no reliable way and on the last occurrence we tried multiple restarts and it didn't help. After a few days one of the team members restarted only a single node for debugging purposes and then it started working normally again.
After setting up more detailed log messages and turning on debug messages for URL rewrite classes we have come to the conclusion that the rule loading fails. We have come to this conclusion because we have added debug message on the very start of our applyExpand() method and it never gets shown. This can be observed on the image below:
All of this leads to conclusion that iterator on line 149 is empty.
Please advise on possible causes of this problem and how to resolve it.
With the help of IS support, we have managed to figure out that the problem was that the URL rewrite rules were in a cartridge that wasn't part of every possible application on server which would result in undefined behavior when loading them (it would load on one appserver and on another it wouldn't).
The fix was to add a new common cartridge for all possible applications which would then hold urlrewrite rules and which would definitely be loaded on server startup.