On a pretty new fresh install using AWS's getting started guide, I'm not getting user logs (in /greengrass/ggc/var/log/user), which makes it impossible to do any Greengrass lambda development.
Here are some symptoms:
- /greengrass/ggc/var/log/user is not being created (and when I manually create it and the subdirectories below it, user/us-east-1/, greengrassd blows away all those subdirectories)
- There's a message in /greengrass/ggc/var/log/system/localwatch/localwatch.log,
failed to list the user log directory, path: /greengrass/ggc/var/log/user
- There are no other error messages in runtime.log, crash.log, or any of the other system logs.
I have a bunch of theories, but I'm not able to confirm this:
I've been swapping out SD cards on the same RPi, each with different versions of the Raspbian (Jessie, Stretch, different kernel versions) and different Greengrass groups on them. Could this swapping of SD cards be a cause (eg., either AWS or some other service they use like overlayFS be taking a fingerprint of the hardware)? On another RPi which I haven't done this swapping, I don't seem to have a problem.
Is there an issue with IAM settings for the group role? I would have thought that IAM only applies to cloud services, and not Greengrass services on a gateway.
Any insights anyone might have on this would be really appreciated!!!
Ultimately, it turns out that this was a configuration error, but I was thrown by the error message in localwatch.log (as I mentioned in my initial post).
What I had done was to set up a number of lambdas running on the RPi to immediately report (via console.logs) what's going on, and I set two of the lambdas to "Make this function long-lived and keep it running indefinitely" (in the "Group-specific lambda configuration" on the Greengrass UI). As a result, every time the Greengrass daemon starts up, there's something going into the user log.
For my second RPi, I configured the lambdas manually, but forgot to make them "long-lived" and so they started off event-based, so no lambdas run by default. When I go into the user log directory, I find nothing there, and when I poked into the localwatch.log, I got the message
"failed to list the user log directory, path: /greengrass/ggc/var/log/user"
which seems to suggest that Greengrass was having difficulty access the user log directory structure. That sent me on a wild goose chase.The fix was simply to go into the GG UI and set these two lambdas to be long-lived, and once the lambdas started running, the user log directories (and then subsequently the logs) were created.
Thanks again for the response, J. Benson. Good to see others actively using Greengrass, and hopefully, we'll start building a community of users who can help each other. Note that I also posted this on AWS Forum, but marked it as resolved.