AWS Greengrass does not generate user logs

Question

On a pretty new fresh install using AWS's getting started guide, I'm not getting user logs (in /greengrass/ggc/var/log/user), which makes it impossible to do any Greengrass lambda development.

Here are some symptoms:

• /greengrass/ggc/var/log/user is not being created (and when I manually create it and the subdirectories below it, user/us-east-1/, greengrassd blows away all those subdirectories)
• There's a message in /greengrass/ggc/var/log/system/localwatch/localwatch.log, failed to list the user log directory, path: /greengrass/ggc/var/log/user
• There are no other error messages in runtime.log, crash.log, or any of the other system logs.

I have a bunch of theories, but I'm not able to confirm this:

• I've been swapping out SD cards on the same RPi, each with different versions of the Raspbian (Jessie, Stretch, different kernel versions) and different Greengrass groups on them. Could this swapping of SD cards be a cause (eg., either AWS or some other service they use like overlayFS be taking a fingerprint of the hardware)? On another RPi which I haven't done this swapping, I don't seem to have a problem.

• Is there an issue with IAM settings for the group role? I would have thought that IAM only applies to cloud services, and not Greengrass services on a gateway.

Any insights anyone might have on this would be really appreciated!!!

Answer 1

I've only had this happen to me once. For me, a clean install of GG and core on a new system helped and removed all of the bugs. That said, there is one place in the console that you should check to verify that logs are enabled:

- Under Greengrass Group, select the group, then go to "settings" from there you'll see "Local logs configuration." Make sure that you have those on and re-deploy to activate.

As far as debugging lambda code, I recommend "python-lambda-local" to help debug lambda code for python. Also, don't forget that some of those file locations need sudo to be able to see the files and subfolders.

Answer 2

Ultimately, it turns out that this was a configuration error, but I was thrown by the error message in localwatch.log (as I mentioned in my initial post).

What I had done was to set up a number of lambdas running on the RPi to immediately report (via console.logs) what's going on, and I set two of the lambdas to "Make this function long-lived and keep it running indefinitely" (in the "Group-specific lambda configuration" on the Greengrass UI). As a result, every time the Greengrass daemon starts up, there's something going into the user log.

For my second RPi, I configured the lambdas manually, but forgot to make them "long-lived" and so they started off event-based, so no lambdas run by default. When I go into the user log directory, I find nothing there, and when I poked into the localwatch.log, I got the message "failed to list the user log directory, path: /greengrass/ggc/var/log/user" which seems to suggest that Greengrass was having difficulty access the user log directory structure. That sent me on a wild goose chase.

The fix was simply to go into the GG UI and set these two lambdas to be long-lived, and once the lambdas started running, the user log directories (and then subsequently the logs) were created.

Thanks again for the response, J. Benson. Good to see others actively using Greengrass, and hopefully, we'll start building a community of users who can help each other. Note that I also posted this on AWS Forum, but marked it as resolved.

Answer 3

As an update if anybody else is having this issue, I was running into the same error using the ggc docker image. As I really didn't wish to do a clean install, i found out that:

 1.Remove all log settings from the greengrass core settings page
 2.Deploy
 3.Remake the settings to match the logs you want to have
 4.Deploy

I'm not sure if the second deploy in the middle there does anything but after doing this, I had the user folder ready and set with logs working.