Difference between **journalctl -u test.service** and **journalctl CONTAINER_NAME=test**

Question

I have a systemd service file which run a docker container with log driver journald.

ExecStart=/usr/bin/docker run \
    --name ${CONTAINER_NAME} \
    -p ${PORT}:8080 \
    --add-host ${DNS} \
    -v /etc/localtime:/etc/localtime:ro \
    --log-driver=journald \
    --log-opt tag="docker.{{.Name}}" \
    ${RESPOSITORY_NAME}/${CONTAINER_NAME}

ExecStop=-/usr/bin/docker stop ${CONTAINER_NAME}

When I check the logs via journalctl I see two different _TRANSPORT. With journalctl -u test.service I see _TRANSPORT=stdout. And with Journalctl CONTAINER_NAME=test I see _TRANSPORT=journal

What is the difference?

Answer 1

The difference here is in how the logs get to systemd-journald before they are logged.

As of right now, the supported transports (at least according to the _TRANSPORT field in systemd-journald) are: audit, driver, syslog, journal, stdout and kernel (see systemd.journal-fields(7)).

In your case, everything logged to stdout by commands executed by the ExecStart= and ExecStop= directives is logged under the _TRANSPORT=stdout transport.

However, Docker is internally capable of using the journald logging driver which, among other things, introduces several custom journal fields - one of them being CONTAINER_ID=. It's just a different method of delivering data to systemd-journald - instead of relying on systemd to catch and send everything from stdout to systemd-journald, Docker internally sends everything straight to systemd-journald by itself.

This can be achieved by using the sd-journal API (as described in sd-journal(3)). Docker uses the go-systemd Go bindings for the sd-journal C library.

Simple example:

hello.c

#include <stdio.h>
#include <systemd/sd-journal.h>

int main(void)
{
    printf("Hello from stdout\n");
    sd_journal_print(LOG_INFO, "Hello from journald");

    return 0;
}

# gcc -o /var/tmp/hello -lsystemd hello.c
# cat > /etc/systemd/system/hello.service << EOF
[Service]
ExecStart=/var/tmp/hello
EOF

# systemctl daemon-reload
# systemctl start test.service

Now if I check journal, I'll see both messages:

# journalctl -u hello.service
-- Logs begin at Mon 2019-09-30 22:08:02 CEST, end at Fri 2020-03-27 17:11:29 CET. --
Mar 27 17:08:28 localhost systemd[1]: Started hello.service.
Mar 27 17:08:28 localhost hello[921852]: Hello from journald
Mar 27 17:08:28 localhost hello[921852]: Hello from stdout
Mar 27 17:08:28 localhost systemd[1]: hello.service: Succeeded.

But each of them arrived using a different transport:

# journalctl -u hello.service _TRANSPORT=stdout
-- Logs begin at Mon 2019-09-30 22:08:02 CEST, end at Fri 2020-03-27 17:12:29 CET. --
Mar 27 17:08:28 localhost hello[921852]: Hello from stdout

# journalctl -u hello.service _TRANSPORT=journal
-- Logs begin at Mon 2019-09-30 22:08:02 CEST, end at Fri 2020-03-27 17:12:29 CET. --
Mar 27 17:08:28 localhost systemd[1]: Started hello.service.
Mar 27 17:08:28 localhost hello[921852]: Hello from journald
Mar 27 17:08:28 localhost systemd[1]: hello.service: Succeeded.