How to Ship Container logs using Filebeat


Use Filebeat to stream files from under the Docker Root Dir on the host

Release or Environment

Sophie version 3.5.x - 3.7.x


Before starting, consider using the collector-container.

Step 1 - Install Filebeat

Download and install Filebeat. We recommend using the deb/rpm  option, but either works.

Step 2 - Determining the location of the logs

First, determine the Root Dir of Docker. By default it's /var/lib/docker, but that's sometimes customised.
Run the following command:

docker info | grep "Docker Root Dir"

Next, determine the location of the logs under that folder. This depends on the Storage driver being used. Run the following command:

find /var/lib/docker -name "*.log"

Look for the files you want to have shipped, and make note of the path.

Step 3 - Configure Filebeat

cd  to the Filebeat folder. It's probably /etc/filebeat, but checkout the directory-layout documentation if it's not there.
Open the filebeat.yml file for editing, and replace the content with the following:

- type: log
  enabled: true
    - <logs-path>/**/*.log
    # loom-application: MyApp
    # loom-service: MySvc
    # loom-sourcetype: MyStp
  ignore_older: 6h
  enabled: true
  slow_start: true
  hosts: ["<data-domain>"]

Replace <logs-path>  with the path you made note of in step 2 (e.g. /var/lib/docker/devicemapper/mnt).
Replace <data-domain>  with your data domain, e.g.
Optional: uncomment the fields entries, specifying the application, service or source-types you want the collected data to be mapped to.

Step 4 - Restart Filebeat

Run the following command:

# debian
service filebeat restart

# rpm
systemctl restart filebeat.service

# Windows, under Powershell
Restart-Service -Name Filebeat

That's it! Open Sophie and see your logs streaming in 👌