fluentd 3-minute cooking How to transfer logs to Swift object storage

Hello, I'm Sato, who has been playing around with fluentd a lot recently

We are using GMO App Cloud for a certain project, and we have received a request to use fluentd to store the logs of the running server in one-hour increments in object storage

"Object storage! It's probably S3 compatible, so it's a breeze!"

But..

GMO AppliCloud Object Storage

Object storage is operated from a customer-managed server using OpenStack's Swift-based API

What? With Swift??

I hit a snag right from the start... Although GMO AppliCloud's object storage reference has examples of operations using curl, Python-swiftclient, and Cyberduck, none of them are suitable for integrating with fluentd, and I didn't want to write my own log transfer program..

I found something useful!
fluent-plugin-swift
for creating such a useful plugin.

So

fluentd 3-minute cooking (GMO App Cloud) Swift object storage edition

Ingredients

1. Fluentd and required plugins

• fluentd(td-agent) Fresh (latest)
• fluent-plugin-swift
• fluent-plugin-forest

*This recipe is based on your preference, but it is recommended to use forest

2. Information required to access swift object storage

Please check the following from GMO App Cloud Control Panel > Services > API Information

Service information: Tenant name (20-digit number starting with "app")
*There is also information called Tenant ID, but we will not use it this time.

Endpoint: identity v2.0 (authentication)
*There are various endpoint information, but this one will be used this time

You will also need your GMO App Cloud control panel login ID/password.
Once you have all the ingredients, you can start cooking.

3. Install fluentd

the recommended pre-installation settings have been completed, install as described on the official page.

# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

4. Installing the Plugin

Let's install fluent-plugin-swift and fluent-plugin-forest

# /opt/td-agent/embedded/bin/gem install fluent-plugin-swift fluent-plugin-forest

5.Preparing the configuration file

Edit /etc/td-agent/td-agentd.conf

Definition of the source (log source) tag

This time, we define /var/log/messages with the tag syslog.messages

<source>type tail format none tag syslog.messages path /var/log/messages pos_file /var/tmp/messages.pos</source>

Match (destination swift object storage) tag definition

, the log will be saved in the object storage container (similar to a bucket in S3)
/hostname/YYYYMM/hostname_logfilename_YYYYMMDD-HH_NN.gz.

<match syslog.*>type forest subtype swift<template> auth_url https://ident-r2nd1001.app-sys.jp/v2.0/tokens auth_user "GMO App Cloud Control Panel Login ID" auth_api_key "GMO App Cloud Control Panel Login Password" auth_tenant "Tenant Name" swift_container "Swift Container Name" swift_object_key_format %{path}${hostname}_${tag_parts[0]}.${tag_parts[1]}_%{time_slice}_%{index}.%{file_extension} path %{hostname}/%Y%m/ ssl_verify false buffer_path /var/log/fluent/swift/${tag_parts[0]}.${tag_parts[1]} time_slice_format %Y%m%d-%H time_slice_wait 10m</template></match>

The container name specified in swift_container can be created in advance, but if it does not exist, it will be created automatically

6. Operation test

# td-agent -vv &

You can try running it straight away, but first let's run it in debug mode for about an hour and check the following:

• There are no errors in the format of the configuration file
• Logs are being transferred to object storage

*If td-agent is already running and you want to test only the settings you are going to add this time, prepare a configuration file with a different name (e.g. td-agent.conf.test)

# td-agent -vv -c /etc/td-agent/td-agent.conf.test &

This will allow you to test without affecting the existing td-agent's operation

If the log is transferred to object storage, the following log will be displayed

YYYY-mm-dd HH:MM:SS +0900 [info]: Put Log to Swift. container="container name" object=hostname/YYYYMM/hostname_log file name_YYYYMMDD-HH_NN.gz

7. Starting td-agent and setting it to start automatically

# chkconfig td-agent on && service td-agent start

Swift-based object storage is rare these days, but the plugin configuration is very similar to the S3 plugin, so it shouldn't be too difficult to set up

Have a good fluentd life