fluentd 3 minute cooking how to forward logs to swift object storage

Hello, I'm Sato, and I've been playing with fluentd a lot lately.

We are using GMO App Cloud for a certain project, and we received a request to use fluentd to divide the logs of the running server into hourly chunks and save them in object storage.

"It's object storage! It's probably compatible with S3, so it's an easy win!"

I made a big deal out of it. . . . .

GMO App Cloud Object Storage

Object storage is operated using OpenStack's Swift-based API from a customer-managed server.

what? What about Swift? ?

I suddenly stumbled from the start... GMO App Cloud's object storage reference has operation examples using curl, Python-swiftclient, and Cyberduck, but none of them are suitable for linking with fluentd, and I have to log on my own. I was thinking about putting together a transfer program. . .

There was something useful!
fluent-plugin-swift
I would like to thank the author for creating a plug-in that can help with this difficult area.

That's why

fluentd 3 minute cooking (GMO App Cloud) swift object storage edition

It is the material

1.fluentd and required plugins

• fluentd(td-agent) fresh (latest version)
• fluent-plugin-swift
• fluent-plugin-forest

*This is your preference, but this recipe assumes the use of forest.

2. Information required to access swift object storage

Please check the following from GMO App Cloud Control Panel > Services > API Information

Service information: Tenant name 20-digit number starting with app
*There is also information called tenant ID, but we will not use it this time

Endpoint: identity v2.0 (authentication)
*There are various endpoint information, but this time we will use this one.

You will also need the login ID/password for the GMO App Cloud control panel
Once you have all the ingredients, let's start cooking.

3.Installing fluentd

the recommended settings before installation, install as written on the official page.

# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

4. Installing plugins

Let's install fluent-plugin-swift and fluent-plugin-forest.

# /opt/td-agent/embedded/bin/gem install fluent-plugin-swift fluent-plugin-forest

5. Preparing the configuration file

Edit /etc/td-agent/td-agentd.conf

Defining the source (transfer source log) tag

This time, we will define /var/log/messages with the tag syslog.messages.

<source>type tail format none tag syslog.messages path /var/log/messages pos_file /var/tmp/messages.pos</source>

Match (swift object storage destination) tag definition

we will save the log in the object storage container (bucket in S3)
/hostname/YYYYMM/hostname_log file name_YYYYMMDD-HH_NN.gz

<match syslog.*>type forest subtype swift<template> auth_url https://ident-r2nd1001.app-sys.jp/v2.0/tokens auth_user "GMO App Cloud Control Panel Login ID" auth_api_key "GMO App Cloud Control Panel Login Password" auth_tenant "Tenant Name" swift_container "swift container name" swift_object_key_format %{path}${hostname}_${tag_parts[0]}.${tag_parts[1]}_%{time_slice}_%{index}.%{file_extension} path %{hostname }/%Y%m/ ssl_verify false buffer_path /var/log/fluent/swift/${tag_parts[0]}.${tag_parts[1]} time_slice_format %Y%m%d-%H time_slice_wait 10m</template></match>

It is OK to create the container name specified in swift_containter in advance, but if it does not exist, it will be created automatically.

6. Operation test

# td-agent -vv &

You can start it all at once, but first, check the status in debug mode for about an hour and check the following.

• There are no errors in the format of the configuration file.
• Logs are being transferred to object storage

*If td-agent is already running and you want to test only the settings you plan to add, prepare a configuration file with a different name (e.g. td-agent.conf.test).

# td-agent -vv -c /etc/td-agent/td-agent.conf.test &

If you do this, you can test without affecting the operation of the existing td-agent.

If the log is transferred to object storage, a log like this will appear

YYYY-mm-dd HH:MM:SS +0900 [info]: Put Log to Swift. container="Container name" object=Host name/YYYYMM/Host name_Log file name_YYYYMMDD-HH_NN.gz

7. Starting td-agent and setting automatic startup

# chkconfig td-agent on && service td-agent start

Swift-based object storage is rare these days, but the plugin settings are very similar to the S3 plugin, so I don't think it's that difficult to set up.

Have a nice fluentd life.