[fluentd] A story about how I got hooked on posting to IDCF object storage

My name is Sashihara and I am an infrastructure engineer.

The other day I had the opportunity to set up a server in IDCF's environment, and at that time I came across the idea of ​​storing logs in object storage. I was hooked, so here are my notes from that time

The configuration looks like this

API server (fluentd) ⇒ log aggregation server (fluentd) ⇒ object storage

I think this is a common configuration

Initial setup

IDCF had an official procedure, so I used that as a reference when setting it up

Since it is an S3 compatible storage, we use fluent-s3-plugin

The initial setup looks like this

<store>type s3 aws_key_id XXXXXXXXXXXXXXXXXXX aws_sec_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX s3_bucket bucket_name s3_endpoint http://ds.jp-east.idcfcloud.com check_apikey_on_start false s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension} path logs/ buffer_path /var/log/fluent/idcf/buffer/ flush_interval 60s time_slice_format %Y%m%d/%Y%m%d-%H</store>

However, for some reason it is not uploaded to object storage

Buffer file creation is no problem

I've looked into the error but I don't really understand it..

I was pretty hooked..

 

The answer was written in the fluent-plugin-s3 documentation

It was written on fluent-plugin-s3's github

signature_version

Signature version for API request. s3 means signature version 2 and v4 means signature version 4. Default is nil (Following SDK's default). It would be useful when you use S3 compatible storage that accepts only version signature 2.

The key is the signature version of the API request

• Specify s3 when using signature version 2, and v4 when using version 4
• This is useful for S3-compatible storage with Signature Version 2
• The default is based on the SDK version

It seems that you need to set the signature version of the storage you are using

Since the object storage only supports signature version 2,

signature_version s3

Once I added this it worked straight away

It seems that fluent-s3-plugin has also supported S3 signature version 4

summary

I learned a lot about the signature version, as it can be used in other S3-compatible storage environments

Not just this time, but official documentation is often in English and people tend to avoid it, but the answers are often written there

Let's improve our English skills!