[fluentd] A story about how I got hooked on posting to IDCF object storage

My name is Sashihara and I am an infrastructure engineer.

The other day, I had a chance to build a server in IDCF's environment, and at that time there was a mention of storing logs in object storage, but I was hooked on it, so here are my notes from that time.

The configuration looks like this

API server (fluentd) ⇒ log aggregation server (fluentd) ⇒ object storage

I think this is a common configuration.

initial configuration

There was an official procedure from IDCF, so I proceeded with the settings using that as a reference.

I am using fluent-s3-plugin because it is S3 compatible storage.

The initial settings look like this

<store>type s3 aws_key_id XXXXXXXXXXXXXXXXXX aws_sec_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX s3_bucket bucket_name s3_endpoint http://ds.jp-east.idcfcloud.com check_apikey_on_start false s3_object_key_format %{path}%{time_slice}_%{index}.%{file_ext extension} path logs/ buffer_path /var/ log/fluent/idcf/buffer/ flush_interval 60s time_slice_format %Y%m%d/%Y%m%d-%H</store>

However, for some reason it is not uploaded to object storage. .

There is no problem in creating the buffer file.

I've looked into the error but can't figure it out...

I was pretty hooked...

 

The answer was written in the fluent-plugin-s3 documentation.

It was written on fluent-plugin-s3's github

signature_version

Signature version for API request. s3 means signature version 2 and v4 means signature version 4. Default is nil (Following SDK's default). It would be useful when you use S3 compatible storage that accepts only version signature 2.

The point is about the signature version of the API request.

• Specify s3 when using signature version 2, and v4 when using signature version 4.
• It is convenient to use it for S3 compatible storage with signature version 2.
• The default is based on the SDK version.

Is that what it feels like? It seems that you need to enter settings according to the signature version of the storage you are using.

Since object storage seemed to only support signature version 2

signature_version s3

I added this and it worked fine.

It seems that the fluent-s3-plugin side has also supported S3 signature version 4.

summary

I learned a lot about the signature version because it can be used in S3 compatible storage in other environments.

I tend to avoid official documents, including this one because they are often in English, but they often have answers written in them.

Let's improve our English skills!