Install fluent-bit using TMC Catalog and forward the logs to Azure Blob storage

Eknath - 18 March 202218 March 2022

Reading Time: 4 mins

Contents

Overview

Fluent Bit is a super fast, lightweight, and highly scalable logging and metrics processor and forwarder that allows you to collect data and logs from different sources, unify them, and send them to multiple destinations. Tanzu Kubernetes Grid includes signed binaries for Fluent Bit, that you can deploy on management clusters and on Tanzu Kubernetes clusters to provide a log-forwarding service. This post explains how to install the fluent-bit package onto a Tanzu Kubernetes (workload) cluster using TMC catalog and forward the logs to Azure blob storage.

Pre reqs

Bootstrap machine with the following installed: Tanzu CLI, kubectl installed as mentioned here
Tanzu Kubernetes Grid management cluster and workload cluster running on vSphere, Amazon EC2, or Azure, with the package repository installed. For this demo, I have deployed TKG on Azure.
Storage Account and a blob container: You can use an existing one if already exists. if not, below are the steps to create a storage account:

Navigate to Azure portal > Storage Accounts > Create

Project details:

Subscription:
- Resource group: Create new or use existing one. In this case, I have used an existing resource group (images-rg)
- Storage account name: provide a name, here I have given capvfluentd

Review + Create > Create
In Azure portal > Storage Accounts > click on newly created storage account (capvfluentd) > Create Container by clicking on + Container

Name it and Create. In this case, I named it as capv-logs

Once successfully created, you can find it under the containers list

Install the package

In TMC Console: Catalog > select the workload cluster (capv-workload in my case) from drop down and click on fluent-bit

Click on Install Package which can be found on top right side of the page

Name the Installed package as : capv-workload-fluentd and click NEXT

Package install resources: Leave to Default and click NEXT
Configure values: Provide below values

config file

namespace: "tanzu-system-logging"
fluent_bit:
  config:
    service: |
      [Service]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    outputs: |
      [OUTPUT]
        Name                  azure_blob
        Match                 *
        account_name          <redacted> # To be collected from Azure portal
        shared_key            <redacted>  # To be collected from Azure portal
        container_name       <redacted>  # To be collected from Azure portal
        path                  kubernetes
        blob_type             blockblob
        auto_create_container on
        tls                   on
    inputs: |
      [INPUT]
        Name              tail
        Tag               kube.<namespace_name>.<pod_name>.<container_name>
        Tag_Regex         (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
        Path              /var/log/containers/*.log
        Exclude_Path      /var/log/containers/*_fluentbit2S3_*.log
    filters: |
      [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Regex_Parser kubernetes-tag
        Labels Off
        Annotations Off
    parsers: |
      [PARSER]
        Name        azure_blob
        Format      json
        Time_Key    requestReceivedTimestamp
        Time_Format %Y-%m-%dT%H:%M:%S.%LZ
        Time_Keep   On
      [PARSER]
        Name        kubernetes-tag
        Format      regex
        Regex       (?<namespace_name>.+)\.(?<pod_name>.+)\.(?<container_name>.+)
  #! optional configuration for the daemonset
  daemonset:
    resources:
    #! limits:
    #!   cpu: 100m
    #!   memory: 128Mi
    #! requests:
    #!   cpu: 100m
    #!   memory: 128Mi
    podAnnotations: {}
    podLabels: {}

Install Package

Validate:

In TMC Console: Clusters > capv-workload > Add-ons > Installed > check if the package ( fluent-bit.tanzu.vmware.com ) is succeeded and healthy

# Get the admin credentials of the workload cluster into which you want to deploy Multus cni. In this case, capv-workload is workload cluster: 

$ tanzu cluster kubeconfig get capv-workload --admin

# Set the context of kubectl to the cluster

$ kubectl config use-context capv-workload-admin@capv-workload

# Verify the deployed pods

$ kubectl get pods -n tanzu-system-logging
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-cnslw   1/1     Running   0          3m55s
fluent-bit-njzqd   1/1     Running   0          3m55s

# Check the pod logs in tanzu-system-logging namespace

$ kubectl logs fluent-bit-cnslw -n tanzu-system-logging

Note: If you see below errors in fluent bit logs, then follow the below steps. If not, then Navigate to Azure portal > Storage Accounts > click on container: capv-logs > kubernetes

Errors

[2022/03/18 11:20:12] [ warn] [engine] chunk '1-1647602406.107346978.flb' cannot be retried: task_id=6, input=tail.0 > output=azure_blob.0
[2022/03/18 11:20:14] [error] [output:azure_blob:azure_blob.0] cannot create upstream connection for container check
[2022/03/18 11:20:14] [ warn] [engine] chunk '1-1647602407.824389152.flb' cannot be retried: task_id=1, input=tail.0 > output=azure_blob.0
[2022/03/18 11:20:15] [error] [output:azure_blob:azure_blob.0] cannot create upstream connection for container check
[2022/03/18 11:20:15] [ warn] [engine] chunk '1-1647602405.828550376.flb' cannot be retried: task_id=4, input=tail.0 > output=azure_blob.0
[2022/03/18 11:20:15] [error] [output:azure_blob:azure_blob.0] cannot create upstream connection for container check

Change the image in daemonset

# Edit the daemon set

kubectl edit daemonset fluent-bit -n tanzu-system-logging

Replace the image name with fluent/fluent-bit:latest

# Check the pods in namespace: tanzu-system-logging

kubectl get pods -n tanzu-system-logging
NAME               READY   STATUS        RESTARTS   AGE
fluent-bit-c4nw2   1/1     Running       0          2s
fluent-bit-njzqd   1/1     Terminating   0          19m

# Once all the pods turn healthy, Check the fluent-bit pod logs and should show no errors/warning as earlier. 

$ kubectl logs fluent-bit-c4nw2 -n tanzu-system-logging

[2022/03/18 11:35:09] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:09] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:09] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMDkuMTk1Ny5pZA== committed successfully
[2022/03/18 11:35:09] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMDkuMjAxNS5pZA== committed successfully
[2022/03/18 11:35:10] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:10] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMTAuMTgyNy5pZA== committed successfully
[2022/03/18 11:35:18] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:18] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMTguNTc0NS5pZA== committed successfully
[2022/03/18 11:35:20] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:20] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMjAuNTc0Ni5pZA== committed successfully
[2022/03/18 11:35:24] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3613516 watch_fd=2
[2022/03/18 11:35:24] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3612839 watch_fd=15
[2022/03/18 11:35:24] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:24] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMjQuNTczOS5pZA== committed successfully
[2022/03/18 11:35:25] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:25] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMjUuNTc0My5pZA== committed successfully
[2022/03/18 11:35:31] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:32] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMzEuNTc1Ni5pZA== committed successfully
[2022/03/18 11:35:36] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:36] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMzUuODc0OC5pZA== committed successfully
[2022/03/18 11:35:36] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:37] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMzYuNTc1NC5pZA== committed successfully
[2022/03/18 11:35:37] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:37] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMzYuNTc0NS5pZA== committed successfully
[2022/03/18 11:35:38] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:38] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzMzguNTgwMC5pZA== committed successfully
[2022/03/18 11:35:43] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:43] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzNDMuNTc4MS5pZA== committed successfully
[2022/03/18 11:35:49] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:49] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2022/03/18 11:35:50] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzNDkuNTc4Mi5pZA== committed successfully
[2022/03/18 11:35:50] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE2NDc2MDMzNDkuNTc3MC5pZA== committed successfully

Navigate to Azure portal > Storage Accounts > click on container: capv-logs > Kubernetes

This confirms that fluent bit is forwarding the logs to Azure blob storage successfully.