Share This:

Logs are crucial to help you understand what is happening inside your Kubernetes cluster.

 

Even though most applications have native logging mechanism out of the box, in the distributed and containerized environment (like Kubernetes), users will be better off with the centralized logging solution. That’s because they need to collect logs from multiple applications with different log formats and send them to some logging backend for subsequent storage, processing, and analysis. Kubernetes provides all the basic resources needed to implement such functionality.

 

In this tutorial, we explore Kubernetes logging architecture and demonstrate how to collect application and system logs using Truesight ITDA, which using Elasticsearch backend offers great full-text search, log aggregation, analysis, and visualization functionality. Let’s get started!

 

Overview of Kubernetes Logging Architecture and Logging Options

Docker containers in Kubernetes write logs to standard output (stdout) and standard (stderr) error streams. Docker redirects these streams to a logging driver configured in Kubernetes to write to a file in JSON format. Kubernetes then exposes log files to users via kubectl logs command. Users can also get logs from a previous instantiation of a container setting the --previous flag of this command to true. That way they can get container logs if the container crashed and was restarted.

 

However, if a pod is deleted from the node forever, all corresponding containers and their logs are also deleted. The same happens when the node dies. In this case, users are no longer able to access application logs. To avoid this situation, container logs should have a separate shipper, storage, and lifecycle that are independent of pods and nodes. Kubernetes does not provide a native storage solution for log data, but you can easily integrate your preferred logging shipper into the Kubernetes cluster using Kubernetes API and controllers.

 

Kubernetes architecture facilitates a number of ways to manage application logs. Common approaches to consider are:

  • using a logging sidecar container running inside an app’s pod.
  • using a node-level logging agent that runs on every node.
  • push logs directly from within an application to some backend.

 

Let’s briefly discuss the details of the first and the second approach.

To complete examples used below, you’ll need the following:

  • A running Kubernetes cluster.
  • A kubectl command line tool installed and configured to communicate with the cluster.

 

Pre-requisite: Creating ITDA collector Docker image

Let’s create ITDA collector Docker image which can be used to collect logs using both the approach.

Use following Dockerfile to create Docker image.

 

FROM centos

MAINTAINER schopda@bmc.com

ENV TARGET Linux-2-6-x86-64-nptl

ENV BASEDIR /opt/

ENV INSTALLDIR /opt/bmc

ENV PATUSER patrol

ADD truesight_itda.tar $BASEDIR

 

RUN useradd -p 8Iq9Omhord6cQ $PATUSER

RUN mkdir $INSTALLDIR

RUN chmod -R 777 $INSTALLDIR

 

RUN echo "cat /etc/hostname" > /usr/bin/hostname

RUN chmod +x /usr/bin/hostname

RUN echo patAdm1n | passwd --stdin root

 

WORKDIR /opt/bmc_products

RUN sh RunSilentInstall.sh

 

WORKDIR $INSTALLDIR/Patrol3

RUN mkdir $INSTALLDIR/Patrol3/admin

RUN mkdir $INSTALLDIR/Patrol3/images

RUN chown -R patrol:patrol $INSTALLDIR/Patrol3

ENV PATH $PATH:$INSTALLDIR/Patrol3/$TARGET/bin

ENV PATROL_HOME=$INSTALLDIR/Patrol3/$TARGET

ENV PATROL_ADMIN=$PATUSER

 

RUN rm -rf /opt/bmc_products

CMD ["PatrolAgent"]

 

There are several things to pay attention to:

· It uses Truesight CMA repository package as installable. Package should have Patrol Agent, IT Data Analytics KM and JRE. Also make sure you apply correct integrationServices variable during package creation. Package should be saved as truesight_itda.tar.

· The collector needs root permission to read logs in /var/log. To avoid permission error, we will run collector with root user inside container. Create CMA repository package with PATROL default account as root. For this exercise root password is set to ‘patAdm1n’. If you change this then make sure to change it in Dockerfile as well.

In this blog, we will refer this image as “vl-pun-lnx-01.bmc.com:80/ITDA/truesight-itda-col”

To know more details on creating Docker image, please check out my blog post. Once the Docker image is created, push it to Docker repository.

 

Approach I: Using Sidecar Containers

 

 

Let’s assume you have an application container producing some logs and outputting them to stdout, stderr , and/or a log file. In this case, you can create one or more sidecar containers inside the application pod. The sidecars will be watching for the log files and an app’s container stdout/stderr  and will stream log data to their own stdout  and stderr  streams. Optionally, a sidecar container can also pass the retrieved logs to a node-level logging agent for subsequent processing and storage. This approach has many benefits described in this great article from the official documentation. Let’s summarize them:

  • With sidecar containers, you can separate several log streams from your app container. This is handy when your app container produces logs with different log formats. Mixing different log formats would deteriorate manageability of your logging pipeline.
  • Sidecar containers can read logs from those parts of your application that lack support for writing to stdout or stderr.
  • Because sidecar containers use stdout and stderr , you can use built-in logging tools like kubectl logs .
  • Sidecar containers can be used to rotate log files which cannot be rotated by the application itself.

At the same time, however, sidecar containers for logging have certain limitations:

  • Writing logs to a file and then streaming them to stdout can significantly increase disk usage. If your application writes to a single file, it’s better to set /dev/stdout as the destination instead of implementing the streaming sidecar container approach.
  • If you want to ship logs from multiple applications, you must design a sidecar(s) for each of them.

 

Sidecar container with logging agent:

 

If the node-level logging agent is not flexible enough for your situation, you can create a sidecar container with a separate logging agent that you have configured specifically to run with your application.

 

Here is the pod configuration manifest that you can use to implement this approach. The pod mounts a volume where ITDA can pick the data.

 

     1 apiVersion: v1

     2 kind: Pod

     3 metadata:

     4    name: counter

     5 spec:

     6    containers:

     7    - name: count

     8      image: busybox

     9      args:

    10      - /bin/sh

    11      - -c

    12      - >

    13        i=0;

    14        while true;

    15        do

    16          echo "$i: This is BMC sidecar container first log file test message $(date)" >> /var/log/1.log;

    17          echo "$(date) INFO:This is BMC sidecar container second log file test message $i" >> /var/log/2.log;

    18          i=$((i+1));

    19          sleep 1;

    20        done

    21      volumeMounts:

    22      - name: varlog

    23        mountPath: /var/log

    24    - name: count-agent

    25      image: vl-pun-lnx-01.bmc.com:80/ITDA/truesight-itda-col

    26      ports:

    27      - containerPort: 3181

    28        hostPort: 5010

    29      volumeMounts:

    30      - name: varlog

    31        mountPath: /var/log

    32    volumes:

    33    - name: varlog

    34 emptyDir: {}

There are some additional steps required to get data into ITDA server because of unknown log data format.

1. Once the pod is running, it should appear in the Managed Devices in Truesight presentation server.

2. Create Monitoring policy for IT Data Analytics solution to connect ITDA agent to ITDA collection station.

3. Once the policy is applied to pod agent. Next step is to create ITDA data collector, with file details and appropriate Data pattern format to index data correctly.

After few mins log data should appear in Truesight ITDA server.

 

Approach II: Using a Node-Level Logging Agent

 

In this approach, you deploy a node-level logging agent on each node of your cluster. This agent is usually a container with access to log files of all application containers running on that node. Production clusters normally have more than one nodes spun up. If this is your case, you’ll need to deploy a logging agent on each node.

 

The easiest way to do this in Kubernetes is to create a special type of deployment called DaemonSet. The DaemonSet controller will ensure that for every node running in your cluster you have a copy of the logging agent pod. The DaemonSet controller will also periodically check the count of nodes in the cluster and spin up/down a logging agent when the node count changes. DaemonSet structure is particularly suitable for logging solutions because you create only one logging agent per node and do not need to change the applications running on the node. The limitation of this approach, however, is that node-level logging only works for applications’ standard output and standard error streams.

 

 

Deploying ITDA Agent as Daemonset to collect system and application Logs

 

Using node-level logging agents is the encouraged approach in Kubernetes because it allows centralizing logs from multiple applications via installation of a single logging agent per each node. We now discuss how to implement this approach using ITDA agent deployed as a DaemonSet in your Kubernetes cluster.

 

We used same Docker image as that of sidecar approach to create Daemonset.

 

Step 1: Deploy a DaemonSet

 

Here is the Daemonset configuration manifest that you can use. Each Daemonset pod mounts a two volumes to host paths /var/log and /var/log/containers.

 

     1  apiVersion: apps/v1

     2  kind: DaemonSet

     3  metadata:

     4    name: truesight-itda

     5    namespace: default

     6    labels:

     7      k8s-app: truesight-itda

     8  spec:

     9    selector:

    10      matchLabels:

    11        name: truesight-itda

    12    template:

    13      metadata:

    14        labels:

    15          name: truesight-itda

    16      spec:

    17        tolerations:

    18        - key: node-role.kubernetes.io/master

    19          effect: NoSchedule

    20        containers:

    21        - name: truesight-itda

    22          image: vl-pun-lnx-01.bmc.com:80/ITDA/truesight-itda-col

    23          ports:

    24          - containerPort: 3181

    25            hostPort: 5003

    26          volumeMounts:

    27          - name: varlog

    28            mountPath: /var/log

    29          - name: varlibdockercontainers

    30            mountPath: /var/lib/docker/containers

    31            readOnly: true

    32        terminationGracePeriodSeconds: 30

    33        volumes:

    34        - name: varlog

    35          hostPath:

    36            path: /var/log

    37        - name: varlibdockercontainers

    38          hostPath:

    39            path: /var/lib/docker/containers

 

Let’s save the manifest in the ts-itda-ds.yaml  and create the DaemonSet:

1

2

kubectl create -f ts-itda-ds.yaml

 

Step 2: Configure log data collection

 

Once Deamonset is created successfully, it needs to be configured to collect data and index correctly to make log search and analysis easy and meaningful. In large cluster this could be cumbersome if one has to configure each node, that’s where Truesight policy configuration and Kubernetes ITDA content pack could make things easier and faster to setup.

 

To import “Kubernetes” content pack, login to ITDA server and use menu Administration -> Content Packs -> Import menu.

 

Once imported successfully, it adds Collection Profile template which includes two data collector configurations.

  1. varlogcontainer – To collect pod logs from each node residing in /var/log/containers
  2. varlogmessages – To collect system logs (kubelet and Docker log) from /var/log/messages file

 

Next step is to create Truesight CMA policy to configure Daemonset agents to connect to ITDA collection station and server with Elasticsearch backend and to use data collector templates created by content pack. The policy configuration remains same as we did for sidecar container with following two additions.

  1. Make sure Agent selection criteria is set as “Agent Host Name” starts with Daemonset name i.e. truesigh-itda. This could be useful in auto scaled cluster where nodes gets added/removed frequently.

   2. In IT Data Analytics monitor configuration, add Collection profile “kubernetes” to use data collector templates to automatically create data collectors in ITDA server.

 

 

With in few minutes ITDA agents will connect to server and automatically create required data collectors. For each node 2 data collectors will be created. For three node cluster it created 6 data collectors.

Data collectors: 

To see logs collected, click on search icon against respective data collector.

 

 

 

 

Conclusion

All things considered, Kubernetes platform facilitates implementation of full logging pipelines by providing such useful abstraction as DaemonSets. We saw how to easily implement cluster-level logging using node agents deployed as DaemonSets .

In this tutorial, we demonstrated how Truesight ITDA can easily centralize logs from multiple applications. Unlike sidecar containers that should be created for each application running in your cluster, node-level logging requires only one logging agent per node.