Skip to content

Latest commit

 

History

History
58 lines (45 loc) · 1.69 KB

Datastructure.md

File metadata and controls

58 lines (45 loc) · 1.69 KB
Raw

Data structure

This article will introduce the internal data structure and data flow of iLogtail.

Overview of data types

This section will introduce some data types related to the plug-in interface. Currently, the data types between iLogtail and the service backend are described by protobuf .

Content & LogTag

Transfer data fields and labels, simple key/value pairs.

message Content
{
required string Key = 1;
required string Value = 2;
}

message LogTag
{
required string Key = 1;
required string Value = 2;
}

Log

Log is the data type that represents a single log. The Time field is the log time. The Contents field maintains the content of this log and consists of a key/value list.

message Log
{
  required uint32 Time = 1;// UNIX Time Format
  repeated Content Contents = 2;

}

LogGroup

LogGroup (log group) is a package of multiple logs:

  • Logs: Contains all logs.
  • Category: The Logstore of SLS, which is the independent unit of data storage and can be compared to Kafka independent cluster.
  • Topic: Topic of the log service. A Category (Logstore) can be divided into multiple topics. If it is not filled in, Topic is an empty string, which can be compared to the concept of Topic of Kafka.
  • Source/MachineUUID: The information of the node where iLogtail is located. The former is the IP and the latter is the UUID.
  • LogTags: The tags to all logs, which also consists of a key/value list.
message LogGroup
{
  repeated Log Logs = 1;
  optional string Category = 2;
  optional string Topic = 3;
  optional string Source = 4;
  optional string MachineUUID = 5;
  repeated LogTag LogTags = 6;
}