dtail

DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once.

View the Project on GitHub

Log Formats

You may have looked at the DTail Query Language and wondered how to make DTail understand your own log format(s). If DTail doesn’t know your log format, it won’t be able to extract much useful information from your logs. This information then can be used as fields (e.g. variables) by the Query Language.

You could either make your application follow the DTail default log format, or you would need to implement a custom log format. Have a look at ./integrationtests/mapr_testdata.log for an example a log file in the DTail default format.

Available log formats

The following log formats are currently available out of the box:

Selecting a log format

By default, DTail will use the default log format. You can override the log format with the logformat keyword:

% dmap --files /var/log/example.log --query 'from EXAMPLE select ....queryhere.... logformat generickv'

Alternatively, you can override the default log format with MapreduceLogFormat in the Server section of dtail.json.

Under the hood: generickv

As an example, let’s have a look at the generickv log format’s implementation. It’s located at internal/mapr/logformat/generickv.go:

// MakeFieldsGENERIGKV is the generic key-value logfile parser.
func (p *Parser) MakeFieldsGENERIGKV(maprLine string) (map[string]string, error) {
	splitted := strings.Split(maprLine, protocol.FieldDelimiter)
	fields := make(map[string]string, len(splitted))

	fields["*"] = "*"
	fields["$line"] = maprLine
	fields["$empty"] = ""
	fields["$hostname"] = p.hostname
	fields["$server"] = p.hostname
	fields["$timezone"] = p.timeZoneName
	fields["$timeoffset"] = p.timeZoneOffset

	for _, kv := range splitted[0:] {
		keyAndValue := strings.SplitN(kv, "=", 2)
		if len(keyAndValue) != 2 {
			// dlog.Common.Debug("Unable to parse key-value token, ignoring it", kv)
			continue
		}
		fields[keyAndValue[0]] = keyAndValue[1]
	}

	return fields, nil
}

… whereas:

Log format variables

Common variables:

The common variables may exist in all log formats:

Default log format variables:

These variables may only exist in the DTail default log format (see internal/mapr/logformat/default.go more details):

Date and time:

Log level/severity:

System and Go runtime:

Implementing your own log format

All what needs to be done is to place your own implementation into the logformat source directory. As a template, you can copy an existing format …

% cp internal/mapr/logformat/generic.go internal/mapr/logformat/yourcustomformat.go

… and replace GENERIGKV with your format’s name in capital letters (the method name string is used by DTail to reflect the log format parser method, so it is important to name it correctly):

// MakeFieldsCUSTOMLOGFORMAT is your own custom log format.
func (p *Parser) MakeFieldsCUSTOMLOGFORMAT(maprLine string) (map[string]string, error) {
	// .. Your own format implementation goes here
	// .. you can parse maprLine and store values into the fields map. 
..
.
.
	return fields, nil
}

Once done, recompile DTail. DTail now understands ... logformat customlogformat (see “Seleting a log format” above).