dtail

DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once.

View the Project on GitHub

Log Formats

You may have looked at the DTail Query Language and wondered how to make DTail understand your own log format(s). If DTail doesn’t know your log format, it won’t be able to extract much useful information from your logs. This information then can be used as fields (e.g. variables) by the Query Language.

You could either make your application follow the DTail default log format, or you would need to implement a custom log format. Have a look at ./integrationtests/mapr_testdata.log for an example a log file in the DTail default format.

Available log formats

The following log formats are currently available out of the box:

Selecting a log format

By default, DTail will use the default log format. You can override the log format with the logformat keyword:

% dmap --files /var/log/example.log --query 'from EXAMPLE select ....queryhere.... logformat generickv'

You can override the default log format with MapreduceLogFormat in the Server section of dtail.json.

Under the hood: generickv

As an example, let’s have a look at the generickv log format’s implementation. It’s located at internal/mapr/logformat/generickv.go:

type genericKVParser struct {
	defaultParser
}

func newGenericKVParser(hostname, timeZoneName string, timeZoneOffset int) (*genericKVParser, error) {
	defaultParser, err := newDefaultParser(hostname, timeZoneName, timeZoneOffset)
	if err != nil {
		return &genericKVParser{}, err
	}
	return &genericKVParser{defaultParser: *defaultParser}, nil
}

func (p *genericKVParser) MakeFields(maprLine string) (map[string]string, error) {
	splitted := strings.Split(maprLine, protocol.FieldDelimiter)
	fields := make(map[string]string, len(splitted))

	fields["*"] = "*"
	fields["$line"] = maprLine
	fields["$empty"] = ""
	fields["$hostname"] = p.hostname
	fields["$server"] = p.hostname
	fields["$timezone"] = p.timeZoneName
	fields["$timeoffset"] = p.timeZoneOffset

	for _, kv := range splitted[0:] {
		keyAndValue := strings.SplitN(kv, "=", 2)
		if len(keyAndValue) != 2 {
			//dlog.Common.Debug("Unable to parse key-value token, ignoring it", kv)
			continue
		}
		fields[keyAndValue[0]] = keyAndValue[1]
	}

	return fields, nil
}

… whereas:

Log format variables

Common variables:

The common variables may exist in all log formats:

Default log format variables:

These variables may only exist in the DTail default log format (see internal/mapr/logformat/default.go more details):

Date and time:

Log level/severity:

System and Go runtime:

Implementing your own log format Foo

What needs to be done is to place your own implementation into the logformat source directory. As a template, you can copy an existing format …

% cp internal/mapr/logformat/generic.go internal/mapr/logformat/foo.go

… and replace generic ` with your format’s name foo:

package logformat

type fooParser struct {
	defaultParser
}

func newFooParser(hostname, timeZoneName string, timeZoneOffset int) (*fooParser, error) {
	defaultParser, err := newDefaultParser(hostname, timeZoneName, timeZoneOffset)
	if err != nil {
		return &fooParser{}, err
	}
	return &fooParser{defaultParser: *defaultParser}, nil
}

func (p *fooParser) MakeFields(maprLine string) (map[string]string, error) {
	fields := make(map[string]string, 3)

	..
	<YOUR CUSTOM CODE HERE>
	..

	return fields, nil
}

Next, NewParser(...) in internal/mapr/logformat/parser.go needs to be extended, so that the new log format is part of the switch statement. If you don’t want to edit parser.go then you could instead use custom1 or custom2 log formats, there are ready templates available in the logformat package.

Once done, recompile DTail. DTail now understands ... logformat foo (see “Seleting a log format” above).