dtail

DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once.

View the Project on GitHub

Examples

This page demonstrates the primary usage of DTail. Please also see dtail --help for more available options.

How to use dtail

Tailing logs

The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all records containing the string STAT. Any other Go compatible regular expression can be used instead of STAT.

% dtail --servers serverlist.txt --files "/var/log/service/*.log" --regex STAT

dtail

Aggregating logs

To run ad-hoc MapReduce aggregations on newly written log lines, you also must add a query. The following example follows all remote log lines and prints out every 5 seconds the top 10 servers with the most average free memory. To run a MapReduce query across log lines written in the past, please use the dmap command instead.

% dtail --servers serverlist.txt  \
    --query 'select avg(memfree), $hostname from MCVMSTATS group by $hostname order by avg(memfree) limit 10 interval 5' \
    --files '/var/log/service/*.log'

For MapReduce queries to work, you have to ensure that DTail supports your log format. You can either use the ones already defined in internal/mapr/log format or add an extension to support a custom log format.

dtail-map

How to use dcat

The following example demonstrates how to cat files (display the full content of the files) of multiple servers at once. The servers are provided as a comma-separated list this time.

% dcat --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org \
    --files /etc/hostname

dcat

How to use dgrep

The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after the swap partition in /etc/fstab. We do that only on the first 20 servers from serverlist.txt. dgrep is also very useful for searching log files of the past.

% dgrep --servers <(head -n 20 serverlist.txt) \
    --files /etc/fstab \
    --regex swap

dgrep

How to use dmap

To run a MapReduce aggregation over logs written in the past, the dmap command can be used. For example, the following command aggregates all MapReduce fields of all the records and calculates the average memory free grouped by day of the month, hour, minute and the server hostname. dmap will print interim results every few seconds. The final product, however, will be written to file mapreduce.csv.

% dmap --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org,serv-021.lan.example.org,serv-022.lan.example.org,serv-023.lan.example.org \
    --query 'select avg(memfree), $day, $hour, $minute, $hostname from MCVMSTATS group by $day, $hour, $minute, $hostname order by avg(memfree) limit 10 outfile mapreduce.csv' \
    --files "/var/log/service/*.log"

Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in internal/mapr/log format or add an extension to support a custom log format.

dmap