.\" Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
.\" $Id: logfmt.1,v 1.12 2007-06-19 09:37:25 daljord Exp $
.\"
.Dd Jun 7, 2017
.Dt vespa-logfmt \&1 "Vespa documentation"
.Os "Yahoo! Vespa" "6.0"
.Os
.Sh NAME
.Nm vespa-logfmt
.Nd select and format messages from Vespa log files
.Sh SYNOPSIS
.Nm
.Op Fl l Ar levellist
.Op Fl s Ar fieldlist
.Op Fl p Ar pid
.Op Fl S Ar service
.Op Fl H Ar host
.Op Fl c Ar regex
.Op Fl m Ar regex
.Op Fl t
.Op Fl f
.Op Fl L
.Op Fl N
.Op Fl ts
.Op Ar
.Sh DESCRIPTION
The
.Nm
utility reads Vespa log files, select messages and writes a formatted
version of selected messages to the standard output.
.Pp
The options are as follows:
.Bl -tag -width ".It Fl l Ar levellist"
.It Fl l Ar levellist
Select which log levels to select.
The default is to select "fatal", "error", "warning" and "info"
levels, and suppress "config", "event", "debug" and "spam" levels;
but when using this option, only the named levels will be selected.
The
.Ar levellist
is a comma separated list of level names.
The name
.Em all
may be used to add all known levels.
Prepending a minus sign will deselect the level named.
Starting the list with a plus sign will add and remove levels
from the current (or default) list of levels instead
of replacing it.
.It Fl s Ar fieldlist
Select which fields of log messages to show.
The order of the actual output fields is fixed.
When using this option, only the named fields will be shown.  The
fieldlist is a comma separated list of field names.  The name
.Em all
may be used to add all possible fields.
Prepending a minus sign will turn off display of the named field.
Starting the list with a plus sign will add and remove fields
from the current (or default) list of fields instead
of replacing it.
.Pp
The fields which may be named are:
.Bl -tag -width component
.It time
Print the time in seconds since the epoch.
Ignored if
.Em fmttime
is shown.
.It fmttime
Print the time in human-readable [YYYY-MM-DD HH:mm:ss] format.
Note that the time is printed in the local timezone; to get GMT
output use
.Nm "\*[q]env TZ=GMT vespa-logfmt\*[q]"
as your command.
.It msecs
Add milliseconds after the seconds in
.Em time
and
.Em fmttime
output.  Ignored if
.Em usecs
is in effect.
.It usecs
Add microseconds after the seconds in
.Em time
and
.Em fmttime
output.
.It host
Print the hostname field.
.It level
Print the level field (uppercased).
.It pid
Print the pid field.
.It service
Print the service field.
.It component
Print the component field.
.It message
Print the message text field.
You probably always want to add this.
.El
.Pp
Using this option several times works as if the given
.Ar fieldlist
arguments had been concatenated into one comma-separated list.
The default fields to show are as if
.Bk
.Op Fl s Ar fmttime,msecs,level,service,component,message
.Ek
had been given.
.It Fl p Ar pid
Select only messages where the pid field matches the
.Ar pid
string exactly.
.It Fl S Ar service
Select only messages where the service field matches the
.Ar service
string exactly.
.It Fl H Ar host
Select only messages where the hostname field matches the
.Ar host
string exactly.
.It Fl c Ar regex
Select only messages where the component field matches the
.Ar regex
given, using
.Xr perlre
regular expression matching.
.It Fl m Ar regex
Select only messages where the message text field matches the
.Ar regex
given, using
.Xr perlre
regular expression matching.
.It Fl f
Invoke tail -F to follow input file
.Bk
.Fl "s +host"
.Ek
as default.
.It Fl N
Dequote quoted newlines in the message text field to an actual newline plus tab.
.It Fl t
Format the component field (if shown) as a fixed-with string,
truncating if necessary.
.It Fl ts
Format the service field (if shown) as a fixed-with string,
truncating if necessary.
.El
.Sh EXAMPLES
The command:
.Pp
.Bd -literal -offset indent
vespa-logfmt -l event -s service,message,fmttime,message
.Ed
.Pp
will display only messages with log level "event",
printing a human-readable time (without any fractional seconds),
the service generating the event and the event message, like this:
.Bd -literal -offset indent
[2004-12-07 18:43:01] config-sentinel  starting/1 name="logd"
[2004-12-07 18:43:01] logd             started/1 name="logdemon"
[2004-12-07 18:45:51] rtc              starting/1 name="rtc.index0"
[2004-12-07 18:45:51] rtc.index0       started/1 name="flexindexer.index"
[2004-12-07 18:45:51] rtc.index0       stopping/1 name="flexindexer.index" why="done"
[2004-12-07 18:45:53] rtc              stopped/1 name="rtc.index0" pid=50600 exitcode=0
[2004-12-07 18:46:13] logd             stopping/1 name="logdemon" why="done ok."
[2004-12-07 18:46:13] config-sentinel  stopped/1 name="logd" pid=49633 exitcode=0
.Ed
.Pp
Note that the second "message" item in the fieldlist is redundant,
and that order of printed field is fixed no matter what the fieldlist
order is.
.Pp
The command:
.Pp
.Bd -literal -offset indent
vespa-logfmt -l all-info,-debug -s level \e
       -s time,usecs,component,message -t -l -event
.Ed
.Pp
will display messages with log levels that are
.Em not
any of
.Em info, debug,
or
.Em event,
printing the time in seconds and microseconds, the log level, the
component name, and the message text, possibly somewhat like this:
.Bd -literal -offset indent
1102441382.530423 CONFIG  : nc              Config handle: 'pandora.0-rtx'
1102441551.471568 CONFIG  : flexindexer.doc Adding document type typetest-0
1102441573.148211 WARNING : logdemon        stopping on signal 15
1102441887.158000 WARNING : com.yahoo.fs4.m read exception
1102441935.569567 WARNING : rtc             Dispatch inherited job failed for dir dispatch0
1102442474.205920 WARNING : rtx             RTC (tcp/172.24.94.75:10161) : DOWN
1102442983.075669 ERROR   : flexindexer.std Unable to find cluster map defaultcluster
.Ed
.Sh FILES
If no file argument is given,
.Nm
will read the last vespa log file $VESPA_HOME/logs/vespa/vespa.log (this also works with the
.Fl f
option).
Otherwise, reads only the files given as arguments.
To read standard input, supply a single dash '-' as a file argument.
.Sh SEE ALSO
Documentation in the "log" module for input file format.
.Sh HISTORY
Developed as part of Vespa 1.1.  The default output format
reflects the old "fastlib" log formatting, with minor differences
and is intended to be human-readable, not parsed.