lnav – install and configure for log analysis

/ code-commands, SprUng, KEEPERS, CODE-NOW-NOW-NOW, CODING, coding cheat / 1 minute of reading
logs - common commandsInstall / Use lnavmore...

LOGS

https://lnav.org

more, grep, head, cat, multitail, and System Log Viewer

Some important commands for viewing Linux log files include cat, less, tail, head, and grep.

 

  • cat:

    Displays the contents of one or more files in the terminal

     

  • less:

    Displays the contents of a file one screen at a time, allowing the user to scroll through it

     

  • tail:

    Displays the last ten lines of a file, or continuously monitors a file for new lines

     

  • head:

    Displays the first ten lines of a file

     

  • grep:

    Searches for text patterns within files or output

     

  • lastb:

    Views failed login attempts

     

Some important Linux log files include:

  • /var/log/syslog:

    Contains messages from system services and daemons

     

  • /var/log/messages:

    Contains general system messages, including kernel messages

     

  • /var/log/auth.log:

    Contains authentication-related events, such as user logins

     

  • /var/log/dmesg:

    Contains kernel ring buffer messages, including information about hardware detection

     

  • /var/log/secure:

    Contains security-related events, including authentication attempts

     

  • /var/log/boot.log:

    Contains boot-time messages and events

     

  • /var/log/faillog:

    Records failed login attempts

     

  • /var/log/lastlog: Maintains a record of the last successful login for each user

cd /var/log.

Now issue the command ls and you will see the logs housed within this directory.

dmesg | less

tail /var/log/syslog

It will print out only the last few lines of the syslog file.

 

tail -f -n 5 /var/log/syslog

last 5 lines of log

The tail command has a very important trick up its sleeve, by way of the -f option. When you issue the command tail -f /var/log/syslog, tail will continue watching the log file and print out the next line written to the file. This means you can follow what is written to syslog, as it happens, within your terminal window (Figure 3).

Installing and Using lnav for Log Analysis

lnav (Logfile Navigator) is a powerful log file analyzer and viewer for Linux and macOS systems. Here’s how you can install and use lnav for analyzing logs for all your locally hosted and cloud-hosted sites.

Step 1: Install lnav

On Linux

  1. Using Package Manager:

    • For Debian/Ubuntu:

      sudo apt install lnav

    • For CentOS/RHEL/AlmaLinux:

      sudo yum install lnav

  2. Using Precompiled Binary: Download the latest release from the official GitHub repository:

    

wget https://github.com/tstack/lnav/releases/download/v0.X.X/lnav-0.X.X-linux.tar.gz
tar -xzf lnav-0.X.X-linux.tar.gz
sudo mv lnav /usr/local/bin/

On macOS

  1. Install via Homebrew:

    

brew install lnav

On Windows

lnav is not natively supported on Windows. Use Windows Subsystem for Linux (WSL) or a Linux virtual machine to install and run lnav.

Step 2: Centralize Your Log Files

To analyze logs from multiple sites (local and cloud):

  1. Centralize Logs:

    • Use rsync or SCP to copy logs from remote servers to a central location.

    • Example:

      

rsync -avz user@remote:/var/log/nginx/*.log /central/logs/nginx/

  2. Mount Logs from Cloud Locations:

    • Use tools like sshfs to mount remote directories locally.

    • Example:

      

sshfs user@remote:/var/log/nginx/ /mnt/logs

Step 3: Use lnav for Analysis

Basic Usage

  1. Open lnav in a directory containing log files:

    

lnav /path/to/logs/

  2. For specific files:

    

lnav /path/to/logs/access.log /path/to/logs/error.log

Common Commands

  • Press ? within lnav to see a help menu.

  • Navigate logs:

    • Arrow keys or j/k to scroll up/down.

  • Search for keywords:

    • Press / and type a keyword (e.g., error).

  • Filter logs:

    • :filter-in error
    • :filter-out debug

Advanced Features

  • Combine Logs:

    • Use lnav to merge and analyze logs from multiple sites.

    • Example:

      

lnav /central/logs/nginx/*.log

  • SQL Queries:

    • lnav supports SQL-like queries to analyze log data.

    • Example: Find the number of 404 errors:

      

SELECT COUNT(*) FROM access_log WHERE status = 404;

  • Auto-detect Formats:

    • lnav automatically detects log formats, such as NGINX, Apache, and syslog.

Step 4: Automate Analysis for Multiple Sites

  1. Create a Script to Centralize Logs: Example:

    

#!/bin/bash
mkdir -p /central/logs/nginx/
rsync -avz user@remote:/var/log/nginx/*.log /central/logs/nginx/
lnav /central/logs/nginx/

  2. Schedule with Cron: Automate the script to fetch and analyze logs regularly:

    

crontab -e

    Add a cron job:

    

0 * * * * /path/to/script.sh

Step 5: Integrate lnav with Your Workflow

  1. Remote Access: Use ssh to analyze logs directly on remote servers:

    

ssh user@remote "lnav /var/log/nginx/*.log"

  2. Include in CI/CD Pipeline: Add lnav to your CI/CD process for quick log review during deployments.

  3. Monitor Real-time Logs: Use tail with lnav:

    

tail -f /var/log/nginx/access.log | lnav

  4. Analyze Specific Logs Dynamically: Combine filters and queries for insights:

    

lnav /var/log/nginx/*.log

This setup ensures lnav is versatile for managing logs from all your sites, whether local or cloud-based.

 

 

 

You said:

so bascilly you ahve to somehow “pull” logs to you locally – then run lnav on them from there?

 

ChatGPT said:

ChatGPT

Yes, lnav operates on local log files, so you need to either centralize or “pull” logs from remote servers to analyze them locally. Here’s a detailed look at your options for working with lnav:

1. Pull Logs Locally

If you want to analyze logs across multiple servers or cloud environments, you must consolidate them locally.

Methods to Pull Logs

  • rsync:

    Efficiently copy logs from remote servers to a central location.

    

rsync -avz user@remote:/var/log/nginx/*.log /central/logs/nginx/

  • SCP (Secure Copy):

    

scp user@remote:/var/log/nginx/*.log /central/logs/nginx/

  • SFTP or FTP Tools: Use GUI tools like FileZilla or WinSCP to fetch logs.

  • Log Aggregation Tools: Use tools like Fluentd or Logstash to centralize logs, then use lnav on the aggregated data.

2. Real-Time Remote Analysis

If you prefer not to pull logs locally, you can use SSH to work with logs directly on the server:

  • Run

    remotely:

    

ssh user@remote "lnav /var/log/nginx/*.log"

  • Tail logs with

    :

    

ssh user@remote "tail -f /var/log/nginx/access.log" | lnav

3. Mount Remote Logs Locally

If you want seamless access to remote logs:

  • Use

    sshfs

    to mount remote directories locally:

    

sshfs user@remote:/var/log/nginx/ /mnt/logs
lnav /mnt/logs/*.log

4. Centralized Logging System

For multiple sites, consider setting up a centralized logging system:

  1. Log Forwarding with Syslog or Rsyslog: Configure servers to forward logs to a central logging server.

    • On the remote server, configure

      to send logs:

      

*.* @central-log-server:514

    • On the central server, set up rsyslog to receive logs.

  2. Use lnav on the central logging server:

    

lnav /central/logs/nginx/*.log

5. Automate Log Consolidation

Write a script to automate the log-pulling process:



#!/bin/bash
# Pull logs from multiple servers
servers=("192.168.1.50" "192.168.1.60" "cloud.example.com")
log_dir="/central/logs"

mkdir -p $log_dir

for server in "${servers[@]}"; do
rsync -avz user@$server:/var/log/nginx/*.log $log_dir/$server/
done

# Run lnav on the consolidated logs
lnav $log_dir/**/*.log

6. Alternatives for Cloud Sites

If your sites are hosted on cloud platforms:

  • AWS CloudWatch Logs: Export logs to local storage and analyze with lnav.

  • Google Cloud Logging:

    Use the

    CLI to download logs.

    

gcloud logging read "resource.type=gae_app" > logs.txt
lnav logs.txt

  • Azure Log Analytics: Export logs to files and analyze locally.

Summary

  • Yes, logs must be accessible locally for lnav to work.
  • You can either pull logs, mount them remotely, or use SSH for real-time access.
  • For multiple sites, centralized logging simplifies log management.
more …
Scroll to Top