Linux Mac

How To Make Better Sense Of Log Files On Linux and Mac

Matthew Hughes 24-06-2016

When things go wrong with Linux, it can be a nightmare to troubleshoot. The inherent challenges associated with this are doubled when the problem you’re facing is intermittent, and you don’t know what’s causing it.


I suppose you could spend hour after hour perusing Stack Overflow, and asking Reddit for help The Awesome Guide to Reddit Wonder how your friends always find cool stuff on the Internet before you? They're probably using Reddit, the self-proclaimed "front page of the Internet". Read More . Or you could take things into your own hands, and dive into your system’s log files, with the aim of finding out what the problem is.

What Are Log Files?

Many programs — be they for Windows, Mac, or Linux — generate log files as they go. Even Android generates them How to Get a Logcat for Reporting Bugs on Android If you've ever been asked for a logcat, this is how to do it! Read More . These are plain-text files that contain information about how a program is running. Each event will be on its own line, time-stamped to the second.

Although this isn’t universally true for all applications, log files typically tend to be found in the /var/log directory.


Overwhelmingly, the data in these files will be mundane. It won’t necessarily be indicative of a problem. It’ll just be updates on what the program was doing at a given time.


But when there is a problem, you can guarantee that information about it will be contained in the log files. This information can be used to remedy it, or to ask a descriptive question of someone who might know.

So, when dealing with log files, how do you isolate the information you care about from the stuff you don’t?

Using Standard Linux Utilities

Like we mentioned before, log files aren’t exclusive to any one platform. Despite that, the focus of this article is going to be Linux and OS X, because these two operating systems ship with the essential UNIX command line tools A Quick Guide To Get Started With The Linux Command Line You can do lots of amazing stuff with commands in Linux and it's really not difficult to learn. Read More required to parse through them.

Since log files are plain-text files, you can use any tools that you’d use to view such files. Of these, grep is probably the most difficult to learn, but also the most useful. It allows you to search for specific phrases and terms within a particular file. The syntax for this is grep [term] [filename].



At their most advanced, you can use regular expressions (RegEx) to search for terms and items with a laser-focus. Although RegEx often looks like wizardry, it’s actually pretty simple to get the hang of.

Then there are the ‘head’ and ‘tail’ commands. No points for guessing what these do. They show you the top and bottom ten lines of a file, respectively. So, if you wanted to see the latest items on a logfile, you’d run “tail filename”.



You can change the number of lines displayed by using the ‘-n’ trigger. So, if you wanted to see the first 20 lines of a file, you’d run

head -n 20 [filename]


If you want to look at the entire contents of a file, you can use the ‘cat’ utility. This can be a bit unwieldy though, as log files can often measure in the hundreds of thousands of lines. A better idea would be to pipe it to the less utility, which will let you view it one page at a time. To do that, run


cat [filename] | less


Alternatively, you could use sed and awk. These two utilities allow you to write simple scripts which process text files. We wrote about them last year Every Linux Geek Needs To Know Sed and Awk. Here's Why... Two of the most criminally under-appreciated Linux utilities are the admittedly arcane Sed and Awk. But what are they? How are they used? And how do they make it easier to process text? Read More .

Finally, if you’re confident with it, you might also want to try the vim text editor The Top 7 Reasons To Give The Vim Text Editor A Chance For years, I've tried one text editor after another. You name it, I tried it. I used each and every one of these editors for over two months as my primary day-to-day editor. Somehow, I... Read More . This has a bunch of built-in commands that make it trivial to parse through log files. The 32-bit version of vim also has a maximum file size of 2 GB, although I wouldn’t recommend you use it on files that large for performance reasons.

Using Log Management Software

If that sounds like too much hard work, or you fancy using something more visual, you might want to consider using a log management application (often confused with SIEM, or Security Information and Event Management).

What’s great about these is that they do much of the hard work for you. Many of them can look at logs, and identify issues automatically. They can also visualize logs in all sorts of pleasing graphs and charts, allowing you to get a better understanding of how reliably an application is performing.

One of the best known log management programs is called Splunk. This log management tool lets you traverse files using a web interface. It even has its own powerful and versatile search processing language, which allows you to drill down on results in a programmatic manner.

Splunk is used by countless large businesses. It’s available for Mac, Windows, and Linux. But it also has a free version, which can be used by home and small-business users to manage their logs.

This version — called Splunk Light — shares much in common with the enterprise versions. It can browse logs, monitor files for problems, and issue alerts when something is awry.

Having said that, Splunk Light does have some limitations, which are pretty reasonable. Firstly, the amount of data it can consume is limited to 500 MB a day. If that’s not enough, you can upgrade to the paid version of Splunk Light, which can consume 20 GB of logs per day. Realistically though, most users won’t get anywhere near that.

It also only supports five users, which shouldn’t be a problem for most people, especially if it’s only being run on household web and file servers.

Splunk offers a cloud version, which is ideal for those not wishing to install the whole client on their machines, or those with a number of remote servers. The downside to this is the massive cost involved. The cheapest Splunk plan costs $125.00 per month. #

That’s a lot of cash.

How Do You Handle Your Log Files?

So, we’ve looked at ways you can interrogate your log files and find the information you need to troubleshoot, either in person, or with remote assistance. But do you know of any better methods? Do you use a log management software, or the standard Linux utilities?

I want to hear about it. Let me know in the comments below.

Related topics: Linux, Troubleshooting.

Affiliate Disclosure: By buying the products we recommend, you help keep the site alive. Read more.

Whatsapp Pinterest

Leave a Reply

Your email address will not be published. Required fields are marked *

  1. Rob Burton
    October 29, 2017 at 11:23 am

    These cloud-based log management solutions all work quite well, but there are companies and cases where cloud is not an option, for obvious security reasons. When it comes to software, the best option regarding price is choosing a free software, but since security is also important, only open source projects would suffice. For that matter I wanted to point out that there is free, open source log management solution called NXLog, which has similar features as the listed cloud applications, but it's available as a free instant download. It is also able to collect logs from multiple operation systems, like Windows, Linux, Android, etc. - and provides high-perfromance even when it's scaled up to 1000s of servers. It definitely worth to be considered as a log management solution, when cloud is not an option, or when NXLog's other features or it's performance is preferred compared to other solutions.

  2. Lazza
    June 27, 2016 at 8:49 pm

    Nice post, but I see no love for ccze. :P That nifty tool is essential to color the output of log files with good patterns so that it works for any log.

    cat /var/log/apache2/error.log | ccze -m ansi

    Always add the ansi mode to prevent ccze output from suddenly disappearing sometimes.

  3. Luke
    June 25, 2016 at 9:22 pm

    You should mention the ELK stack -- elastic search, log stash and kibana. It's free, open source and can do most of what splunk does. The system requirements are better and it runs on more platforms

    • Matthew Hughes
      June 27, 2016 at 4:24 pm

      Nice. I'll check it out. I mentioned Splunk because it's what I've used personally in my professional life.