2008-01-02: first entry
2008-02-03: second entry on two lines
    here is the additional line
2008-03-04: third entry
   has
   three
   extra lines
2008-04-05: fourth entry has just one on line again

If you need to search for all entries that have “line” in the text, and display the entire entry when found, you can’t just search line-by-line — that would work for the first and fourth entries, but the second entry would miss the additional line, and in the third entry the word “line” is on the fourth line so you’d miss the first three.

What you need to do in a case like this is read line-by-line, but only process an entry once you’ve found the end of the entry. There are two ways to solve this, depending on your data and what your needs are:

1. If the file is not very large (and never will be), and you need to do the search multiple times, then you could load the entire file into memory as an array of entries, and then search that array using grep or foreach.
2. If the file is very large, or you only need to scan through it once to find one result, then just load each entry into a string, and display that string if it matches.

First I’ll show how to load the entire file since I think it’s easier to understand:

my @stuff;
while (<IN>) {
    if (/^\s/) { $stuff[-1] .= $_; }
    else { push @stuff, $_;  }
}
print grep { /line/ } @stuff;

If the line begins with space, then it’s a continuation line, so modify the previous entry found (the last item of the array, using index -1) to add the text to it. If the line doesn’t begin with space, it’s a new entry so push it onto the end of the array. Once the entire file is read, each element in @stuff would correspond to one record, including the multiple extra lines, so it’s easy to scan using grep to find what you need.

The second approach involves using a scalar, rather than an array, to build up each record. When the next new record starts, or end of file is reached, we check to see if the record we’ve just read matches the pattern:

my $last_entry;
while (<IN>) {
    if (/^\s/) {
        $last_entry .= $_;
    }
    else {
        print $last_entry if $last_entry =~ /line/;
        $last_entry = $_;
    }
    print $last_entry if $last_entry =~ /line/ && eof(IN);
}

If you wanted to sort these, you could just use a simple sort() command: @sorted=sort(@names); That uses alphabetical order for sorting by default. The sort criteria is not given, but you could get the same results by giving a longer version of the sort function call, like so: @sorted=sort {$a cmp $b} @names;

Here, $a and $b are special variables which are used to compare two of the values in @names to see which should come first in the sort order. The cmp operator returns a positive value if $a > $b, a negative value if $a < $b, or zero if they are equal. By changing the formula that follows the sort keyword you can change the order of the sort function.

If you wanted to sort in reverse order, you could just use Perl's reverse() function: @revsort=reverse(@sorted); or in one statement, @revsort=reverse sort @names;. This is inefficient however as it must make a temporary copy of the list of names, which could get expensive if the array is large. A more efficient way is to just change the sort criteria to produce the reverse result. @revsort=sort { $b cmp $a } @names;. Now, the values returned by cmp are the opposite of what they were above, and so the sort order is the opposite.

First of all if the directory has no sub-directories, it’s easy. Just list the files sorted by size, which any operating system can do. But if there are sub-directories, or if you’re talking about the entire filesystem, it’s not so easy. Here’s a way to do it using Perl:

The Unix command “find” can recursively scan all the directories under a given point, and perform some action on each file. It has a lot of options, and if combined with commands such as “ls” and “sort” it can be done, but it is not trivial to get it right. It would be nice if we could do this within a program so we could have the full power of Perl to work with as we scan these files. For this reason, the “File::Find” module was created. It ships with Perl so you already have it on your system. And to make it easier to convert a Unix “find” command to a “File::Find” program, the script “find2perl” is included with Perl as well.

To get started, use the “find2perl” command to create the Perl script that scans the files:

% find2perl . -type f -print > find_biggest.pl

The file “find_biggest.pl” is created with a Perl script that just displays the files found. If we’re going to find the biggest files, we need to store the file sizes in some kind of data structure so that we can sort them by size. What I suggest is putting them into a hash with the filenames as keys and sizes as values. The “find_biggest.pl” script looks something like this so far:

#! /usr/bin/perl -w
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
if 0; #$running_under_some_shell

use strict;
use File::Find ();

# Set the variable $File::Find::dont_use_nlink if you're using AFS,
# since AFS cheats.

# for the convenience of &wanted calls, including -eval statements:
use vars qw/*name *dir *prune/;
*name = *File::Find::name;
*dir = *File::Find::dir;
*prune = *File::Find::prune;

sub wanted;

# Traverse desired filesystems
File::Find::find({wanted => \&wanted}, '.');
exit;

sub wanted {
my ($dev,$ino,$mode,$nlink,$uid,$gid);

(($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) &&
-f _ &&
print("$name\n");
}

The important parts to look at are the call to File::Find::find() and the subroutine wanted. File::Find will execute the subroutine once for each file that it finds. So what we need to do is modify the subroutine to record the file names, and then after File::Find::find exits, sort the files.

First, we change wanted to store the filenames in a hash rather than print them. Change it to this:

sub wanted {
my ($dev,$ino,$mode,$nlink,$uid,$gid);

(($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) &&
-f _ &&
($size{$name} = -s _);
}

Since we’re introducing a new variable %size we need to declare it: add “my %size;” just before calling File::Find::find. Then we need to sort the keys of that hash according to the values. So, after calling File::Find::find but before exit, we want to do the following:

my @files = sort { $size{$a} < => $size{$b} } keys %size;
print "Biggest file: $files[-1] (Size: $size{$files[-1]} bytes)\n";

You can download the final script here.

File::Slurp provides a function read_file, which given a filename, returns its contents as a single string if called in scalar context (in array context, returns an array of lines, as defined by whatever delimiter $/ is set to). It’s basically the same thing as setting $/ to the empty string and reading, but contained in a subroutine.

There is also a subroutine write_file, which lets you “spew” the contents of a string into a file. It saves you a few lines of code: open, print, and close. It can also be called as overwrite_file as a synonym, or you can call append_file to add to rather than overwrite a file.

Finally, read_dir lets you get the contents of a directory in one go, which is a lot more convenient than using opendir/readdir.

Say, for example, that you need to change permissions on a file. On Unix and related operating systems (including Mac OS X and Linux) you would use the chmod function, but that doesn’t do much on Windows. Although the chmod command will execute on Windows it doesn’t do much. Traditional FAT filesystems have only four attributes per file: archived (A), read-only (R), hidden (H), and system (S). These can be checked and set with the Win32::File module. For NTFS, use Win32::FileSecurity.
(For more information about Perl on Windows, see the Perl Win32 FAQ.)

So, in your program, if you want to be able to do one thing on Windows and another on Unix-like systems, you need to test what platform you are on. The way to do that in Perl is to look at the special variable $^O. (That’s the letter O, not the number 0). If you’re on Windows, it will have the value “MSWin32″ (sadly, it doesn’t give you any clue what version of Windows you’re on). So you would do something like this:

if ($^O eq "MSWin32") {
    require Win32::FileSecurity;
    Win32::FileSecurity::Set($file, { ... })
        or die "Error in FileSecurity for $file: $^E";
}
else {
    chmod 0644, $file
        or die "Error in chmod for $file: $!";
}

To find out what the value of $^O is on your platform, run this little command from your shell:

perl -e "print $^O";

my($name) = /hello, (\w+)/

In this example, we look in $_ for the word “hello” followed by a comma, space, and a word. Since the word, \w+, has parentheses around it, the part of the string that it matches gets memorized. In this example, we are assigning the return value of the regular expression match to $name. So if $_ contains “hello, world” then $name gets “world” – very convenient.

But parentheses also do other things besides memorize their contents, and this feature can become annoying. Here’s an example.

In a regular expression the | symbol indicates “or” – either the stuff to the left of it or the stuff to the right of it will match. For example, /hello|hi/ will match either “hello” or “hi” in the string. You can even have more than one of these: /hello|hi|howdy|greetings/ will match any of those four words.

The trouble is, what if you want the “or” to apply to only part of the string? That’s where parentheses come in. Let’s combine the previous two examples to show what I mean:

my($name) = /(hello|hi|howdy|greetings), (\w+)/

In this example, we want any of “hello,” “hi,” “howdy,” or “greetings”, followed by a comma, space, and a word which is memorized. The problem is, the greeting word is also memorized, and so $name gets that word instead of the name that we want it to get!

The easy solution is to allocate a variable for that word:

my($x, $name) = /(hello|hi|howdy|greetings), (\w+)/

But here, we don’t care about the value in $x so why bother allocating a variable for it? Can’t get just this one benefit of parentheses without having them memorize anything? For years, the answer was no. But then a few years back the Perl regex guys came up with a syntax to do it – just add ?: to the beginning of the parenthesized block, making it:

my($name) = /(?:hello|hi|howdy|greetings), (\w+)/

Gee, that was awfully obvious, wasn’t it? NOT! Why do they have to make these things so unintelligible? I hear you cry.

The answer is backward compatibility. Think about it – all the obvious characters already mean something, or if they don’t, chances are someone’s used them in a regular expression already to search for that character. So the only way to introduce a new feature into regular expressions is to use something that previously was a syntax error. Since the “?” character in a regex means “the previous thing zero or one times” and the thing before the “?” in this syntax is “(” (which if you recall means “start memorizing here”), it didn’t make sense to say “start memorizing here, zero or one times” so it was a syntax error. Since it was an error, nobody would have used it in an existing Perl script. So by giving (? a meaning that wasn’t a syntax error, backward compatibility is preserved.

But why “(?:” and not just “(?“? I wasn’t there, but I would assume they wanted to add more features to the parenthesized syntax and were running out of previously-bad syntax that they could give meaning to. For example, you may know that you can make a regex case-insensitive by adding /i to the end. Well, you can also insert the “i” between the “?” and “:” to make only part of the regex be case-insensitive: /(?i:hello|hi), world/ would allow “hello” or “HELLO” but “world” would have to be all lowercase.

So, the bottom line: if you find yourself wanting to use parentheses in your regex for reasons other than memorizing, and memorizing gets in your way (or you want to save a little on performance, since memorizing can slow things down a little), then just remember to insert ?: at the start of the parenthesized part of your pattern.

my($name) = /(?:hello|hi|howdy|greetings), (\w+)/

In particular, let’s say you want to go to Hawaii and your kids are in school for the spring semester from January 9 through June 2. Your travel agent gives you a list of possible dates when you can go to Hawaii really cheaply, and you want to know which ones conflict with your kids’ school schedule so you can include the budget for a babysitter in the cost of the trip.

The easiest way to compare dates in a date range is to get them into a format that you can compare using regular comparison operators. There are two typical approaches for doing this:

The first is by converting the dates to numbers. This is the way C programmers do it. You convert the dates into the number of seconds since January 1 1970 at midnight, GMT. (And you thought Perl programmers were arbitrary?) Perl supports this by providing interfaces to C library functions such as localtime() and timelocal(). You parse the dates into a set of numbers for year, month, day, hours, minutes, and seconds and pass it to the timelocal() function (which is found in the Time::Local module, natch), and out comes a big number. Do that for the start and end dates of the school calendar, and dates of your Hawaii trip, and you can compare them as numbers.

The trouble with this solution is that it’s difficult to parse dates. You have to deal with all the different formats they can be written in (or use a fussy user interface that requires them in a very particular format). You have to worry about weird things like subtracting 1 from the month so it’s in the range 0-11 (which timelocal needs) and more importantly, every time you need to write a script or module to deal with dates you have to do it all over again!

So a much better approach is to use a Perl module that has built-in support for dates. Perl doesn’t come with any but there are a couple you can download from www.cpan.org (or if you’re using ActivePerl on Windows, installed using PPM). They handle all the date parsing and can even do calculations to determine not just where the date of your trip to Hawaii lies within the semester at your kids’ school, but how many days (or minutes, or seconds…) away it is from summer vacation.

Two modules that I can suggest for doing this are Date::Manip and Date::Calc. I don’t love the API for Date::Manip, and it can be a little slow, but I love what it does. It can parse just about any date format you can think of (plus a few you probably can’t think of) and generate dates in just about any format. It can do calculations between dates and even consider business days vs. calendar days. It’s written all in Perl so you can install it easily on any platform. The other, Date::Calc, is faster and better designed, and does almost all the things Date::Manip does, but it has a C-compiled back-end which means you have to do a little more work to install it.

For now we’ll look at Date::Manip, but the same approach can be taken with the other modules such as Date::Calc. The strategy we’ll take is to use string comparison rather than numeric, and convert the dates to a common string format that can be compared easily. The format used by Date::Manip is YYYYMMDDHH:MM:SS. So in our example, your school semester runs from 2006010600:00:00 to 2006060200:00:00 (we don’t care about the time of day, so just use zeros). You get the list of dates from your travel agent and maybe each tour company returns the information in a different date format, so you get a list such as “April 1st” and “July 11″ and “2/26/2006″ and you need to parse them.

Date::Manip’s ParseDate subroutine can convert each of those to the right string format, and then getting the answer is easy using Perl’s gt and lt operators. Remember, don’t use > and < for comparing strings! (Although in this case the results would be the same, they wouldn’t if the minutes & seconds mattered. And besides if you have warnings enabled – and you should – it’ll complain when it hits the colons in the strings.)

So there you have it – run the dates through ParseDate subroutine and then just compare them as strings. The answer? 2006040100:00:00 and 2006022600:00:00 fall within the school year, but 2006071100:00:00 doesn’t. Don’t forget your sunscreen!

Technorati Tags: Perl, parsing dates, comparing dates, date formats

First, open the directory to read the list of file names in it.

  opendir DIR, $directory
      or die "Error reading $directory: $!\n";

Then, read the file names and sort according to size.

  my @sorted =
      sort {-s "$directory/$a" <=> -s "$directory/$b"}
	 readdir(DIR);

Finally, close the directory.

  closedir DIR;

Now you can print the name of the largest file.

  print "Largest file in $directory is $sorted[-1].\n";

Variations:

• To get the oldest file, use -M instead of -s.
• To get the smallest file, or the newest file, swap $a and $b.