Comments on: Regular Expressions to Parse Data Files http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/ Bay View Consulting Services, Inc. Thu, 20 Nov 2008 15:35:36 +0000 http://wordpress.org/?v=2.5.1 By: William Ward http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6775 William Ward Sun, 22 Apr 2007 19:42:54 +0000 http://www.bayview.com/blog/?p=15#comment-6775 That's trickier. The way you're reading line-by-line won't let you do that. You might want to try experimenting with $/ (see the perlvar docs) to read in chunks other than line-by-line. Paragraph mode ($/="") might help you, for example. That’s trickier. The way you’re reading line-by-line won’t let you do that. You might want to try experimenting with $/ (see the perlvar docs) to read in chunks other than line-by-line. Paragraph mode ($/=”") might help you, for example.

]]> By: Benjamin http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6770 Benjamin Sun, 22 Apr 2007 17:09:48 +0000 http://www.bayview.com/blog/?p=15#comment-6770 Thanks, that works much better. I've hit another snag though. After running a number of successful regexs on the file, I am now trying to replace multiple newlines (\n\n\n) with </RECORD&gt:\n but I can't seem to change more than one \n at a time. I have tried s/\n\n\n//g s/\n{3,}//g and others etc but have had no luck. Thanks, that works much better.

I’ve hit another snag though. After running a number of successful regexs on the file, I am now trying to replace multiple newlines (\n\n\n) with </RECORD&gt:\n but I can’t seem to change more than one \n at a time. I have tried s/\n\n\n//g s/\n{3,}//g and others etc but have had no luck.

]]> By: William Ward http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6744 William Ward Sun, 22 Apr 2007 07:47:40 +0000 http://www.bayview.com/blog/?p=15#comment-6744 Yeah that's probably the best way to do it. I'd let Perl use $_ as the default variable for something like that though... while(<IN>) { s/old/new/g; print OUT; } Yeah that’s probably the best way to do it. I’d let Perl use $_ as the default variable for something like that though…

while(<IN>)
{
s/old/new/g;
print OUT;
}

]]> By: Benjamin http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6722 Benjamin Sat, 21 Apr 2007 19:24:22 +0000 http://www.bayview.com/blog/?p=15#comment-6722 How do you run multiple (or a series of) regexs over a text file when using a while ( $line = <$IN>) { $line =~ s///g; print (OUT $line); } Or is there another better method? How do you run multiple (or a series of) regexs over a text file when using a

while ( $line = <$IN>)
{
$line =~ s///g;
print (OUT $line);
}

Or is there another better method?

]]> By: William Ward http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6638 William Ward Thu, 19 Apr 2007 19:02:56 +0000 http://www.bayview.com/blog/?p=15#comment-6638 I've never used Tie::File - it sounds interesting. Why not just use a traditional approach of opening the file, iterating over each line, and doing your substitutions on each one? Or you could use the input record separator $/ to have it load one whole record at a time instead of one line at a time... I’ve never used Tie::File - it sounds interesting.
Why not just use a traditional approach of opening the file, iterating over each line, and doing your substitutions on each one?
Or you could use the input record separator $/ to have it load one whole record at a time instead of one line at a time…

]]> By: Benjamin http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6632 Benjamin Thu, 19 Apr 2007 15:29:46 +0000 http://www.bayview.com/blog/?p=15#comment-6632 **the last line should read "woth each HEADING+data on the same line, here is my $lt;/CLOSE> method from my first post. **the last line should read “woth each HEADING+data on the same line, here is my $lt;/CLOSE> method from my first post.

]]> By: Benjamin http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6631 Benjamin Thu, 19 Apr 2007 15:27:50 +0000 http://www.bayview.com/blog/?p=15#comment-6631 Thank you, but I managed to get that bit working with a bit of tinkering. I have another question though. I have a text file of about 100,000 lines that I am trying to run a series of regexs over and I am using Tie::File to feed everything into an array first. It works fine if I only give it 10,000 lines or so, but anything over that it seems to insert a couple of hundred extra lines somewhere near the end and messes up the data. Here is a sample of the code and data:- ######### AUTHOR Aagaard, L. and Schmid, M. and Warburton, P. and Jenuwein, T. TITLE Mitotic phosphorylation of SUV39H1, a novel component of active centromeres, coincides with transient accumulation at mammalian centromeres JOURNAL J Cell Sci VOLUME 113 ISSUE Pt 5 ########## tie @array, 'Tie::File', \*FILE, recsep => '\n'; for (@array) { s/\nAUTHOR\n/&\n<AUTHOR>\n/g; #AUTHOR print "AUTHOR done...\n"; s/\nTITLE\n/\n<TITLE>\n/g; #TITLE print "TITLE done...\n"; s/\nJOURNAL\n/\n<JOURNAL>\n/g; #JOURNAL print "JOURNAL done...\n"; } untie @array; There are about 2500 records in the above format in one text file and I want to turn each of the main headings into XML tags. Not evey record uses all of the headings which is why i'm trying to combine this meathod to open the tags and the method in my first post to close them (which works if I manually insert open tags with Textpad). With each " + data" on the same line, here is my method from my first post:- ($line =~ s/<([A-Z_]+)>([^<]+)/<$1>$2<\/$1>/g ) Thank you, but I managed to get that bit working with a bit of tinkering.

I have another question though.

I have a text file of about 100,000 lines that I am trying to run a series of regexs over and I am using Tie::File to feed everything into an array first. It works fine if I only give it 10,000 lines or so, but anything over that it seems to insert a couple of hundred extra lines somewhere near the end and messes up the data. Here is a sample of the code and data:-

#########
AUTHOR
Aagaard, L. and Schmid, M. and Warburton, P. and Jenuwein, T.
TITLE
Mitotic phosphorylation of SUV39H1, a novel component of active centromeres, coincides with transient accumulation at mammalian centromeres
JOURNAL
J Cell Sci
VOLUME
113
ISSUE
Pt 5
##########

tie @array, ‘Tie::File’, \*FILE, recsep => ‘\n’;
for (@array)
{

s/\nAUTHOR\n/&\n<AUTHOR>\n/g; #AUTHOR
print “AUTHOR done…\n”;
s/\nTITLE\n/\n<TITLE>\n/g; #TITLE
print “TITLE done…\n”;
s/\nJOURNAL\n/\n<JOURNAL>\n/g; #JOURNAL
print “JOURNAL done…\n”;
}
untie @array;

There are about 2500 records in the above format in one text file and I want to turn each of the main headings into XML tags. Not evey record uses all of the headings which is why i’m trying to combine this meathod to open the tags and the method in my first post to close them (which works if I manually insert open tags with Textpad). With each ” + data” on the same line, here is my method from my first post:-

($line =~ s/<([A-Z_]+)>([^<]+)/<$1>$2<\/$1>/g )

]]> By: William Ward http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6572 William Ward Tue, 17 Apr 2007 21:03:46 +0000 http://www.bayview.com/blog/?p=15#comment-6572 You can replace the < character with &lt; and > with &gt; to display angle brackets. Otherwise your angle brackets are being interpreted as an attempt at HTML. Also try using <pre> and </pre> tags around your quoted code so the indentation is preserved. You can replace the < character with &lt; and > with &gt; to display angle brackets. Otherwise your angle brackets are being interpreted as an attempt at HTML. Also try using <pre> and </pre> tags around your quoted code so the indentation is preserved.

]]> By: Benjamin http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6423 Benjamin Thu, 12 Apr 2007 21:19:58 +0000 http://www.bayview.com/blog/?p=15#comment-6423 Ah this form does not seem to like angle brackets. How can I post text containing them? Ah this form does not seem to like angle brackets. How can I post text containing them?

]]> By: Benjamin http://www.bayview.com/blog/2004/03/25/regular-expressions-to-parse-data-files/#comment-6422 Benjamin Thu, 12 Apr 2007 21:17:24 +0000 http://www.bayview.com/blog/?p=15#comment-6422 I have a text file with records in the form xxxxxxxxxxxxxxxxx and I would like to insert the end for each one. The current approach I am attempting is along these lines; while ( ) { if ($_ =~ s/(.+)()/g ) { print (OUT "$2$3") ; } } print "All done\n"; As this doesn't seem to work can you suggest an alternative? Thanks I have a text file with records in the form

xxxxxxxxxxxxxxxxx

and I would like to insert the end for each one. The current approach I am attempting is along these lines;

while ( )
{
if ($_ =~ s/(.+)()/g )
{
print (OUT “$2$3″) ;
}
}
print “All done\n”;

As this doesn’t seem to work can you suggest an alternative?

Thanks

]]>