Jeremy Mates’s Blog

Periodic jobs must often not run more than one instance at a time. Unfortunately, simple solutions often fail to account for common edge cases. For example, assume a need to synchronize files each hour with rsync. On unix, a cron job is perhaps the quickest solution:

# run just past the top of the hour # as many other things run then 7 * * * * rsync -e ssh -az --delete /source desthost:/dest

However, this solution has a major edge case that can bring down the system. Worse, simplistic attempts to fix this fault can result in rsync not running.

Technorati Tags: cron

Continue reading "Locking from Cron" »

Posted by Jeremy Mates at 14:10 | Permalink

Folks often wander into #perl, asking about “Wide character …” messages. For unknown reasons, some people are reluctant to read the perldiag documentation that explains this message, and, having been cajoled into reading the documentation, reluctant to follow the advice it outlines, instead insisting on a regex solution to perhaps strip the offending characters. An unrelated discussion at work educed two must reads regarding Unicode:

1. Thou shalt grok Unicode!
2. Unicode-processing issues in Perl and how to cope with it

More Unicode.

Technorati Tags: characters, utf8

Posted by Jeremy Mates at 10:13 | Permalink

Even after being warned of the various drawbacks of glob, a #perl user decided to use a sloppy glob call to count the number of files in a directory. Example code:

#!/usr/bin/perl -wl use strict; my $count = () = glob("/the/dir/* /the/dir/.*"); # Maybe need -2 to remove the Unix standard . and .. (and # then only on Unix! How do you know that?) print $count;

The reasons we warned against this quick and dirty code include:

• Perl glob performs lstat - even if you are only counting files, Perl is also running an lstat call against each and every file. Why even pick a solution that does this?

• The glob may not return the correct count, should something in the file path conflict with the special rules of glob. Spaces in the directory name is one way to trigger this. There might be more. Would you code with glob if it might randomly fail, especially when a solution without the wacky edge conditions exists?

The proper solution—opendir, readdir loop, and a counter—perhaps was “too long” or “not one line”, so disfavored by the user. This is bad laziness: a count_files_in_dir subroutine can contain the actual code, and be called from elsewhere. A subroutine also allows unit tests, unlike a single random line of code mixed into a larger block. Finally, the glob solution the user picked has holes in it—“oh, they would never happen”—sure they wouldn’t. For a while, and then people forget about the limited, buggy code, copy it somewhere else, and then bam! Bug.

Many small edge conditions and inefficiencies needlessly count against performance, and worse, increase the odds that some condition will tickle a bug, possibly causing a massive outage. I’ve seen a single misplaced $ in a shell script cause a multi-million dollar outage. Granted, there were also no unit tests and other best practices, but the point remains: avoid the small errors that could blow up if a condition changes, especially when other solutions abound.

A proper solution using opendir and readdir follows.

#!/usr/bin/perl -w use strict; my $directory = shift || die "Usage: $0 directory\n"; print count_files_in_dir($directory); sub count_files_in_dir { my $directory = shift; opendir( DIR, $directory ) || die "cannot open $directory: $!\n"; my $count = 0; # grep out the . and .. files on Unix here, # if necessary $count++ while readdir(DIR); return $count; }

Technorati Tags: coding, Perl

Posted by Jeremy Mates at 04:24 | Permalink

The TLS Protocol [RFC 2246] mandates that the Unix epoch time be used in the ClientHello handshake message. Unix epoch time (as presently implemented) will eventually reach a limit: the INT_MAX of 2147483647 defined in limits.h will occur at 2038-01-19 03:14:07 UTC. Without fixes to the time(3) subroutine, what happens? Will this affect TLS? Since this happens in 2038, will anyone care prior to, say, 2035?

OpenSSL 0.9.8d employs the following code in s3_clnt.c:

p=s->s3->client_random; Time=(unsigned long)time(NULL); /* Time */ l2n(Time,p); if (RAND_pseudo_bytes(p,SSL3_RANDOM_SIZE-4) <= 0) goto err;

C code that uses the same unsigned long cast is shown below. This code can be double checked by running perl -le 'print scalar time'.

#include <stdio.h> #include <time.h> main() { unsigned long Time; Time=(unsigned long)time(NULL); fprintf(stdout, "%u\n", Time); }

The unsigned long type offers greater range than the signed int used by time(3). Casting to this unsigned value prevents negative numbers from leaking into the ClientHello message, which indicates TLS should not be affected by the Unix epoch time problem.

Technorati Tags: coding, cryptography, Unix

Continue reading "OpenSSL ClientHello Concerns" »

Posted by Jeremy Mates at 21:47 | Permalink

2008-05-11 update: shUnit2 is a unit testing framework for shell scripts modeled after the JUnit framework. It is designed to make unit testing in shell as simple as possible. I have not had a chance to review it yet, though may serve shell unit testing needs better than ShUnit (detailed below) does.

ShUnit is a unit testing framework for Bourne shell code. It does not ameliorate problems with syntax errors, but will confirm that the code behaves as expected, to the depth that unit tests can be written. Scripts bereft of modular design will benefit the least, as discrete functions offer more testing opportunities. The say_something script, shown below, employs a say_something function. This function is called directly from the unit test script. Additional unit tests could also run the say_something script and parse the output. Without this modular design, the unit test would be restricted to running the script and inspecting the output.

#!/bin/sh # say_something - a testable script say_something() { echo "$1" } broken_say_something() { echo $$ } say_something "uh oh I ran"

#!/bin/sh # say_something.test - unit tests for say_something . ShUnit-1.3/shUnit # source here requires no modification of the script being tested . say_something TestSaySomething() { EXPECTED="this is a test" RESULT=`say_something "$EXPECTED"` test "${EXPECTED}" = "${RESULT}" shuAssert "Test say_something" $? } TestBrokenSaySomething() { EXPECTED="this is a test" RESULT=`broken_say_something "$EXPECTED"` test "${EXPECTED}" = "${RESULT}" shuAssert "Test broken say_something" $? } InitFunction() { shuRegTest TestSaySomething shuRegTest TestBrokenSaySomething } shuStart InitFunction

$ sh say_something.test uh oh I ran ****** say_something.test ****** 2 tests to run: Test 1: TestSaySomething . Test 2: TestBrokenSaySomething E "Test broken say_something" failed. 2 tests run. 1 test succeeded.

However, whether sourced or executed, the code tested will run, evidenced by the uh oh I ran output, above. This may require an “is this the test mode?” workaround in the script, or that all the functions be moved to a library file that both the actual and any unit test scripts read from (an excellent design idea).

After my admittedly brief investigation, shUnit requires more code per test than Test::More for Perl. However, the included shUnitPlus library contains additional convenience functions that could further simplify the shell code. As a final plus, ShUnit employs clean design and clever code, and therefore makes for an edifying study.

Technorati Tags: Unix

Posted by Jeremy Mates at 06:18 | Permalink

The & operator provides another way to check whether a number is even or odd, for example when interleaving new data following each even line number:

$ (echo 1; echo 2; echo 3; echo 4) \
| perl -nle 'print; print "stuff" unless $. & 1' 1 2 stuff 3 4 stuff $ (echo 1; echo 2; echo 3; echo 4) \
| perl -nle 'print; print "stuff" unless $. % 2' 1 2 stuff 3 4 stuff

That is all.

Posted by Jeremy Mates at 05:13 | Permalink

There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.¹

Read the article by Knuth for more context and elaboration. In Perl, a common mistake is to suffix regular expressions with the /o flag. I can personally attest to this being a bad idea: as Knuth predicted, a bug later emerged due to /o being set in one of my scripts. Instead, use qr// to create the expression, and later m/$re/.

# Good to define these early on, # not buried deep in the code. # Use a better variable name. my $re = qr{e/ge}x; … if ($string =~ m/$re/) { …

Inexperienced coders should not fret long over optimizations. First, hammer out something that works. Then have someone review it, or revisit the code some time later, asking:

• Where does a profiler show time being wasted?
• Can anything make the expensive spots less so?
• Would caching of some sort help?
• Does any of the code seem strange or awkward?
• Could a comment help explain the wacky bits?
• Is there any other way to write the code?
• Can the code be reused? Scale to new uses or more input or different output formats?
• In the future, would you code it differently?
• How long would it take to fix any problems? (Managers love time estimates.)

¹ Knuth, Donald E. "Structured Programming with go to Statements." Computing Surveys Vol. 6 No. 4, December 1974: 268.

Posted by Jeremy Mates at 07:39 | Permalink

A common Perl programming question is “how can I remove one or more elements from an array?” Some approaches to this problem have unexpected side-effects best revealed by Data::Dumper and minimal test data.

#!/usr/bin/perl -w use strict; use Data::Dumper; my @a = qw(a b c);

The following methods may or may not work correctly:

Technorati Tags: coding, Perl

Continue reading "Delete Element from Array" »

Posted by Jeremy Mates at 03:17 | Permalink

Hyphens in Filenames

Fun with deleting a file named -rf (or some other name that the shell will blindly trip over). Easier than my previous method of “find the inode, and delete by that”, or using a language that does not blow up when leading hyphens or other $IFS characters appear in filenames:

$ ls -rf $ rm -rf $ ls -rf $ rm * $ ls -rf $ rm -- -rf $ ls $ touch ./-rf $ ls -rf $ rm ./-rf $ ls $

The leading path (./) hides the otherwise “hey! a command option” from the shell. The -- only works on systems with getopt(3) libraries that support that (good luck making scripts portable, and also not prone to scary bugs or unpredictable behavior when leading hyphens or spaces sneak into the filename).

Missing Trailing Newlines

Files on Unix without trailing newlines (/proc/$A_PID/cmdline on Linux or files edited on Windows editors that omit the trailing newline by default) can cause problems. Certain crond will not run the last command listed in a crontab file, if that line does not end with a newline. Programs such as vi will warn if the trailing newline is missing, and for good reason! Utilities to debug this problem include od(1) or hexdump(1):

$ echo -n nonewline > thefile $ tail -c 4 thefile | od -bc 0000000 154 151 156 145 l i n e 0000004 $ tail -c 4 thefile | hexdump -C 00000000 6c 69 6e 65 |line| 00000004

If only shell builtins are available, such as when a system cannot fork new processes, the read builtin has trouble with files that lack a trailing newline (such as the aforementioned /proc/$A_PID/cmdline on Linux):

$ while read line; do echo $line; done < thefile $

The read failed, and nothing was printed due to the lack of a trailing character. To workaround this, simply print the variable to see the “missing” data:

$ while read line; do echo $line; done < thefile $ read line < thefile $ echo $? 1 $ echo $line nonewline $

Technorati Tags: Unix

Posted by Jeremy Mates at 06:09 | Permalink

Example why Shell Escapes in Perl are usually a mistake. Problems with the PATH and similar portability problems will not be discussed.

`find $dir $prune -o -type f -print0 \ | xargs -0 grep @Args \ | grep -v '^[ ]*--'`

No idea what the user on #perl was trying to accomplish, claimed their use case require this horrid escape. Problems include the need to run quotemeta on the interpolated variables, ensuring $prune is not empty (otherwise find(1) blows up). No idea why @Args is an array, as $" can influence how an array will interpolate, which would change the grep(1) results unexpectedly. And, more than one element would break grep…

Error handling and debugging also problematic: if nothing returned, is that expected, or due to a bug somewhere in the pipe chain ($dir not a directory or permissions problem; $prune not set; grep(1) passed regular expression with a typo; other)? Where is standard error going, in the event an exit code is not zero?

Better code would use the File::Find or similar module, read through the files, skip unwanted lines, and return the matches. More lines than the shell code takes to write, but easier to debug, and far more portable.

Technorati Tags: Perl

Posted by Jeremy Mates at 06:09 | Permalink

Apparently, programmers still write “shell scripts inside Perl”, using backticks `` and system, often where a pure Perl solution could replace the shell calls. These programmers seem ignorant of the portability, security, and maintainability problems of shell code:

• Portability

  Shell that uses find(1) with options appropriate to only one flavor of find(1) will fail if moved to a new system. Or, if the new system does have a compatible version of find(1) in a different path, errors may result if the PATH environment variable changes (a maintainability problem). Many other commands suffer from portability problems: consult Portable Shell Programming for tips on how to mitigate these issues.

• Security

  Handling backticks in Perl covers the security pitfalls of shell escapes in more detail.

• Maintainability

  Shell escapes mix a new language into Perl code, complicating syntax checking: the Perl may check out while the shell escapes still contain bugs. These would be time consuming and difficult to write unit tests for, and troublesome to debug if portability problems emerge.

  Quoting is another problem with shell escapes, especially if multiple levels of shell commands are executed. Maintaining the correct set of quotes, backslashes, and quotemeta calls is again time consuming and difficult.

Technorati Tags: Perl

Posted by Jeremy Mates at 16:11 | Permalink

In Perl, the map function can simplify the code required to SQL quote a string:

#!/usr/bin/perl -w use strict; #use Data::Dumper; my @list = ( qw{abe baker}, "can't" ); my $result = sqlify_with_map( \@list ); #print Dumper \@list; print $result, "\n"; $result = sqlify_with_for( \@list ); #print Dumper \@list; print $result, "\n"; sub sqlify_with_for { my $list_ref = shift; my $result; for my $i ( 0 .. $#$list_ref ) { my $item = $list_ref->[$i]; $item =~ s/'/\\'/g; $result .= "'$item'"; if ( $i != $#$list_ref ) { $result .= ','; } } return $result; } sub sqlify_with_map { my $list_ref = shift; return join ',', map { my $item = $_; $item =~ s/'/\\'/g; "'$item'"; } @$list_ref; }

The $item variable in both cases avoids modification of the original list elements. The map allows the escaped and quoted items to be passed as a list to join, while the for loop requires more code to decide when not to add a joining comma (leading to many false checks in a loop, which is usually not a Good Thing). Another option: always add a comma, then strip off the final comma after the for loop. But why bother, with map available?

Technorati Tags: coding, Perl

Continue reading "map for x" »

Posted by Jeremy Mates at 16:17 | Permalink

The shell exec built-in can be used without a command to perform I/O redirection:

“If no command is given except for I/O redirection, the I/O redirection is permanent and the shell is not replaced. Any file descriptors which are opened or dup(2)'d in this way are made available to other executed commands (note that the Korn shell differs here: it does not pass on file descriptors greater than 2).” — sh(1)

This allows a running shell script to change where standard output goes on the fly, for example to reproduce the gist of a “send output into files that change over time” feature as better implemented by httplog and syslog-ng:

#!/bin/sh while sleep 1; do exec >> `date +log.%H:%M` echo "The time is now `date`" done

Other uses include setting the output log after code first determines the filename based on script arguments.

Technorati Tags: coding, Unix

Posted by Jeremy Mates at 04:29 | Permalink

The strftime(3) call should replace convoluted code required after gmtime(3) or localtime(3) calls. For comparison, using Perl, two subroutines that both produce a ISO 8601 calendar date such as 2007-04-14, based on the current time:

sub date_from_gmtime { my ( $mday, $mon, $year ) = (gmtime)[ 3 .. 5 ]; $year += 1900; $mon += 1; return sprintf( '%4d-%02d-%02d', $year, $mon, $mday ); } sub date_from_strftime { use POSIX qw(strftime); return strftime( '%Y-%m-%d', gmtime ); }

strftime requires fewer lines of code, requires no local array slicing and number fiddling, and easily supports different date templates.

Technorati Tags: coding, Perl

Posted by Jeremy Mates at 01:12 | Permalink

The Z Shell (ZSH) allows complex parameter expansions, well beyond the simple $VARNAME and similar offered in traditional /bin/sh. For example, toupper(3) case changing functionality can be written without calling a program such as tr(1) or awk(1).

$ echo $PAGER less $ echo $PAGER | awk '{print toupper}' LESS $ echo $PAGER | tr a-z A-Z LESS $ echo ${(U)PAGER} LESS

Finding this feature took several minutes pawing through the zshall(1) manual, searching on the word “case”. See also bash2zsh, which includes a ZSH reference card.

Technorati Tags: Unix

Posted by Jeremy Mates at 01:53 | Permalink

Perl Best Practices (p. 56-57) warns against use constant, and advises using the Readonly module if possible. use constant constants cannot be interpolated, nor can they be created at run time:

use constant ( PI => 3 ); print 'PI is ' . PI . "\n"; use Readonly; Readonly my $PI => atan2( 0, -1 ); print "PI is $PI\n";

Additional dangers of use constant involve open calls that mistakenly pick a file handle name that conflicts with a constant:

#!/usr/bin/perl -w use strict; use constant ( PASSWD_FILE => '/etc/passwd' ); my $tmp_passwd_filename = shift || die "Usage: $0 filename\n"; # … and much later, a nasty bug results open PASSWD_FILE, '<', $tmp_passwd_filename or die "error: could not open: file=" . $tmp_passwd_filename . "\n";

Instead, always use variables to hold file handles, and avoid the risk of a constant name conflicting with a file handle. This method also avoids conflicts between subroutines imported from other modules and constants:

open my $passwd_fh, '<', $tmp_passwd_filename;

Technorati Tags: coding, Perl

Posted by Jeremy Mates at 19:03 | Permalink

Why it wasn't threadsafe. cross the road? did the chicken

Technorati Tags: coding

Posted by Jeremy Mates at 03:18 | Permalink

Output from the following code may fail validation where a downstream system expects key:value pairs and does not support empty fields at the end of the record.

print "OK:foo:" . join(':', @key_value_pairs) . "\n"

If @key_value_pairs is empty, resulting in the unacceptable OK:foo:. Safer code would include this prefix as an argument to the join function:

my @OUTPUT_PREFIX = qw(OK foo); print join(':', @OUTPUT_PREFIX, @key_value_pairs) . "\n"

This way a proper colon delimited list is generated, or an empty string should both arrays be empty.

No, I haven’t been bitten by this gotcha. No, really!

Technorati Tags: coding, Perl

Posted by Jeremy Mates at 01:32 | Permalink

Much code, even if bug free, will leave a system in an inconsistent state should the unexpected occur. The resulting mess often requires extensive time and effort fix. Once running again, the code is no more than a house of cards waiting the next bump to fly apart. Writing fault tolerant code should be considered in addition to bug and security flaw free code.

Technorati Tags: coding

Continue reading "Guarded Commands" »

Posted by Jeremy Mates at 02:43 | Permalink

#!/bin/sh FILE=$1 ls $FILE

The above Bourne shell code works as expected, until a filename with a space enters the equation. Then things fall apart:

$ cat test # !/bin/sh FILE=$1 ls $FILE $ sh test /etc/passwd /etc/passwd $ touch "a file" $ sh test "a file" ls: a: No such file or directory ls: file: No such file or directory

Harmless in this test case: the script fails with an somewhat decipherable error. At worst, highly unexpected and unlogged behavior ensues. For example, an install script shipped with iTunes 2.0 ran into this problem, and sometimes erased entire disks.

An ugly workaround entails quoting anything that might be affected by this problem:

#!/bin/sh FILE=$1 ls "$FILE"

One could also mess with the IFS variable, though that change would affect all sorts of operators that act on IFS delimited data.

I instead favor Perl over shell code, which has the benefit of avoiding the shell (excepting various unsafe functions detailed in perlsec), and offering testable code through the -c command line argument and various test modules, such as Test::More.

A quoting gotcha aside, too many scripts (for the Bourne shell or other languages) suffer from a lack of test driven development. If possible, move code to modules, where each routine may be tested, in addition to sets of routines and then the entire script. Fragile black boxes that only offer inputs and outputs with no insight into the various substeps will never be as safe.

Technorati Tags: coding, Unix

Posted by Jeremy Mates at 06:34 | Permalink

Contrary to my previous (untested!) expectations, the close(2) system call does not issue an error when closing a file on a full filesystem. Running writetest.c on Mac OS X results in fflush(2) throwing the error, and no error from the close statement.

$ writetest /Volumes/test/file error: problem flushing: file=/Volumes/test/file, errno=No space left on device $ df `pwd` Filesystem 512-blocks Used Avail Capacity Mounted on /dev/disk2s2 1032 1032 0 100% /Volumes/test

Programs, if paranoid, should check each flush or print statement (depending on the language), or checksum the data in memory, then verify what actually ends up in a temporary file, before an atomic rename(2) moves the data into place. System monitoring should also alert on high disk space use, as odds are not all programs behave properly when the disk is full. Or perhaps legal requirements frown on missing logs resulting from negligent disk space management.

Technorati Tags: close(2), coding, Unix

Posted by Jeremy Mates at 02:39 | Permalink

Not sure why Unicode contains a ☃ character, but hey…

❆ ❄ ❅ ❆ ❄ ❄ ❆

Posted by Jeremy Mates at 21:41 | Permalink

Thoughts on marking significant changes in computer systems with a somewhat formalized change notice format.

Change notices become increasingly relevant when multiple groups (Developers, Quality Assurance, Operations) manage an application. All information about the current state of systems and how they differ from the norm cannot be communicated, as the meetings would be onerous, and humans will forget things, especially at 04:20 following an hour of bad sleep. Detailed change information next to an important change, where the on call should eventually look, will clue them in, and save time wasted hunting down who made the change, or worse, enabling of jobs that should not run.

Significant changes must be commented, so another person can revert the changes to their normal state, or know why the value differs from the norm. Uncommented, the changes could be undone by someone attempting to solve another problem. This could include an application setting, such as a timeout, or whether or not special jobs run from crontab(5).

If the change will be soon reverted, include the previous values—increase timeout to support large files (was 30)—and sufficient instructions on how and when to revert the change. Periodic changes due to load, for example during the holiday season rush, may be grouped into known configuration blocks and switched between depending on the time. These blocks should use a common keyword easily searched for, or perhaps could be hosted on an wiki.

Technorati Tags: coding

Continue reading "Commenting Changes" »

Posted by Jeremy Mates at 20:45 | Permalink

Coding bugs in Unix epoch time calculations may be avoided using the graphing method outlined here. Reasoning, however logical and detailed, does not work for me: I need to reference the visual diagram before trusting the resulting code. But first, a diversion down memory lane.

HISTORY
  A time() function appeared in Version 2 AT&T UNIX 
  and used to return time in sixtieths of a second in
  32 bits, which was to guarantee a crisis every 2.26
  years. Since the Version 6 AT&T UNIX time() scale
  was changed to seconds extending the pre-crisis
  stagnation period up to a total of 68 years.

Source: OpenBSD time(3) Documentation.

Technorati Tags: coding

Continue reading "Unix Epoch Calculations" »

Posted by Jeremy Mates at 23:24 | Permalink

Great programming Recommended Reading links put together by Limbic~Region.

Technorati Tags: coding

Posted by Jeremy Mates at 22:52 | Permalink

Today’s installment of execrable Perl inspired by inexperienced coders asking koan-esque questions “how do I loop over something, but without looping over it?” Actual user question in this case turned out to be $ref->foo->foo->… iteration, which would require a slightly different recursive call. But this let me fiddle with &… prototypes to good effect.

#!/usr/bin/perl -w # Abuse recursion and so forth to # "loop without looping". use strict; # So don't need () and can specify # a sub-less sub. sub loop (&$;$); my @list = qw(a b c d); # No way known to specify @list, then # use reference in subsequent calls loop { print } \@list; sub loop (&$;$) { # Use @_ directly instead of named # variables as prototype disallows # repassing code ref (insists # on a block...) local $_ = $_[1]->[ $_[2]++ ]; eval { &{ $_[0] } }; if ($@) { # ed(1) style error messages :) die "!\n"; } $_[2] > $#{ $_[1] } || &loop; }

Posted by Jeremy Mates at 04:36 | Permalink

Quick Perl script to match dates in input data by regex, and convert between the specified time zones.

#!/usr/bin/perl -w # # Usage: convert-date < input > output use strict; use Date::Manip qw(UnixDate ParseDate Date_ConvTZ); # should be arguments my $from_tz = 'UTC'; my $to_tz = 'PDT'; my $output_format = '%Y-%m-%d %H:%M:%S'; while (<>) { # How to match the date in the input data. May # need to munge to something Date::Manip can # grok, or add HH:MM:SS, depending. s/ (\d{4}-\d\d-\d\d [T\s] \d\d:\d\d:\d\d) /fix_date($1)/ex; print; } sub fix_date { my $input = shift; my $date_in = ParseDate($input); warn "$0: error: could not parse: date=$input\n" if length $date_in < 1; my $date_out = Date_ConvTZ( $date_in, $from_tz, $to_tz ); warn "$0: error: could not convert: " . "date=$input, from=$from_tz, to=$to_tz\n" if length $date_out < 1; my $output = UnixDate( $date_out, $output_format ); return $output; }

Technorati Tags: coding, Perl

Posted by Jeremy Mates at 04:12 | Permalink

Many programs either contain poor or nonexistent logging, or log so much that any useful messages drown in the noise. This post concerns improving logs generated by scripts running on Unix, which usually suffer from the poor or nonexistent logging. Witness the succinct ! log message from the original ed(1) in contrast to Java stack trace barfs. Middle ground for usable logs can be found.

Technorati Tags: coding, Perl, Unix

Continue reading "Improve Log Messages" »

Posted by Jeremy Mates at 20:42 | Permalink

Great summary of function calling conventions for Perl.

# disfavored for some reason! push @_, 42 && &churn_away;

Technorati Tags: Perl

Posted by Jeremy Mates at 22:09 | Permalink

Process restart scripts on Unix will normally send TERM (-15) signal via kill(1) or kill(2) then move on, or send a brutal KILL (-9). Neither approach should be used: a TERM signal may leave a process running, and a KILL must only be sent as a last resort. KILL prevents cleanup of shared memory, temporary files, and other open resources. Instead, send a TERM signal, then check whether the process has exited properly. If not, only then use the KILL signal.

The following script will help test processing killing code. The script ignores the default TERM signal, requiring some other signal to stop the process, such as INT (-2, or ctrl+c) or KILL. The perlipc documentation contains more information on signal handling in Perl.

#!/usr/bin/perl -l print $$; $SIG{TERM} = 'IGNORE'; $SIG{INT} = sub { print "whoa"; exit }; sleep 3 while 1;

GNU ps contains options to match running processes, such as:

pid_check=`ps ho pid $pid` if [ -z "$pid_check" ]; then echo "info: process not running: pid=$pid" fi

However, these options do not work on other ps(1) implementations. Inspecting the result of kill -0 $pid should be more portable, though must be tested on each new flavor of Unix. Example shell code to kill a process and ensure it exits:

#!/bin/sh # Returns 0 if supplied pid not found, # 1 if still running. Back-off delay # allows slow processes to spin down. confirm_process_exit () { PID=$1 for delay in 0 1 2 3 5 8; do echo -n . sleep $delay if ! kill -0 $PID >/dev/null; then return 0 fi done return 1 } # Kill process, then ensure exits kill $1 confirm_process_exit $1 STATUS=$? if [ $STATUS -eq 1 ]; then kill -9 $1 fi

The Portable Shell Programming book covers ps(1) and other shell portability concerns. If possible, use Perl or another modern language, as the loop handling code around kill(2) will be more testable and portable than the equivalent shell code.

Slow to exit applications will also require special handling, as they may take upwards of a minute to spin down. Java embedded with Oracle… uggh.

On a somewhat related note, ensure new application code load tested before seeing production use. Systems often exhibit unexpected behavior under heavy CPU or memory load.

Technorati Tags: coding, Unix

Posted by Jeremy Mates at 05:26 | Permalink

The length of a RSA signature varies in direct proportion to the RSA key size, not the amount of data encrypted. The Perl script below demonstrates the length of signatures for several RSA key sizes. Also, larger keys allow more data to be encrypted with RSA, minus overhead for various encoding and security measures. Large amounts of data should be encrypted using a symmetric cipher, and the private key for this cipher encrypted via RSA.

#!/usr/bin/perl -wl use strict; use Crypt::OpenSSL::Random; use Crypt::OpenSSL::RSA; Crypt::OpenSSL::Random::random_status() or die "single and thine image dies with thee\n"; my $string = 'foo'; KEYSIZE: for my $ks (qw{512 1024 2048}) { my $pk = Crypt::OpenSSL::RSA->generate_key($ks); my $sig = $pk->sign($string); print $ks, ' -> ', length $sig; } __DATA__ 512 -> 64 1024 -> 128 2048 -> 256

Technorati Tags: coding, cryptography, Perl

Continue reading "RSA data length limits" »

Posted by Jeremy Mates at 07:54 | Permalink

Test Perl code illustrating how to read a file from the position last read to. Handy for log processing agents run multiple times on a growing file, where repeated scans would otherwise duplicate previous matches. Re-reads entire file if last position past end of current file contents.

#!/usr/bin/perl use Fatal qw(open); # filename => last read offset my %file_position_stash = ( test => 5 ); my $file = shift || die "Usage: $0 filename\n"; open my $fh, '<', $file; # Try to resume where left off if ( exists $file_position_stash{$file} ) { seek $fh, $file_position_stash{$file}, 0 or warn "whoa: $!\n"; # If at end of file already, file truncated # since last read? Start from beginning, unless # file same size as last read position. if ( eof $fh and $file_position_stash{$file} != -s $fh ) { seek $fh, 0, 0; } } while (<$fh>) { print; } # Save where read to $file_position_stash{$file} = tell $fh or warn "whoa: $!\n"; use Data::Dumper; warn Dumper \%file_position_stash;

If possible, avoid copying and truncating log files. Instead, use software such as httplog to direct logs into files by date-based patterns.

Technorati Tags: coding, Perl

Posted by Jeremy Mates at 04:57 | Permalink

Unix shells allow subshells, which can aggregate output from multiple commands into a single stream, or pass input to multiple commands. Aggregation allows the output from two commands to be piped into a single e-mail:

$ (echo data; echo otherdata) | mail nobody@example.org

Use the normally useless cat(1) command to join compressed and uncompressed log data together for searching:

$ ( gunzip -c log.2006-09-01.gz; cat log ) | grep …

When passing a single stream to multiple commands, typically only one of those commands handles the input. In the following example, cd ignores standard input, but does change the working directory for the subsequent mkdir. When the subshell exits, the working directory will not have been changed in the parent shell.

$ rpm2cpio some.rpm | ( cd SOURCES && cpio -id 2>/dev/null )

Note the use of &&. A cd SOUCES; cpio … semicolon between the commands will ignore any chdir(2) errors, and blindly run the cpio(1). By using &&, the cpio will only be called if the chdir succeeds. Always check the exit status of every chdir call made.

To learn more, use pwd(1) to show the current working directory:

$ cd /var/tmp; pwd | ( cat; cd /tmp && pwd ); pwd /var/tmp /tmp /var/tmp

If the command to run only accepts one argument, use either a shell loop, or xargs(1).

$ ( echo dir1; echo dir2 ) | ( cd /tmp && xargs mkdir ) $ ls -d /tmp/dir* /tmp/dir1 /tmp/dir2 $ rmdir /tmp/dir*

A shell loop allows multiple commands:

$ ( echo dir1; echo dir2 ) | \ ( cd /tmp && while read dir; do \ mkdir $dir; chmod 700 $dir; done )

However, increasing complexity on the command line at some point mandates a wrapper script to encapsulate the logic. This provides an easy to edit file that can be tested, have appropriate input checks and error handling, and be saved under a version control repository.

Technorati Tags: Unix

Posted by Jeremy Mates at 03:07 | Permalink

Ideally, all systems would use the UTC timezone, and avoid problems with daylight savings shifts and timezone conversions. However, in practice, systems will use the local timezone, even if the company later follows the British Empire. Deployed systems will not migrate to UTC, as other projects will take priority, and the timezone change would require extensive testing and likely reveal previously unknown bugs. This article contains thoughts on better handling time related data, such as logfile entries or database records.

Technorati Tags: coding, Unix

Continue reading "Timezone Troubles" »

Posted by Jeremy Mates at 02:56 | Permalink

Using CPAN with a non-root account details customizing ~/.cpan/CPAN/MyConfig.pm and the Unix shell to use a custom directory for perl modules. While the following customize the PERL5LIB and MANPATH environment variables, editing the shell code is difficult:

if [ -d $HOME/lib/perl5 ]; then PERL5LIB=${PERL5LIB:+$PERL5LIB:}$HOME/lib/perl5 fi MANPATH=${MANPATH:+$MANPATH:}$HOME/share/man export MANPATH PERL5LIB

I currently use the following, which allows multiple directories to be listed, and the ordering of the directories changed easily in a text editor:

while read pf; do PERL5LIB=${PERL5LIB:+$PERL5LIB:}$pf done << EOPERL5LIB /sw/lib/perl5 $HOME/lib/perl5 EOPERL5LIB typeset -U PERL5LIB export PERL5LIB while read pf; do if [ -d $pf ]; then MANPATH=${MANPATH:+$MANPATH:}$pf fi done << EOMANPATH /sw/share/man /sw/man /usr/share/man /usr/X11R6/man /usr/local/share/man /usr/local/man /usr/local/pgsql/man $HOME/share/man EOMANPATH typeset -U MANPATH export MANPATH

This method works for all colon delimited environment variables, such as LD_LIBRARY_PATH, CLASSPATH, and others. Duplicate supression could be added, though would require additional code.

An even better option may be to template the shell configuration files, depending on the system. This way, complex source files would be rendered into minimal shell configuration files that need not perform any number of calculations for each new shell.

Technorati Tags: Perl, Unix

Posted by Jeremy Mates at 16:53 | Permalink

The following ZSH functions, once added to ~/.zshrc and loaded, allow convenient lookup of Perl module versions and locations.

function pm-version { perl -M$1 -le "print \$$1::VERSION" } function pm-path { perl -l -M$1 \ -e "(\$mp=q{$1})=~s{::}{/}g;\$mp.=q{.pm};" \ -e "print \$INC{\$mp}" }

For example:

$ pm-version Text::Template 1.44 $ pm-path Text::Template /home/jmates/lib/perl5/Text/Template.pm

This leads to quick perldoc(1) lookups or viewing of module source code:

$ pm-path Text::Template /home/jmates/lib/perl5/Text/Template.pm $ perldoc `!!` … $ less !$ …

More perl tricks available in Perl One Liners.

Technorati Tags: Perl, Unix

Posted by Jeremy Mates at 11:26 | Permalink

Software must not emit warnings that are not errors. Needless errors clog log files, increase data processing and storage costs, and greatly complicate log analysis. At best, a new hire will debug a script, and waste time asking “is this message normal?” Better sites might Wiki “ignore this log” and hope the new hire can find it. Best sites kill off the message (or lower the severity to notice or below), and the time is never wasted wondering, documenting, and retraining. Without clear mappings of log levels to actions, one enters expression hell, where long action lists evolve: warnings X, Y, and Z require action but not M, Q, or Y. Except on Tuesday. Maintaining such lists is both time consuming and error prone.

Instead make logs actionable: specific priorities must map to specific actions. For example, a emerg or alert syslog(3) message always results in a severity 1 (highest priority) ticket and a page, crit or err messages a severity 2 ticket and page, warning messages a severity 3 ticket but no page, and no action for any lower priority. Simple to code for, and easy to decide what sort of response (and therefore priority) a new log message requires. Actionable log levels also create automation. Under Tomcat, developers could mandate any FATAL logs mean the instance requires immediate restart. Easy to check for in a log file, and automatically thread dump, kill, and restart java should a FATAL turn up.

In-house code benefits most from actionable logs. Vendor software may emit no logs, or use bizarre priority levels for trivial data: automount on Mac OS X used to log the automount version under daemon.err!. Worse, stock syslogd(8) omit the facility and priority information by default. Use monitoring software such as Nagios to trigger actionable events where vendor logs lack good information, and reserve log-based actions generated by tools like sec.pl to well known errors, such as disk full or kernel panics.

No news is good news: also eliminate spam from cron(8) jobs. Larger sites with high turnover may end up with hundreds of daily notifications, mostly junk, mixed with a few critical messages. Identify the required messages, and direct their output to role based mailing lists (never root or directly to a user), then kill off everything else. If the notification message confirms something ran, instead write a low priority log message (or touch a last-ran-on status file), then have another utility warn if the message (or the last-ran-file) was last updated too long ago.

Technorati Tags: coding, Unix

Posted by Jeremy Mates at 06:11 | Permalink

Despite years of security warnings, people still utilize /tmp on Unix systems in highly unsafe fashions. Attacks include arbitrary and unlogged file deletion against the user writing the file, where a malicious user creates a symbolic link pointing at the file to be evicted. Solutions to this problem: use secure temporary file creation routines, or locate the files in a new directory with the minimum permissions required.

• Secure Temporary File Creation

Use mktemp(1) to create temporary files. Temporary files can be shared by name between scripts, or by filehandle when using the Perl File::Temp module. See also my Term::CallEditor module, which solicits data from an external editor.

For a single temporary file under a shell script, expand on the following template:

#!/bin/sh # ensure temporary files cleaned up (unless an # improper kill -9 used on the script - but # that's a different rant!) cleanup () { rm -f $TMPFILE } # To better trace temporary file back to this script BASENAME=`basename $0` TMPFILE=`mktemp /tmp/$BASENAME.XXXXXXXX` || exit 1 trap "cleanup" 0 1 2 13 15 # add code involving TMPFILE here cleanup

Multiple temporary files could be appended into a string, which during cleanup would be removed in turn. Aadvanced shell scripting languages support arrays, though I would sooner switch to a Perl script.

For portability, try mktemp(1), then perhaps $RANDOM, and failing that a static file under /tmp. Emit a warning if mktemp(1) cannot be found, so customers can correct the deficiency.

• Status Files

The second major use of /tmp is for process tracking or similar status files. These files must have a static name, so other processes can consult the known file location. Solution: use a different directory, and set the minimum required permissions. Either overload the vendor space, using directories such as /var/run or /var/log, or create a new site specific directory structure that does not conflict with the vendor space.

Technorati Tags: Unix

Posted by Jeremy Mates at 21:39 | Permalink

Internet addresses (IP) lead to highly complex regular expressions (regex) that attempt to match only valid addresses. Regex deal poorly with number ranges, and must account for optional portions of the IP address. For example, \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} matches the invalid 999.999.999.999 string, and fails to match the valid IP address of 127.1:

$ ping -c 1 127.1 PING 127.1 (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=0.062 ms --- 127.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.062/0.062/0.062/0.000 ms

Instead, use a well tested and community supported regex from Regexp::Common or similar module. Another option: perform a loose match, then feed the results through the inet_aton and inet_ntoa functions:

$ perl -MSocket -le 'print inet_ntoa inet_aton shift' \ 127.1 127.0.0.1 $ perl -MSocket -le 'print inet_ntoa inet_aton shift' \ 999.999.999.999 Bad arg length for Socket::inet_ntoa, length is 0, should be 4 at -e line 1.

These functions also provide quick hostname resolution:

$ perl -MSocket -le 'print inet_ntoa inet_aton shift' \ sial.org 69.90.43.86

Matches may also fail if IPv4 mapped addresses appear, for example when performing OpenSSH security checks.

Technorati Tags: Perl, Unix

Posted by Jeremy Mates at 03:30 | Permalink

As seen in Mordor: the One True Coding Language! Spend a few years on an IRC channel, and experience the stream of “what’s the best language to code with?” questions and resulting hilarity. Ugh.

• Looking for a job? Pick a language used in the field or company that interests you.
• Unix systems administration? Learn shell scripting, and try out Perl, Python, or Ruby, and see what you still use five years from now (if anything).
• Experimenting? Pick something at random. Or at least from a different family of programming: functional versus object oriented and so forth.
• This language sucks! Patches welcome, or see above.

For the record, I use Perl, plus a dash of Unix shell scripting. Perl suits me perfectly: great for text processing, systems administration, and other random tasks. Huge amounts of code available in CPAN. No need to learn another language for my job (gainful computer entropy reduction), and other projects more important right now.

Next week: the One True Filesystem Layout!

Technorati Tags: coding, Perl

Posted by Jeremy Mates at 00:48 | Permalink

In addition to grep -v grep, consider the following Unix shell script improvements when using grep(1), to impress your coworkers or babes:

• Count matching lines: grep foo | wc -l. Replace with: grep -c foo.
• Find something, then apply awk(1): grep foo | awk '{print $1}'. Omit the useless grep via: awk '/foo/{print $1}/'.
• Calculate disk usage: du -k . | grep -v '/.*/' | sort -n. Old shell alias I keep around. Now mostly use my diskusage Perl script, which collects and templates the output from du(1). But I digress.
• Fun with pipes and remote systems:

ssh -n $hostname crontab -l | grep -v somecronjob | \ ssh $hostname crontab -

Though Configuration Management should be used to edit system configuration, as random shell scripts like this do not handle future systems of the same type, nor account for unavailable nodes. Also, guess what happens when the following script runs multiple times against the same system(s)?

( ssh -n $hostname crontab -l cat <<'EOF' # cleanup logfiles poorly @daily find /var/log/foodir -mtime +30 -print0 | xargs -0 rm EOF ) | ssh $hostname crontab -

Now I'm really off topic.

$ exunt stage left

Technorati Tags: Unix

Posted by Jeremy Mates at 07:21 | Permalink

The Camel Has Two Humps. Great paper on programming aptitude.

Posted by Jeremy Mates at 20:40 | Permalink

In Unix shell scripting, grep -v grep wastes yet another call to grep(1). Some Unix now ship with pgrep(1). If the script requires portability, deal with the vendor ps(1) options (BSD versus SysV), then use a regular expression that excludes grep(1) from matching itself: ps -axo pid,command | grep 'ss[h]'. Another option: first gather the results from ps(1), and loop over them in a while loop. This method avoids false positives on the username or other columns, if only searching the command:

ps -axo pid,command | while read pid command; do # deal with $pid and $command … done

Peruse this and other Unix shell scripting tips.

Technorati Tags: Unix

Posted by Jeremy Mates at 19:01 | Permalink

Possible job interview question:

“Write a program to match the first unique character in a string.”

First, clarify what character means, as a C program without libraries will act differently on Unicode data than other languages might. One solution, using Perl:

#!/usr/bin/perl # # Returns first unique character in a string passed as argument. use warnings; use strict; die "Usage: $0 the string\n" if not @ARGV; my @characters = split //, "@ARGV"; my %char_count; for my $char (@characters) { $char_count{$char}++; } for my $char (@characters) { if ($char_count{$char} == 1) { print $char, "\n"; last; } }

Note the use of warnings, strict, a script usage message, and the quick summary at the top of the script. These enable better code checks, and help people unfamiler with the script or code learn without wasting time looking through the code. Scripts without these elements should be fixed before production use. Avoid bad code from the start by creating a template for new scripts. This template should include standard option handling, leading documentation and license information, and for longer scripts, a perldoc section.

Another challenge: only use a Perl regular expression instead of the data structures shown above.

Technorati Tags: coding, Perl

Continue reading "Match the first unique character in a string" »

Posted by Jeremy Mates at 21:01 | Permalink