List comprehensions in Perl (almost)

[I wrote this about two years ago and waited for some inspiration that would make it a little better. That inspiration never showed up. In the new year I’m cleaning out all the draft articles though.]

I went off to see what list comprehensions are all about. Lately I’ve run across several bits of Python I’ve wanted to use. I don’t mind Python so much but I’m certainly rusty; I spend some time on StackOverflow where I run into the term “list comprehension” quite a bit. They sure like whatever that is so I went off to investigate. Someone might want one of those in Perl, hence the click-baity title of this post (a better one might be “12 Ways to…”).

In the wide world, these things are powerful. Programming in something like C is a good way to remind you how much work some of these features save you (or work in Smalltalk and lament that we took the wrong path). In the Perl world, you’re probably thinking “what’s the big deal?” We tend to do that a lot, but exploring features from other languages can teach us quite a bit about the one we think we know.

From the Python documentation:

List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.

In short, list comprehensions make lists out of other lists. Big deal; everyone can do that. But, the Python syntax can combine several postfix thingys into an inline expression:

[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

Python has lists of lists, so the generated list has pairs:

[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]

Written out, you have nested for loops and a condition, but you can’t use this inline with something else:

combs = []
    for x in [1,2,3]:
        for y in [3,1,4]:
            if x != y:
                combs.append((x, y))

Perl doesn’t allow multiple postfix looping controls but a nested map can do the same thing. We introduced Perl’s map at the end of Learning Perl and used it heavily in Intermediate Perl:

my $lc = [
	map {
		my $x = $_;
		map {
			$x != $_ ? [ $x, $_ ] : ()
			} ( 3,1,4 )
		} (1..3)
	];

That’s not as satisfying as the Python syntax, especially since I have to mess around with multiple $_s. Python also has builtin ways to turn objects into iterable thingys—a feature I wish Perl had.

I could try Set::CrossProduct, but this doesn’t create lazy lists and requires a module:

use Set::CrossProduct;

my $n = [
	grep { $_->[0] != $_->[1] }
		Set::CrossProduct->new( [ [1,2,3], [3,1,4] ] )->combinations
	];

The Perl6::Gather module brings in the gather feature that can add to an output list one element at a time by take-ing something:

use Perl6::Gather;

my $g = [ gather {
	for my $x ( qw(1 2 3 ) ) {
		for my $y ( qw(3 1 4) ) {
			take [$x, $y] unless $x == $y
			}
		}
	} ];

Using Perl’s prototypes, I can make grep-like behaviour although I have to pass references instead of un-referenced arrays. I push the complexity into a subroutine, but it’s still there and I have to do a lot of work that Python handles easily:

use feature qw(postderef);
no warnings qw(experimental::postderef);

sub comprehend (&$$) { # see perlsub
	my( $sub, $array1, $array2 ) = @_;

	my @results;

	foreach my $i ( $array1->@* ) {
		foreach my $j ( $array2->@* ) {
			local( $a, $b ) = ( $i, $j );
			push @results, $sub->();
			}
		}

	return @results;
	}

my @one = qw( 1 2 3 );
my @two = qw( 3 1 4 );
my $r = [ comprehend { say "a: $a b: $b"; $a != $b ? [ $a, $b ] : () } \@one, \@two ];

If I cared about this more than as an example, I’d want to do even more work to check the comprehend argument types to ensure that they are array references and in the subroutine argument to check that $a and $b are the right sort of values (although a Python list comprehension might need to do that too).

I can get around the hard-coded loops again, but the code doesn’t look that much shorter for the two array case:

use v5.22;
use feature qw(postderef);
no warnings qw(experimental::postderef);
our( $a, $b, $c );

sub comprehend (&@) {
	my( $sub, @arrays ) = @_;
	local( $a, $b, $c );

	state $loaded = require Set::CrossProduct;
	my $set = Set::CrossProduct->new( [ @arrays ] );

	my @results;
	while( ( $a, $b, $c ) = eval { $set->get->@* } ) {
		push @results, $sub->();
		}
	return @results;
	}

my @one = qw( 1 2 3 );
my @two = qw( 3 1 4 );
my $r = [ comprehend { say "a: $a b: $b"; $a != $b ? [ $a, $b, $c ] : () } \@one, \@two ];

Since $c isn’t a special Perl variable and the use v5.22 line enables strictures explicitly (see Implicitly turn on strictures with Perl 5.12), I need to declare those. I moved the hard-coded loop complexity into variable declaration complexity. That’s the Law of Conservation of Complexity at work. Complexity that I remove in one place must show up somewhere else. It all about how I get to hide that complexity and how I feel about that.

There’s a Perl trick that’s not immediately apparent there. I use $a and $b just like I’d use them in a subroutine I give to sort. That familiar interface is part of the attraction. I can do this because Perl does not warn about the variables $a and $b because it already knows they are special. If I want to work with more than two lists, I need more variables. To comply with strictures, I need to declare variables. I also need to make these package variables so the subroutine argument to comprehend nows about it:

use v5.22;
use feature qw(postderef);
no warnings qw(experimental::postderef);
our( $a, $b, $c, $d, $e );

sub comprehend (&@) {
	my( $sub, @arrays ) = @_;
	local( $a, $b, $c, $d, $e );

	state $loaded = require Set::CrossProduct;
	my $set = Set::CrossProduct->new( [ @arrays ] );

	my @results;
	while( ( $a, $b, $c, $d, $e ) = eval { $set->get->@* } ) {
		push @results, $sub->();
		}
	return @results;
	}

my @one = qw( 1 2 3 );
my @two = qw( 3 1 4 );
my $r = [ comprehend { say "a: $a b: $b"; $a != $b ? [ $a, $b, $c ] : () } \@one, \@two, \@one ];

This is still much uglier than Python in punctuation and nesting, but it works. It creates one element as a time unlike Set::CrossProduct. However, there’s almost nothing to recommend this over nested maps. I hate hardcoded nested loops, though.