This week (which includes today) saw the development and merge of asynchronous input/output operations to MoarVM and the introduction of timers to both the JVM and MoarVM backends. Here’s the details on this little piece of the puzzle:

This is not the only thing that’s happened over the week. There’s also been a release of Rakudo, NQP and MoarVM. The corresponding parrot release was sadly not terribly exciting. Pretty much all the things that are new in this release have been covered before on this blog, but you can look at the release announcement to see what we deemed exciting.

There are of course notable changes to Rakudo in general:

  • thanks to lue, Pod code blocks are now properly parsed if they are inside formatting codes.
  • there is now a :rw adverb you can pass for opening files. Nami-doc was kind enough to fix this.
  • Mouq fixed how delimiters inside a quoting construct are handled. Notably, the code q< \< > will now properly generate ” < ” rather than ” \\< “.
  • A bunch of multithreading related bugs have been found and fixed over time.
  • lizmat implemented a whole bunch of supply features (map, uniq, squish, zip, buffering, …) along with a bunch of tests

Following this week’s Rakudo release, we’ll be releasing a Rakudo Star that supports building all three backends, though only Rakudo-Parrot and Rakudo-Moar are able to pass all tests that come with the modules. This has been a long time coming, and I’m quite happy that we’ve finally reached this point :)

In other news, tadzik blogged about his new game, which is part of an on-going effort to build small games regularly and develop the game framework “Steroids”.

Finally, I’ve saved the coolest thing for last:

The Perl Foundation has been accepted for Google’s Summer of Code with the following projects:

  • brrt will be developing a JIT compiler for MoarVM. He’ll be blogging about his progress on a blog he set up for the project.
  • Chirag Agrawal will be improving the performance of Parrot’s Method Signatures implementation.
  • Filip Sergot will be working on LibWWWPerl for Perl 6 including SSL/TLS support!

And there’s two more projects that are not directly related to Perl 6:

  • skullbocks will be working on Google@Home compatibility/libraries for perl5.
  • Talina Shrotriya will be improving MetaCPAN by way of bug fixes, Web of Trust functionality and API documentation.

However, I think since Perl 6 modules are going to be on CPAN in the future, the MetaCPAN project could very well benefit us directly as well.

I have to apologize for the timing of this post; since today’s a holiday, I totally forgot it’s actually a monday until very late in the evening.

That’s it for today. I wish you all a pleasant week! :)


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-04-21 20:50:05

(I’m really sorry for the name; I couldn’t think of anything better :))

Image

 

This game, apart from (obviously) being a showcase for a new Steroids iteration, is all about switching lanes on a high traffic road in a fast car. Yay!

It’s really no rocket science compared to ThroughTheWindow from the last post – even code even looks similar. One obvious improvement (beside finally using proper PNGs instead of silly BMPs – timotimo++!) is a built-in collision detection:

my $s self.add_sprite(‘othercar’$_0);

# …

$s.when({ $_.collides_with($!player}, {

    # …

});

No more cheating with collisions like I did with ThroughTheWindow. The existing solution uses the entire image sprite as a hitbox; I’m hoping to make it customizable one day (it’s a simple thing really, code-wise).

All in all, the game isn’t all that much more sophisticated than the last one; I was really just looking for a good excuse to write a new game (and add some new stuff to Steroids), and I sort of came up with a nice theme to follow: ThroughTheWindow used just one key (spacebar), so the next step was to use two (thus RetroRacer) uses left and right arrow keys. What will the next game use? 3 keys? 4 keys? Is it an arithmetical or geometrical series? Oh my, I can’t wait to find out myself.

Now go and grab it at https://github.com/tadzik/RetroRacer, and don’t forget about the soundtrack!

 


Tadeusz Sosnierz (tadzik) | Whatever but Cool Perl | 2014-04-20 22:23:11

On behalf of the Parrot team, I'm proud to announce the supported
release Parrot 6.3.0,
also known as "Black-cheeked Lovebird". Parrot (http://parrot.org/)
is a virtual machine
aimed at running all dynamic languages.

Parrot 6.3.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/supported/6.3.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 6.3.0 News:
- Tests
+ Fixed tests for cygwin and cygwin64
+ Added 2 new examples/benchmarks/ files and benchmarks/run.sh
+ Fixed socket tests without IPv6 support at all [GH #1068]
- Community
+ New Benchmark results at https://github.com/parrot/parrot-bench
for all releases from 1.8.0 - 6.2.0


The SHA256 message digests for the downloadable tarballs are:
42aa409fa82d827019ebd218e8f9501b50e04ee81f3ccf705e03f59611317a1b
parrot-6.3.0.tar.gz
8d64df21751770741dac263e621275f04ce7493db6f519e3f4886a085161a80d
parrot-6.3.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 20 May 2014.

Enjoy!
--
Reini Urban
http://cpanel.net/ http://www.perl-compiler.org/

Perl 6 Announce | perl.perl6.announce | 2014-04-19 06:26:59

In the Perl world I’m mostly known as a guy who hacks on Perl 6 stuff. Less known is that outside of the Perl world, I spend a lot of my time with the .Net platform. C#, despite a rather uninspiring initial couple of releases, has escaped Java-think and grown into a real multi-paradigm language. It’s not Perl, but it’s certainly not unpleasant, and can even be a good bit of fun to work with nowadays. My work with it right now typically involves teaching, along with various mentoring and trouble-shooting tasks.

The Windows world has always been rather into threads – at least as long as I’ve been around. .Net is also, as a result. Want to do some computation in your GUI app? Well, better farm it off to a thread, so the main thread can keep the UI responsive to the user’s needs. Want to do network I/O? Well, that could probably use some asynchronous programming – and the completion handler will be run on some thread or other. Then the results will probably want marshaling somehow. (That used to hurt; now things are better.) Building a web application? Better learn to love threads. You don’t actually get any choice in the matter: having multiple request-processing threads is the unspoken, unquestioned, default in a web application on .Net.

Of course, just because threads are almost ubiquitous doesn’t mean the average developer – or even the above-average developer – gets things right. A bunch of my recent trouble-shooting gigs have boiled down to dealing with a lack of understanding of multi-threaded programming. “So, we embed this 80s library in our web application, but things tend to crash under load.” “How do you deal with the fact that 80s library likely isn’t threadsafe?” “It…what?” “Oh, my…”

So anyway, back to Perl 6. Last year, we managed to get Rakudo on the JVM. And, given we now ran on a VM where folks deploy heavily threaded software every day, and with no particular progress to be seen on concurrency in Perl 6 for years, I did what I usually seem to end up doing: get fed up of the way things are and figured I should try to make them better. Having spent a bunch of years working with and teaching about parallel, asynchronous, and concurrent programming outside of the Perl world, it was time for worlds to collide.

And oh hell, collide they do. Let’s go word counting:

my %word_counts;
for @files -> $filename {
    for slurp($filename).words {
         %word_counts{$_}++;
    }
}

OK, so that’s the sequential implementation. But how about one that processes the files in parallel? Well, it seems there are a bunch of files, and that seems like a natural way to parallelize the work. So, here goes:

my %word_counts;
await do for @files -> $filename {
    start {
        for slurp($filename).words {
            %word_counts{$_}++;
        }
    }
}

Here, start creates a Promise, which is kept when the code inside of it completes. That work is scheduled to be done on the thread pool, and the code calling start continues onward, moving on to create another Promise for the next file. Soon enough, the thread pool’s input queue is nicely occupied with work, and threads are chugging through it. The loop is in a context that means it produces results - the Promise objects – thanks to our use of the do keyword. We give them to await, which waits for them all to get done. Perfect, right?

Well, not so fast. First of all, are hashes thread safe? That is, if I try to write to a hash from multiple threads, what will happen? Well, good question. And the answer, if you try this out on Rakudo on JVM today, is you’ll end up with a hosed hash, in all likelihood. OK. Go on. Say what you’re thinking. Here’s one guess at a common response: “But…but…but…OH NO WE MUST MAKE IT WORK, this is Perl, not Java, dammit!” Well, OK, OK, let’s try to make it work…

So the hash ain’t threadsafe. Let’s go put implicit locking in hash access. We’ll slow down everything for it, but maybe with biased locking it won’t be so bad. Maybe we can build a smart JIT that invalidates the JITted code when you start a thread. Maybe escape analysis will save the common cases, because we can prove that we’ll never share things. Maybe we can combine escape analysis and trace JIT! (Hey, anybody know if there’s a paper on that?) Heck, we gotta build smart optimizations to make Perl 6 perform anyway…

So anyway, a patch or two later and our hashes are now nicely thread safe. We’re good, right? Well, let’s run it and…ohhhh…wrong answer. Grr. Tssk. Why, oh why? Well, look at this:

%word_counts{$_}++;

What does the post-increment operator do? It reads a value out of a scalar, gets its successor, and shoves the result in the scalar. Two threads enter. Both read a 41. Both add 1. Both store 42. D’oh. So, how do we fix this? Hm. Well, maybe we could make ++ take a lock on the scalar. Now we’re really, really going to need some good optimization, if we ever want tight loops doing ++ to perform. Like, inlining and then lifting locks…if we can get away with it semantically. Or one of the tricks mentioned earlier. Anyway, let’s suppose we do it. Hmm. for good measure, maybe we’d better ponder some related cases.

%word_counts{$_} += 1;

Not idiomatic here, of course, but we can easily imagine other scenarios where we want something like this. So, we’d better make all the assignment meta-ops lock the target too…uh…and hold the lock during the invocation of the + operator. Heck, maybe we can not do locks in the spin-lock or mutex sense, but go with optimistic concurrency control, given + is pure and we can always retry it if it fails. So, fine, that’s the auto-increment and the assignment meta-ops sorted. But wait…what about this:

%word_counts{$_} = %word_counts{$_} + 1;

Well, uhh…darn. I dunno. Maybe we can figure something out here, because having that behave differently than the += case feels really darn weird. But let’s not get bogged down with side-problems, let’s get back to our original one. My hash is thread safe! My ++ is atomic, by locks, or some other technique. We’re good now, aren’t we?

Nope, still not. Why? Turns out, there’s a second data race on this line:

%word_counts{$_}++;

Why does this work when we never saw the word before? Auto-vivification, of course. We go to look up the current scalar to auto-increment it. But it doesn’t exist. So we create one, but we can’t install it unless we know it will be assigned; just looking for a key shouldn’t make it come to exist. So we put off the installation of the scalar in the hash until it’s assigned. So, two threads come upon the word “manatee”. Both go and ask the hash for the scalar under that key. Access to the hash is already protected, so the requests are serialized (in the one-after-the-other sense). The hash each time notices that there’s no scalar in that slot. It makes one, attached to it the fact that it should be stored into the hash if the scalar is assigned to, and hands it back. The ++ sees the undefined value in the scalar, and sticks a 1 in there. The assignment causes the scalar to be bound to the hash…uh…wait, that’s two scalars. We made two. So, we lose a word count. Manatees may end up appearing a little less popular than dugongs as a result.

hue-manatee

How do we fix this one? Well, that’s kinda tricky. At first, we might wonder if it’s not possible to just hold some lock on something for the whole line. But…how do we figure that out? Trying to work out a locking scheme for the general case of auto-viv – once we mix it with binding – feels really quite terrifying, as this REPL session reveals:

> my %animals; my $gerenuk := %animals<gerenuk>; say %animals.perl;
().hash
> $gerenuk = 'huh, what is one of those?'; say %animals.perl;
("gerenuk" => "huh, what is one of those?").hash

So, what’s my point in all of this? Simply, that locking is not just about thread safety, but also about the notion of transaction scope. Trying to implicitly lock stuff to ensure safe mutation on behalf of the programmer means you’ll achieve thread safety at a micro level. However, it’s very, very unlikely that will overlap with the unspoken and uncommunicated transaction scope the programmer had in mind – or didn’t even know they needed to have in mind. What achieving safety at the micro level will most certainly achieve, however, is increasing the time it takes for the programmer to discover the real problems in their program. If anything, we want such inevitably unreliable programs to reliably fail, not reliably pretend to work.

I got curious and googled for transaction scope inference, wondering if there is a body of work out there on trying to automatically figure these things out. My conclusion is that either it’s called something else, I’m crap at Google today, or I just created a thankless PhD topic for somebody. (If I did: I’m sorry. Really. :-) My hunch is that the latter is probably the case, though. Consider this one:

while @stuff {
    my $work = @stuff.pop;
    ...
}

Where should the implicit transaction go here? Well, it should take in the boolification of @stuff and the call to pop. So any such general analysis is clearly inter-statement, except that we don’t want to hard-code it for boolification and popping, so it’s interprocedural, but then method calls are late-bound, so it’s undecidable. Heck, it’d be that way even in boring Java. With Perl you can go meta-programming, and then even your method dispatch algorithm might be late bound.

At this point, we might ponder software transactional memory. That’s very much on-topic, and only serves to re-inforce my point: in STM, you’re given a mechanism to define your transaction scope:

my %word_counts;
await do for @files -> $filename {
    start {
        for slurp($filename).words {
            # THE FOLLOWING IS IMAGINARY SYNTAX. No, I can't 
            # hack you up a prototype down the pub, it's *hard*!
            atomic { %word_counts{$_}++ }
        }
    }
}

This looks very nice, but let’s talk about the hardware for a bit.

Yes, the hardware. The shiny multi-core thing we’re trying to exploit in all of this. The thing that really, really, really, hates on code that writes to shared memory. How so? It all comes down to caches. To make this concrete, we’ll consider the Intel i7. I’m going to handwave like mad, because I’m tired and my beer’s nearly finished, but if you want the gory details see this PDF. Each core has an Level 1 cache (actually, two: one for instructions and one for data). If the data we need is in it, great: we stall for just 4 cycles to get hold of it. The L1 cache is fast, but also kinda small (generally, memory that is fast needs more transistors per byte we store, meaning you can’t have that much of it). The second level cache – also per core – is larger. It’s a bit slower, but not too bad; you’ll wait about 10 cycles for it to give you the data. (Aside: modern performance programming is thus more about cache efficiency than it is about instruction count.) There’s then a
level 3 cache, which is shared between the cores. And here’s where things get really interesting.

As a baseline, a hit in the level 3 cache is around 40 cycles if the memory is unshared between cores. Let’s imagine I’m a CPU core wanting to write to memory at 0xDEADBEEF. I need to get exclusive access to that bit of memory in order to do so. That means before I can safely write it, I need to make sure that any other core with it in its caches (L1/L2) tosses what it knows, because that will be outdated after my write. If some other core shares it, the cost of obtaining the cache line from L3 goes up to around 65 cycles. But what if the other core has modified it? Then it’s around 75 cycles. From this, we can see that pretty much any write to shared memory, if another core was last to write, is going to be incurring a cost of around 75 cycles. Compare that to just several cycles for unshared memory.

So how does our approach to parallelizing our word count look in the light of this? Let’s take a look at it again:

my %word_counts;
await do for @files -> $filename {
    start {
        for slurp($filename).words {
            %word_counts{$_}++;
        }
    }
}

Locks are just memory, so if we inserted those automatically – even if we did work out a good way to do so – then taking the lock is a shared memory write. That’s before we go updating the memory associated with the hash table to install entries, and the memory of the scalars to update the counts. What if we STM it? Even if we keep modifications in a local modification buffer, we still have to commit at some point, and that’s going to have to be a write to shared memory. In fact, that’s the thing that bothers me about STM. It’s a really, really great mechanism – way superior to locks, composable, and I imagine not too hard to teach – but its reason for existing is to make writes to shared memory happen in a safe, transactional, way. And its those writes that the hardware makes costly. Anyway, I’m getting side-tracked again. The real point is that our naive parallelization of our program – even if we can find ways to make it work reliably – is a disaster when considered in the light of how the hardware works.

So, what to do? Here’s an alternative.

# Produce a word counts hash per file - totally unshared!
my @all_counts = await do for @files -> $filename {
    start {
        my %word_counts;
        for slurp($filename).words {
            %word_counts{$_}++;
        }
        %word_counts
    }
}

# Bring them together into a single result.
my %totals;
for @all_counts {
    %totals{.key} += .value;
}
say %totals.elems;

Those familiar with map-reduce will probably have already recognized the pattern here. The first part of the program does the work for each file, producing its own word count hash (the map). This is completely thread local. Afterwards, we bring all of the results together into a single hash (the reduce). This is doing reads of data written by another thread, of course. But that’s the cheaper case, and once we get hold of the cache lines with with the hash and scalars, and start to chug through it, we’re not going to be competing for it with anything else.

Of course, the program we get at the end is a bit longer. However, it’s also not hard to imagine having some built-ins that make patterns like this shorter to get in place. In fact, I think that’s where we need to be expending effort in the Perl 6 concurrency work. Yes, we need to harden MoarVM so that you can’t segfault it even if you do bad things. Yes, we should write a module that introduces a monitor keyword, which is a class that automatically takes a lock around each of its method calls:

monitor ThreadSafeLoggingThingy {
    has @!log;

    method log($msg) {
        push @!log, $msg;
    }

    method latest($n) {
        $n < @!log
            ?? @!log[*-$n .. *]
            !! @!log[]
    }
}

Yes, we should do an Actor one too. We could even provide a trait:

my @a is monitor;

Which would take @a and wrap it up in a monitor that locks and delegates all its calls to the underlying array. However, by this point, we’re treading dangerously close to forgetting the importance of transaction scope. At the start of the post, I told the story of the hopelessly unsafe calls to a legacy library from a multi-threaded web application. I had it hunted down and fixed in a morning because it exploded, loud and clear, once I started subjecting it to load tests. Tools to help find such bugs exist. By contrast, having to hunt bugs in code that is threadsafe, non-explosive, but subtly wrong in the placing of its transaction boundaries, is typically long and drawn out – and where automated tools can help less.

In closing, we most certainly should take the time to offer newbie-friendly concurrent, parallel, and asynchronous programming experiences in Perl 6. However, I feel that needs to be done by guiding folks towards safe, teachable, understandable patterns of a CSP (Communicating Sequential Processes) nature. Perl may be about Doing The Right Thing, and Doing What I Mean. But nobody means their programs to do what the hardware hates, and the right thing isn’t to make broken things sort-of-work by sweeping complex decisions on transaction scope under the carpet. “I did this thing I thought was obvious and it just blew up,” can be answered with, “here’s a nice tutorial on how to do it right; ask if you need help.” By contrast, “your language’s magic to make stuff appear to work just wasted days of my life” is a sure-fire way to get a bad reputation among the competent. And that’s the last thing we want.


Jonathan Worthington | 6guts | 2014-04-17 00:09:34

Last week, a lot of optimization work has been done. While the week before was mostly about making NQP faster, this week (or should I call it “the last week”?) shifted the focus on optimizing pieces of Rakudo instead.

  • The spesh branch of MoarVM, which contains the bytecode specializer, has been merged into the master branch.
  • a long-standing inconsistency concerning how protos have been skipped or not depending on whether or not they have been compile-time-inlined has been fixed
  • on MoarVM, this previous bit also enables an optimization to make multiple dispatch cheaper. This will come to the JVM at some point, too.
  • Hash assignment used to use a very indirect way of getting values into the actual hash, which caused lots of unnecessary work in the most general case. There is now a method “assign_key”, that works much faster.
  • Array assignment got a similar improvement involving an “assign_pos” method.
  • Using some subs on lists, like push and unshift, used to come with a lot of extra overhead, as it created a slurpy list even if you only passed a single parameter. This rather common case is now much cheaper.
  • Rakudo’s optimizer now removes some instances of $*DISPATCHER, $_, $/, and $! if they are not used.
  • … and many more little improvements all over the place

I’ve actually posted a benchmark run with perl5, nqp-moarvm and the 2014.03 releases of rakudo-parrot, rakudo-moar and rakudo-jvm to compare them against the master branch at that time. You can find it here. It’s a bit of a mess, so I’ll point out a few things:

  • The graphs have log-log scales. One step to the right means twice the amount of work, one step up means half the time taken.
  • When hovering over a data point, it will display “$foo x times slower than fastest”. The “fastest” it refers to is the highest data point anywhere in the graph. Thus, if the graph has a little spike on the far left, or the amount of work doesn’t scale linearly with the “scaling factor” (like in the 2d visit tests, or the man-or-boy tests), you can’t rely on it. You’ll have to count grid lines instead (or … you know … submit a pull request to  the perl5 script that generates the benchmark plots)
  • Clicking on a name in the legend will toggle visibility of the corresponding line. With so many lines, you can easily lose track of what’s going on.

There’s of course not only performance improvements to be found:

  • The implementation and semantics of “winner” have been refined further by lizmat
  • Mouq has fixed both the samespace method for Str and the ss/// regex form
  • lizmat has started implementing the “is cached” routine trait, that would automatically cache return values based on all parameters.
  • SetHash, BagHash and MixHash now have minpairs and maxpairs methods that give you the value with the most or fewest occurences together with the number of occurences.
  • Using threads on MoarVM used to cause a messy shutdown, as one thread was going ahead to destroy the VM object and other threads still tried to access it. Now it properly shuts down and just lets the OS free all the resources.
  • This has been the case for a somewhat longer time, but I forgot to mention it: linenoise is now in use on MoarVM again, so the REPL actually has some nice line editing.

In the spectest repository, the following things have happened:

  • dwarring has added more and more tests based on the advent calendar blog.
  • moritz has fudged more failing tests and created RT tickets to go with them.
  • Mouq has added a couple of tests for RT tickets.

And here’s a thing I forgot to mention in last week’s post:

  • retupmoca created a module named “LibraryMake”, that greatly eases shipping native C code with your Perl 6 module. It comes with an example Makefile.in that you can adapt to your needs and it will fill in all the necessary flags and values to build a compatible library. After that, it helps you find exactly where the library was installed.

For more information about what’s been happening with MoarVM recently, you can also check out jnthn’s blog post about the topic.

As you can see in his post, jnthn will be ensuring we’ll finally get a multi-backend Rakudo * release this month. Color me excited!

Something for you to try

Currently, Rakudo sorts lists by generating a list of indices, sorting this index list based on the original values and then using list subscript splicing to re-order the objects into the correct order. This was once needed so that we could use Parrot’s built-in sort function, but on MoarVM and JVM this is quite a lot of wasted effort. If you (yes you!) would like to contribute a little something to Rakudo, find us on the IRC channel and ask for directions.


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-04-14 16:16:28

It’s been a while since I wrote an update here. Happily, timotimo has taken up the role of being our weekly Perl 6 reporter, so there’s a good place to follow for regular updates. However, I wanted to take a moment to provide the bigger picture of what’s been happening in the last couple of months, share my perspectives on it, and talk a bit about where things are headed from here.

Optimization, optimization, optimization!

A lot of recent effort has gone on optimization. NQP, the subset of Perl 6 that we use to implement much of the Rakudo Perl 6 compiler, has performance on MoarVM that starts to approach that of Perl 5, and on JVM sometimes exceeds that of Perl 5 for longer running things (it typically runs the forest fire benchmark from our benchmark suite faster once the JIT has had time to get going, for example). By contrast, Rakudo’s performance itself has been comparatively awful. Thankfully, things have been getting better, as we’ve worked to improve optimization, reduce costs of common things, and gradually begun to close the gap. This has involved work on both Rakudo and NQP’s optimization phases, along with work on improving the built-ins and some performance-oriented factoring changes. There’s still plenty of work to go, but anybody using Rakudo on MoarVM will likely feel the results in the next release. To give an idea of the improvement in HEAD Rakudo on MoarVM, which will appear in the April monthly:

  • Array and hash access is more than 3 times faster
  • Most multi-dispatches are now enormously cheaper
  • Many, many unrequired scalar allocations are optimized away
  • The forest fire benchmark on Rakudo can render twice as many frames per second
  • Compiler performance is around 15% better (estimate going on CORE.setting compile time)

Compared to Rakudo on Parrot the difference is more marked. On compiler performance alone, the difference is enormous: you can build the entire of Rakudo on MoarVM on my box in around 80s (which I’m not happy with yet, though you rarely change so much that you have to do the whole thing). That is less time than it takes for Rakudo on Parrot to complete the parse/AST phases of compiling CORE.setting (the built-ins). Running the spectest suite happens in half the time. Both of these times are important because they influence how quickly those of us working on Perl 6 implementation can try out our changes. Unsurprisingly, most of us do the majority of our development on MoarVM first these days.

One consequence of this work is that Rakudo on MoarVM is often sneaking ahead of Rakudo on JVM on some benchmarks now, even once the mighty JVM JIT kicks in. This won’t last long, though; a couple of the optimizations done will not be a great deal of work to port to the JVM, and then it can re-claim its crown. For now! :-)

Introducing “spesh”

A whole other level of performance related work has been taking place in MoarVM itself. The first goal for the project was “just run NQP and Perl 6″, and the VM simply got on with interpreting the bytecode we threw at it. That doesn’t mean it wasn’t carefully designed along the way – just that the focus in terms of execution was to be simple, correct and sufficiently complete to serve as a solid Rakudo backend. With that goal achieved, the next phase of the project is underway: implementing dynamic optimization based on program information available at runtime, speculative optimizations that can be undone if things they rely on are broken, and so forth.

The first steps in that direction will be included in this month’s MoarVM release, and are to thank for much of the improved compiler performance (since the compiler is a program running on the VM too). The easiest way to get an overview is for me to walk you through the pieces in src/spesh in MoarVM.

  • graph is about building a representation of the bytecode (at a frame level) suitable for analysis and transformation (the two steps involved in optimization). It starts out by building a Control Flow Graph. It then computes the dominance tree, which it uses to rename variables so as to produce a Static Single Assignment form of the bytecode. This is a form whereby a given name is only written to once, which eases many, many aspects of analysis.
  • args takes a tuple of incoming arguments, considers their types, arity, and so forth. It produces a set of guard clauses that indicate when a given specialization of the code applies (that is, a version of the code improved by making assumptions about what was passed), and then re-writes various argument access instructions to “unsafe” but fast ones that it can prove will always work out.
  • facts takes the graph, looks through it for sources of type information (including the incoming arguments) and does an initial propagation of that information through the graph. It creates usage counts to be later used in dead code elimination.
  • optimize takes this annotated graph, and makes a pass through it, applying a number of optimizations. Granted, there are not so many yet; so far we’ve mostly worked on getting to the point of having a framework to prove safety of transformations, and adding more of them comes next. However, those there so far can do things like:
    • Resolve methods to avoid hash lookups
    • Install monomorphic method caching if that doesn’t work out
    • Resolve type checks, possibly eliminating entire branches of code
    • Re-write attribute binds into “unsafe” pointer operations
    • Eliminate dead code
  • codegen takes the optimized graph and produces bytecode from it again. However, in the future (if we’re lucky, then hopefully through a GSoC project), this is where we would produce machine code instead.
  • deopt deals with the situation where some invariant specialized code may be relying on gets broken, and yet that specialized code is still on the call stack. It walks the call stack, looking for specialized code on it and tweaking return addresses and other data structures so that when we return into the code, we’re back in the safe (though of course slower) unspecialized code that checks things as needed.

By and large, this is essentially giving MoarVM a JIT. Of course, it’s not producing machine code yet, but rather JITing back to improved bytecode. While we tend to think of JITs primarily as “turn the program into machine code”, that’s really just one small part of any modern JIT. Analysis and specialization of the program before the machine code is produced is just as important; with this approach, we get to focus in on that part first and get some of the benefits now.

Concurrency

Progress on Perl 6 concurrency continues. The JVM implementation of the concurrency features has had various performance improvements since the March release, and MoarVM now has support for most of the Perl 6 concurrency features also. However, the MoarVM support for these features is early and most certainly not yet production ready. We’ll include it in the April release, but stick with Rakudo on the JVM for concurrency things if you’re doing anything that matters. If you just want to play with the basics, either will do.

Rakudo on MoarVM takes the spectest crown

Rakudo on its various backends hold the top spots on the Perl 6 specification test suite pass rate. However, nowadays Rakudo on MoarVM has worked its way into the lead. How so? Because it has the best Unicode database introspection support, opening up a range of tests that no other backend handles yet. Additionally, because it gets some of the Unicode stuff right that that Parrot does, but JVM doesn’t. And, finally, because on top of that it can now pass a bunch of the concurrency tests.

A multi-backend Rakudo Star

I’d hoped we would get a Rakudo Star release with support for all three backends out in March. It didn’t happen; the module tests showed up some holes. We’ve by now largely fixed those for Rakudo on MoarVM, and we’re looking quite good for the April Rakudo Star coming with support for both Parrot and MoarVM. With some effort, I’m optimistic we’ll have a JVM Star release in good shape for April too. This will provide users who want the “batteries included” release a choice of backends, and of note give those using Parrot a chance to switch over to using MoarVM, getting some substantial performance improvements on most programs and lower startup time and memory use.

Where next?

In the coming months, I’m going to be focusing on the following areas:

  • More improvements to the Rakudo and NQP optimizers, so we generate better (faster, smaller) code.
  • More improvements to the Rakudo built-ins, so they operate more efficiently.
  • Making the MoarVM concurrency support more robust, and improving the parallel GC.
  • Advancing the state of asynchronous I/O, so we’ll have good support for it on both the JVM and MoarVM.
  • Teaching spesh to specialize better. There are a bunch of data access things that can be made cheaper, as well as being able to better optimize multiple dispatch. Beyond that, both inlining and escape analysis are on the menu.
  • Improving native type support, including providing native arrays.

I’ve been hidden away coding for much of the year so far, apart from putting in an appearance at FOSDEM. But I’m getting on the road soon! I’ll be at the Dutch, Polish and Czech Perl Workshops, and look forward to seeing folks and sharing what I’ve been working on. Hope to see some of you out there!


Jonathan Worthington | 6guts | 2014-04-12 00:21:53

Hey there,

if you belong to the many people who are annoyed by the somewhat mediocre performance characteristics of rakudo, last week’s changes may get you quite excited :)

Let’s dig into the performance changes first:

  • Rakudo used to box a fresh instance of 1 for each ++ or — you put in your code. Now it just uses the ones found in the constant pool.
  • In the new and shiny “spesh” branch of MoarVM, jnthn has published his work for creating a Control Flow Graph and Single Static Assignment representation of MoarVM bytecode that can then be optimized based on a “facts database”. In particular, the following optimizations for this “bytecode specializer” have been created so far:
    • method calls on known types will – if possible – get resolved at “specialize-time”. if the type is not known, we add a little cache for the cases where the type turns out to be the same many times in a row.
    • the hllize operation (which is used to ensure things like boxed strings or integers, or arrays and such have been transferred into the right High Level Language, for example from NQP to Perl 6) gets turned into a set operation instead if the type is known and it’s already in the right HLL.
    • the decont operation can likewise turn into a set operation instead if the objects that come in are known to already be decontainered.
    • the operations to get arguments that were passed to the sub or method usually check for the proper amount of arguments passed before hand, but at the time we specialize, we already know exactly how many arguments got passed, so these operations all get replaced with quicker, specialized operations.
    • likewise, the operations to get optional parameters are all conditional jumps in order to set default values if values were not passed; these jumps are now turned unconditional at specialize-time and the code that turns unreachable gets removed from the specialized bytecode
    • operations that belong to the “if” or “unless” category that operate on known values will now be turned into unconditional jumps (or removed completely) at specialize-time.
    • if the type of something is known at specialize-time, the “istype” operation will turn into a “load a constant 0 or 1 into the register” operation, if possible.
    • a bunch of operations that usually would have to go indirectly through the REPR of the given type can now be inlined (when the type is known) and generate code that is faster and has less indirection. Currently, creating objects, getting and setting attributes, boxing and unboxing values, and finding out the number of elements an object has can be optimized by each REPR, though it has to be implemented for each REPR + operation individually. In particular, no boxing/unboxing or getattr speshes are implemented yet, but they likely take only a few minutes each to write
  • There are a few things still missing, for example information gathered from turning operations into “constants” (like istype) are not yet aggressively propagated and the specializer will currently specialize anything that gets called more than 10 times, so it will waste a lot of time on things that are not actually “hot”, but it does help find out early if the specializer does anything wrong.
  • The spesh branch will likely be merged this week in order for it to show up in the next MoarVM release. It currently regresses no spectests in Rakudo’s test suite, so that’s a good sign already!
  • After I’ve tried – unsuccessfully – for a long time to get this particular optimization off the ground, jnthn implemented a much cleaner approach at it in an evening. The optimization in question is turning lexical variables into locals if they are not used in nested blocks and then turning nested blocks into in-lined blocks if they don’t define any lexical variables in them. These two optimizations harmonize perfectly and since the specializer doesn’t know how to operate on lexicals yet, it’s worth twice as much. Sadly, this optimization is only possible in NQP so far, as the analysis and care needed to make the same thing work in Perl 6 are much more complicated. On the other hand, every piece of compilation you do now is a bit faster.
  • The work done in the spesh branch will serve as the basis for the JIT project that has been proposed for the GSoC.

And here’s the non-performance related changes:

  • FROGGS introduced a variable $*EXECUTABLE which is an IO::Path object that points to the executable used in the given process
  • FROGGS also worked on a tool to build actual executables that carry MoarVM bytecode and commandline parameters inside them. These can then be statically linked to libmoar and be completely stand-alone.
  • lizmat worked on “winner” quite a lot and through her work found out that the construct is in desperate need of a re-think and re-spec.
  • dwarring has constantly been creating test cases for the advent calendar posts of the past years
  • retupmoca has merged patches to rakudo and panda to support handling multiple state files, for example if you install some modules system-wide and other modules in your home directory.
  • vendethiel helped me flesh out the tests for the “orelse” operator. Then we found out that I have had a wrong understanding of how orelse is supposed to handle exceptions. Oh well, lesson learned!
  • last week I pointed out that the I/O subsystem of MoarVM was lacking locks. This has been addressed and the I/O stuff should now be in very good shape for concurrency.

Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-04-07 17:33:35

Announce: Rakudo Star Release 2014.03

A useful, usable, “early adopter” distribution of Perl 6

On behalf of the Rakudo and Perl 6 development teams, I’m happy to
announce the March 2014 release of “Rakudo Star”, a useful and usable
distribution of Perl 6. The tarball for the March 2014 release is
available from http://rakudo.org/downloads/star/. A Windows .MSI
version of Rakudo star is also available at that location.

In the Perl 6 world, we make a distinction between the language
(“Perl 6″) and specific implementations of the language such as
“Rakudo Perl”. This Star release includes [release 2014.03] of the
[Rakudo Perl 6 compiler], version 6.1.0 of the [Parrot Virtual
Machine], plus various modules, documentation, and other resources
collected from the Perl 6 community.

[release 2014.03]:
https://github.com/rakudo/rakudo/blob/nom/docs/announce/2014.03.md
[Rakudo Perl 6 compiler]: http://github.com/rakudo/rakudo
[Parrot Virtual Machine]: http://parrot.org

Some of the new features added to this release include:

  • The core of Rakudo::Debugger is now part of Rakudo itself and works across all backends.
  • “make” no longer itemizes its arguments.
  • for-loops at the statementlist level are now sunk by default.
  • better parsing of unspaces and formatting codes inside Pod blocks.
  • Fix for for-loops to be properly lazy
  • Numerous Pod parsing and formatting improvements
  • @ as shortcut for @$, % as shortcut for %$
  • list infix reductions no longer flatten
  • Numerous compiler suggestion improvements

Please note that this release of Rakudo Star does not support the JVM
nor the MoarVM backends from the Rakudo compiler. While the other backends
mostly implement the same features as the Parrot backend, some bits are
still missing that lead to module build problems or test failures.
We hope to provide experimental JVM-based and MoarVM-based Rakudo Star
releases in April 2014.

There are some key features of Perl 6 that Rakudo Star does not yet
handle appropriately, although they will appear in upcoming releases.
Some of the not-quite-there features include:

  • advanced macros
  • threads and concurrency (in work for the JVM and MoarVM backend)
  • Unicode strings at levels other than codepoints
  • interactive readline that understands Unicode
  • non-blocking I/O
  • much of Synopsis 9 and 11

There is an online resource at http://perl6.org/compilers/features
that lists the known implemented and missing features of Rakudo and
other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the
programmer that a given feature isn’t implemented, but there are many
that we’ve missed. Bug reports about missing and broken features are
welcomed at rakudobug@perl.org.

See http://perl6.org/ for links to much more information about
Perl 6, including documentation, example code, tutorials, reference
materials, specification documents, and other supporting resources. A
draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in
the release tarball.

The development team thanks all of the contributors and sponsors for
making Rakudo Star possible. If you would like to contribute, see
http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

rakudo.org | rakudo.org | 2014-04-01 17:48:38

A few nice things have been merged last week, but many things are still kind of in-between. Let’s see what we’ve got …

  • arnsholt has merged the “jastcompiler” branch for our JVM backend. We used to have a parrot-backed NQP generate a text representation of the bytecode that needed to end up on a JVM-backed process when we were still bootstrapping the JVM backend. That was necessary, because we were using the ASM java library to generate class files. Nowadays, we have an NQP or even Rakudo running directly on the JVM, which means we don’t have to pass serialized (textified, really) data between two processes, so turning the bytecode into text and then parsing that text directly afterwards is no longer necessary. This branch does just that. It should give a noticable improvement on compile times, but I haven’t seen anyone post any precise numbers.
  • jnthn felt it was the right time to merge the moar-conc branches into NQP and Rakudo. While they are not yet stable enough for production use, it is now easier for everyone to jump in and hunt bugs. Yes, this means you can now use Promises, Channels, Supplies, low-level Threads, … on MoarVM! However, the I/O subsystem has almost no locks in it, so doing I/O from multiple threads simultaneously is likely going to produce “interesting” results. The implementation of stdout has already got a bit of locking applied to it, so it shouldn’t break spectacularly to “say” multiple times in parallel, but other things still need looking at.
  • The German Perl Workshop happened in Hannover and there were a few presentations on Perl 6. I don’t think videos have been uploaded anywhere yet, though. Additionally, Damian Conway held a presentation on Perl 6. On top of that, lizmat and wendy are giving talks in Cluj.PM. And Larry Wall is currently touring China. I’m not sure if he’s doing talks on Perl 6; he probably is.
  • Newcomer simula67 has been improving the test suite with regards to sockets. I haven’t done much with Sockets in Rakudo, so I can’t judge the current state. But improvements in test coverage and robustness are always very nice to have!
  • Moritz has turned a lot of failing tests into Todo’d tests and corresponding tickets in RT. This way we’ll get a state of the test suite where all known-to-work tests will pass and every improvement or regression will be very easy to spot.

On a less positive note:

  • Rakudo * for the three backends is not likely going to happen this month; It would require point releases of NQP and Rakudo and we’re kind of not too fond of that idea. This month will see a regular Rakudo * with the parrot backend and next month we’ll try again – likely with much greater success.
  • I spent all my time before the game development competition deadline trying to get PNG images loaded properly just to realize in the final moments that I’ve been linking SDL_image rather than SDL2_image. D’oh!

And some things that are coming up:

  • jnthn said, he’ll have something showable for MoarVM’s type specializing subsystem this week.
  • CPAN, PAUSE and Panda are moving towards a compatible position. There’s some exciting stuff coming up in this area!
  • At the time of writing this post, arnsholt is working to refactor the way operators are defined in the grammar of NQP and Rakudo. It may give a slight improvement to compile times, but it’s mostly crufty legacy code being replaced by much saner code – and less code overall!

I wish you a pleasant week! You can play with MoarVM’s very latest development code and let us know what oddities you encounter! :)


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-03-31 17:03:33

Last week, there has been a Rakudo release (the Rakudo * releases happen later in the month), which gives you most of the changes I’ve talked about in the recent weeklies. Sadly, there was a bug in the way GC threads pass work to each other in the MoarVM Concurrency branch. Therefore, that branch hasn’t been merged for the release.

We didn’t bump the minimum parrot revision to the 6.2.0 release yet, which would have given us some optimizations (I’ve yet to run the benchmarks), but next month’s parrot release will likely bring handling environment variables back to the qx quoting construct, so I’ll bug the release manager to bump parrot in any case :)

arnsholt has been working on the JAST compiler – an intermediate stage of compiling NPQ or Rakudo QAST to JVM .class files. So far, it has been creating a string representation of a tree, which was then parsed by some java code. With the new branch, the java code will directly get passed proper data structures, thus eliminating redundant parsing.

brrt has pushed his GSoC application out to hopefully be accepted to work on a JIT compiler for MoarVM. jnthn will be his mentor and until the GSoC starts, he’ll have made public his work towards control flow graph generation and bytecode specialization. I’m really looking forward to that!

dwarring has added a few spectests for the p6advent 2013. It’s always good to immediately see when a change to the implementation or spec causes these texts to become outdated. Though I don’t know yet if we have yet decided how to exactly handle outdated posts from years ago.

Spectest-wise, Rakudo-moar is still at the 100% spot with Rakudo-jvm at 99.27% and Rakudo-parrot is now down to 99.00% relative to Rakudo-moar. I imagine that is the good Unicode DB support we have in MoarVM. When the concurrency branch of MoarVM is merged to master, that lead will increase even more. Oh my!

I kept the best thing for last: tadzik has been working on a tiny game development framework on top of SDL2 and has made an example game with that. On top of that, he’s called out a game development contest that’ll go until sunday. Find all the details on his blog.

I wish you all a pleasant week! :)


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-03-24 18:05:32

Image

 

I got into programming because I wanted to write games. I played games as a kid (wolfenstein 3d, putt-putt, I don’t remember the rest of the names), and I thought “when I grow up, I’m going to write games!”

I never really did; at some point I realized I’ve written more compilers than games: whatever happened to the childhood dream? So I thought I’ll write some, to try and learn something new.

With two friends from work I went to javascript game programming conference – it was the only game conf I’ve ever heard of, so I thought “javascript or not, let’s see how gamedev geeks party”. The universal “let’s create idiotic games and make a lot of money on ads and In-App-Purchases” attitude of the startup crowd discouraged me a little bit, but I tried to ignore that bit and focus on the technical content. I never liked javascript, I didn’t really want to use it for any kind of programming, and frankly, working in Perl and Python I grew tired of dynamic typing altogether, but I thought “hmm, maybe if I created a superset of JS that has nice type annotations, that the compiler checks and then drops, emitting vanilla JS, that wouldn’t be so bad to write code in”. I consulted a friend of mine, and, as it usually happens, it turned out that such thing already exists: it’s called Typescript, and Microsoft created it long ago. Oh well, let’s give it a try.

Why am I writing about all this? Where does Perl 6 come in? Thing is, when I started programming in Typescript, I got annoyed. It’s severly underdocumented, undersupported, development is not pleasant, because you get some errors from the compiler and then different errors from a browser, but the worst thing is: it was slow! It was so slow it was unbearable, and I thought “ah, screw it. I’ll be better off with Perl 6″.

I chose Perl 6 for performance reasons! Ain’t that something to tell my grandkids about.

Of course, creating games in Perl 6 is not so trivial: I’ll have to write the engine/framework/whatever myself. Time to roll the sleeves up and get to work.

I got quite motivated by http://lessmilk.com. This guy creates a new game every week to learn game development. Cool thing! He was describing phaser.js in one of his articles, so I created Steroids, and modelled it after Phaser.

Why steroids? Well, at some point I ported my C Asteroids to Perl 6, as a proof of concept, to see if it can indeed handle 60fps games (it can), and the “steroids” bit somewhat got stuck in my mind. Also, being on steroids gives you much more flexibility than, say, being on the rails. Don’t worry, nothing bad about being on Steroids: Just ask Duke Nukem, he got by just fine.

So, Perl 6 on Steroids was born. I started writing a running-jumping game, and abstracting the commonly used bits to a module as I went on. Why a running-jumping game? Well, you asked for it: it’s time for another backstory:

Ever wrote in a backseat of the car as a kid, looking out the window? Did you imagine a person running along the car, jumping over obstacles? I did, and from what I’ve heard I am not the only one. Thus, “Through the Window” was born: a game where a man runs along the horizon, jumping over trees and cows, being the first showcase for Steroids, and a reason for it to exist.

The post is getting lenghty enough, and there’s much to announce still, so I’ll run through the 80 lines of code really quickly to show you what Steroids gives you. You can read the entire source code here

class Game is Steroids::Game

You create a class that inherits from Steroids::Game. You need to define at least two methods for it to make any sense: create() and update(). The former initializes the game, and the latter is called 60 times per second to update the game state.

Some of the things you may want to do in the create() method:

self.load_bitmap(‘background’, ‘assets/background.bmp’);

self.add_sprite(‘background’, 0, 0);

Pretty self-explanatory. Steroids handles loading bitmaps from disk for you, and putting them in a scene.

self.load_bitmap(‘runner’, ‘assets/runner.bmp’);

$!runner = self.add_sprite(‘runner’, 50, GROUNDLEVEL);

$!runner.y -= $!runner.h;

$!runner.gravity = 1;
$!runner.when({ $_.y > GROUNDLEVEL – $_.h }, {
$!runner.y = GROUNDLEVEL – $!runner.h;
    $!runner.velocity[1] = 0;
})

;

Plenty of things here: we add a runner to the scene. We move him up a little bit, so he’s actually standing on the ground, rather than having his head on the ground level, we give him a gravity (so he’s falling down all the time), and we add an event, so when he hits the ground with his feet we stop him, so he doesn’t fall any further down. This should probably be handled by a collision detection at some point, but I didn’t get around to write collision detection yet.
 
That’s the interesting part from create(), now let’s look at update() really quickly.
 
method update {
        if @!obstacles
        and @!obstacles[0].x < ($!runner.x + $!runner.w) < (@!obstacles[0].x + @!obstacles[0].w)
        and $!runner.y + $!runner.h > @!obstacles[0].y {
            say “===========================================”;
            say ” Game over! You traveled $!distance pixels “;
            say “===========================================”;
            say “”;
            self.quit;
        }
Here’s our half-assed collision detection: if the front foot of the runner is inside the first obstacle (so, the leftmost), then the game is over.
 
How do the obstacles get there in the first place?
 if $!distance > $!last-obstacle + 40 and rand > 0.9 {
     my $s = self.add_sprite(<cow tree>.pick, self.width, GROUNDLEVEL);
     $s.y -= $s.h;
     $s.velocity[0] = -12;
     @!obstacles.push: $s;
     $s.when({ $_.x < 0 – $_.w }, {
     self.remove_sprite($_);
         @!obstacles.shift;
     });
     $!last-obstacle = $!distance;
}
If enough time has passed since we put an obstacle on the road (we don’t want the road to be impossible to travel), we add either a tree or a cow on the ground, as far on the right as we can. We make it slowly move to the left, add it to the list of obstacles, and add an event so when it reaches the left edge of the screen we remove it from the scene.
This part features a hack: @!obstacles.shift removes the first element of the list, and it just so happens that the object disappearing from the scene will always be the first one on the list: we don’t need to look through @!obstacles to find which one it is.
 
if self.is_pressed(“Space”) and $!runner.y == GROUNDLEVEL – $!runner.h {
     $!runner.velocity[1] = -22;
}
Pretty obvious: if Space is pressed while the runner is on the ground, he bounces upwards.
 
That’s just about all that it’s there to describe. Go play around with it, and remember about the soundtrack (in the README) – it’s a very important part of the game :)
 
(I was informed that the build process can be a bit more complicated on OSX; the entire Steroids development team is working hard to fix it, but if you have a good and ready solution, please send me a pull request).
 
But wait, there’s more! To celebrate the best game I’ve ever written, I’m announcing a contest: The task is to write a game using Steroids (with as many patches to it as you want). Let’s see how much can we squeeze out of those 120 lines of code to create something fun. One week from now, next sunday, I’m going to pick a winner and reward the author with a game that I like, and the author doesn’t yet have. Have an inappropriate amount of fun!

Tadeusz Sosnierz (tadzik) | Whatever but Cool Perl | 2014-03-23 14:36:45

"How can I parse indented text with a grammar?" has turned into a frequently-asked question recently. People want to parse Python and CoffeScript.

My fix is double. First, here's Text::Indented, a module that does it.

Secondly, I'll now recreate my steps in creating this module. Each section will have a description of what needs to be done, a failing test, and then the appropriate implementation code to pass the test.

Quite a simple indent

We want to be able to handle indentation at all.

    my $input = q:to/EOF/;
    Level 1
        Level 2
    EOF

    parses_correctly($input, 'single indent');

Well, that's easy. This grammar will do that:

regex TOP { .* }

(Kent Beck told me I can cheat, so I cheat!)

Too much indent for our own good

But there are some indent jumps that we're not allowed to make. Anything that indents more than one step at a time, basically. Let's check for that.

    my $input = q:to/EOF/;
    Level 1
            Level 3!
    EOF

    fails_with($input, Text::Indented::TooMuchIndent);

This takes a little more code to fix. We declare an exception, start parsing lines, and separate each line into indent, extra whitespace, and the rest of the line. Finally we check the line's indent against the current indent — mediated by the contextual variable @*SUITES. You'll see where I'm going with this in a minute.

class TooMuchIndent is Exception {}

constant TABSTOP = 4;

regex TOP {
    :my @*SUITES = "root";

    <line>*
}

sub indent { @*SUITES.end }

regex line {
    ^^ (<{ "\\x20" x TABSTOP }>*) (\h*) (\N*) $$ \n?

    {
        my $new_indent = $0.chars div TABSTOP;

        die TooMuchIndent.new
            if $new_indent > indent() + 1;
    }
}

(The <{ "\\x20" x TABSTOP }> is a bit of a workaround. In Wonderful Perl 6 we would be able to write just [\x20 ** {TABSTOP}].)

Actual content

Having laid the groundworks, let's get our hands dirty. We want the content to end up, line by line, on the right scoping level.

    my $input = q:to/EOF/;
    Level 1
        Level 2
    EOF

    my $root = parse($input);

    isa_ok $root, Text::Indented::Suite;
    is $root.items.elems, 2, 'two things were parsed:';
    isa_ok $root.items[0], Str, 'a string';
    isa_ok $root.items[1], Text::Indented::Suite, 'and a suite';

We need a Suite (term borrowed from Python) to contain the indented lines:

class Suite {
    has @.items;
}

This requires a slight amending of TOP:

regex TOP {
    :my @*SUITES = Suite.new;

    <line>*

    { make root_suite }
}

The logic in line to create new suites with new indents:

# ^^ (<{ "\\x20" x TABSTOP }>*) (\h*) (\N*) $$ \n?

my $line = ~$2;

if $new_indent > indent() {
    my $new_suite = Suite.new;
    add_to_current_suite($new_suite);
    increase_indent($new_suite);
}

add_to_current_suite($line);

For all this, I had to define some convenience routines:

sub root_suite { @*SUITES[0] }
sub current_suite { @*SUITES[indent] }
sub add_to_current_suite($item) { current_suite.items.push($item) }
sub increase_indent($new_suite) { @*SUITES.push($new_suite) }

But what about de-indenting?

We've handled indenting and creating new suites nicely, but what about de-indenting?

    my $input = q:to/EOF/;
    Level 1
        Level 2
    Level 1 again
    EOF

    my $root = parse($input);

    is $root.items.elems, 3, 'three things were parsed:';
    isa_ok $root.items[0], Str, 'a string';
    isa_ok $root.items[1], Text::Indented::Suite, 'a suite';
    isa_ok $root.items[2], Str, 'and a string';

Easily fixed with an elsif case in our line regex:

elsif $new_indent < indent() {
     decrease_indent;
}

And a convenience routine:

sub decrease_indent { pop @*SUITES }

Hah, you missed multi-step de-indents!

Indenting multiple steps at a time isn't allowed... but de-indenting multiple steps is. (This may actually be the strongest point of this kind of syntax. It corresponds to the } } } or end end end case of languages with explicit block delimiters, and is arguably neater.)

    my $input = q:to/EOF/;
    Level 1
        Level 2
            Level 3
            Level 3
    Level 1 again
    EOF

    my $root = parse($input);

    is $root.items.elems, 3, 'three things on the top level';
    is $root.items[1].items[1].items.elems, 2, 'two lines on indent level 3';

Oh, but we only need to change one line in the implementation to support this:

decrease_indent until indent() == $new_indent;

And a half!

Now for some random sins. You're not supposed to indent partially, a non-multiple of the indent size.

    my $input = q:to/EOF/;
    Level 1
          Level 2 and a half!
    EOF

    fails_with($input, Text::Indented::PartialIndent);

So we introduce a new exception.

class PartialIndent is Exception {}

And a condition that checks for this:

# ^^ (<{ "\\x20" x TABSTOP }>*) (\h*) (\N*) $$ \n?

my $partial_indent = ~$1;

die PartialIndent.new
    if $partial_indent;

What do you mean, "jumped the gun"?

Secondly, you're not meant to indent the first line; it has to be at indentation level 0.

    my $input = q:to/EOF/;
        Level 2 already on the first line!
    EOF

    fails_with($input, Text::Indented::InitialIndent);

We introduce another exception for that.

class InitialIndent is Exception {}

And a condition that matches our test case.

die InitialIndent.new
    if !root_suite.items && $new_indent > 0;

The importance of handles

As a final clean-up refactor, let's change @.items in Suite to this:

class Suite {
    has @.items handles <push at_pos Numeric Bool>;
}

It makes Suite more Array-like. Piece by piece:

  • push allows us to push directly into a Suite object, instead of into its .items attribute.
  • at_pos allows us to index Suites directly. Things like $root.items[1] in the tests turn into $root[1].
  • Numeric gets rid of the .elems calls for us in the tests, and we can write $root.items.elems as just +$root instead.
  • Finally, Bool allows us to write !root_suite.items as just !root_suite().

Somehow I liked doing this refactor last, after all the dust around the implementation had settled. It makes the API much more enjoyable to use, and hides a bunch of unnecessary steps along the way. I really like the way handles saves a bunch of boring code.

Enjoy

Anyway, that's parsing of indented code. Not as tricky as I thought.

Now I fear I've damned myself to contribute this solution to arnsholt++'s budding py3k implementation. 哈哈

Carl Masak | Strangely Consistent | 2014-03-23 01:12:03

Beside him, Melvin and Lavender and Allen all seemed to feel like marching too.
And Neville softly began to sing the Song of Chaos.

The tune was what a Muggle would have identified as John Williams's Imperial
March, also known as "Darth Vader's Theme"; and the words Harry had added were
easy to remember.

Doom doom doom
Doom doom doom doom doom doom
Doom doom doom
Doom doom doom doom doom doom
DOOM doom _DOOM_
Doom doom doom-doom-doom doom doom
Doom doom-doom-doom doom doom
Doom doom doom, doom doom doom.

By the second line the others had joined in, and soon you could hear
the same soft chant coming from nearby parts of the forest.

And Neville marched alongside his fellow Chaos Legionnaires,
strange feelings stirring in his heart,
imagination becoming reality,
as from his lips poured a fearful song of doom.

-- Harry Potter and the Methods of Rationality
http://hpmor.com/ chapter/30

On behalf of the Parrot team, I'm proud to announce Parrot 6.2.0, also known
as "Imperial Amazon". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 6.2.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/6.2.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 6.2.0 News:
- Core
+ Re-enable old immc flags for parrot and parrot_old, such as
-On -a -v -y -E -dxx. [GH #1033]
+ Fixed imcc -O1 and -O2
-O1 fixes:
= Special-case get_global for branch_cond_loop_swap, which broke
NCI tests [GH #1037]
= set_addr label does mark a basic_block, dead_code_remove() needs
the label. Fixes nqp [GH #1061].
-O2 used_once fixes:
= Allow used_once elimination only for purely functional ops
without side-effects [GH #1036]
= Empty ins->next in deletion [GH #1042].
-O2 constant propagation fixes:
= Empty ins->op ptrs when deleting or skipping certain instruction
[GH #1039],
= Wrong logic leading to missed detecting writes from get_results
[GH #1041],
= Propagate only matching types in setters [GH #1042],
= Stop at yield or invokecc for possible push_eh/pop_eh non-local
effects [GH #1044]
+ Fixed TT #1930, a floating point optimizer problem on PPC
+ Added cache iterators in encoding_find_*cclass [GH #1027]
to speedup the utf8 pattern "scan a whole string for newlines".
- Build
+ Set emacs buffer-read-only:t tags in generated files [GH #1034]
+ Provide coda for generated include/*.pasm files [GH #1032]
+ Fix parser for bison 3 [GH #1031]
+ Add support for __builtin_expect LIKELY/UNLIKELY branch optimizations
in a new auto::expect step. [GH #1047]
- Deprecations
+ Warn on deprecated usage of append_configure_log()
- Documentation
+ Updated pod for parrot and tools/build/c2str.pl
- Tests
+ Added -O1 and -O2 to fulltest
- Community
+ Parrot has been accepted to Google Summer of Code 2014
+ Got a candidate for "Improve performance of method signatures"


The SHA256 message digests for the downloadable tarballs are:
a4c97e5974cf6e6ee1e34317aafd2d87a3bd63730098a050d4f09802b13da814 parrot-6.2.0.tar.gz
f8b9cd2d558a1517038dc3154343f622ab1fd7b1f1d13f41a5c6dd51425bfe8e parrot-6.2.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 15 Apr 2014.

Enjoy!

Perl 6 Announce | perl.perl6.announce | 2014-03-21 17:36:08

This week has seen changes spread across many different pieces of Perl 6. Let’s have a  look!

  • rurban has gotten the -O2 pbc optimization enabled for parrot and tomorrow’s parrot release will feature that. He measured some 3-5% improvement, but hasn’t benchmarked Rakudo thus far. If nobody beats me to it, I may run some benchmarks this week.
  • rurban has also set up a bunch of buildbots, among others for parrot, NQP and Rakudo.
  • I’ve missed it last week, but lizmat has been working diligently and effectively on both S11 (modules) and S22 (package format) during these last two weeks!
  • At the same time, FROGGS has been moving the integration of PAUSE (the perl programming author’s upload server) with Perl 6 and it sounded like there’ll be a big announcement this or next week regarding that whole complex :)
  • Another thing lizmat did (the week before this one) was improving error messages for when you typo a routine or try to call one that doesn’t exist as well as helpful messages about trying to use .length and .bytes methods on stringy things being banned from Perl 6 (because length and bytes are very ambiguous when applied to a string with regards to Unicode and combining characters and stuff)
  • jnthn has advanced NativeCall support on MoarVM to the point where it passes all tests! These changes are already in the master branch of MoarVM, NQP, Rakudo and zavolaj (aka NativeCall)
  • jnthn has started the “tristar” branch in the rakudo/star repository to hopefully get a three-backend Rakudo Star distribution this month. Mouq and me helped him a bit. Neither of us three are really good at doing build system stuff, though …
  • just as I was writing this, rurban started fixing a nasty regression that made changes to %*ENV not propagate to qx (execute shell commands and return the output as a string) on Rakudo for Parrot.

And here’s a few things people have in the pipe for the near future:

  • as mentioned above, FROGGS is still working on ecosystem/cpan/PAUSE/versioning/installation things
  • lizmat is likely going to continue work on the S11 and S22 synopses
  • I’ve found out that during the CORE.setting compilation, a few strings occur in MoarVM’s generation two more than a few thousand times each. By introducing string interning in the right place, this might bring memory usage down by a significant amount, but my first attempt at string interning just caused mysterious compilation failures.
  • there’s quite a bit of momentum towards having tristar finished for this month’s release, so I’m very much looking forward to that
  • MoarVM concurrency support is probably going to land after this month’s release, so that it can be tested and debugged sufficiently before it gets unleashed on the world.
  • rurban is going to work on a JIT compiler for parrot at some point in the near future
  • Mouq has begun a pure-perl GIF decoder as well as a library to read tar files
  • hoelzro is likely going to have more time for Perl 6 in the future, so I’m hoping for a complete highlighter for Kate (and thus QtCreator) in a relatively short amount of time :)
    • sadly, there’s a billion incompatible syntax highlighting definition schemes … it would be fantastic if we could generate many highlighters from a common format, but I fear that’ll remain a pipe dream.

Not quite bad progress, considering a big portion of the Perl 6 developers have been slowed down by 2048!

Well, I probably forgot to mention something cool again. I’ll probably put it into next week’s post :)


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-03-17 18:27:53

Let’s make this short and sweet:

  • The somewhat long-standing precompilation related bug in MoarVM has been fixed. This brings us quite a bit closer to Rakudo * on MoarVM.
  • A few memory leaks inside MoarVM have been fixed. MoarVM doesn’t rely exclusively on its own garbage-collected heap. It also uses malloc and free for things like C strings and it also has reference counting semantics for frames. These were a little bit buggy. During the core setting compilation, about 100 megabytes of ram are saved by this.
  • MoarVM got some improvements to its unicode database and now offers operations to query it directly. In combination with a few spec changes to the Unicode Synopsis and the corresponding tests, MoarVM is now the leading implementation in number of passing spectests.
  • jnthn has just implemented nqp::queuepoll on MoarVM and except for a bit of design work left for the scheduler, there’s not much keeping the moar-conc branches – containing concurrency support for MoarVM – from being merged.
  • Mouq has continued pushing more fixes and improvements to Pod6 parsing and rendering to rakudo and Pod::To::HTML.
  • Coke has recently started porting Mojolicious to Perl 6. So far it’s only the Mojo::Util package, but it’s surely something to keep an eye on for perl5 users.

What’s cooking?

pmurias has resurfaced and is currently preparing nqp-js for a merge into the main NQP repository.

rurban has started working on parrot performance. Just today he got NQP to compile on a parrot that uses -O2 (not gcc’s -O flag. This one is about the parrot imcc). He’ll probably be improving parrot’s performance for Rakudo and NQP in the near future. He has also said before that parrot’s concurrency model is vastly superior to MoarVM’s. The MoarVM developers don’t agree on that, but I’d love to be surprised.

jnthn said he’ll focus on unblocking Rakudo * on MoarVM and the JVM after the next bits of concurrency work, so that we will hopefully have a triple-backend Rakudo * release this month. This includes NativeCall for MoarVM, which will thankfully be a lot simpler than it was for the JVM. MoarVM and parrot rely on the same library to do this, so there may be a lot of copypesto to be had.

I finally got the tip I needed for how to properly implement cascading block inlining for the NQP optimizer, so I’ll try to get that up and running this week. It ought to improve performance and memory usage on all of our backends.

Low hanging fruit

If you’d like to help out, here’s a little nugget to get you started:

Our evalbot on the IRC server currently outputs the versions of the backends used whenever someone asks it to eval some code. Since we’ve got three backends at the moment, these lines can become quite long:

   moritz | r: 1
 +camelia | rakudo-parrot 1aeb7c, rakudo-jvm 1aeb7c, rakudo-moar 1aeb7c: ( no output )

It would be nice to have it output

rakudo-{parrot,jvm,moar} 1aeb7c:

instead. Camelia is written in perl5 and the source code can be found on github. I’ve started a naive implementation of this in Perl 6, but only perl5 code will be accepted into the evalbot at this point. I’m sure there’s an algorithmically nicer implementation that can be done, though.

Something else that could be done soon-ish even if you don’t have a lot of experience yet is implementing the “locallifetime” op on MoarVM to allow temporary locals to be released by the register allocator earlier.

Something that’s always appreciated is helping out with ecosystem modules. You can try one or two out, report and perhaps golf bugs, weigh in on design decisions, … Or you can port your favourite perl5 (or even python/ruby/…) module over.

If any of those tasks seem interesting to you, feel free to visit us on the IRC. We are on the freenode network in #perl6. I hope you’ll have a wonderful week, my esteemed readers :)


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-03-10 17:07:31

Last week didn’t pack quite as many individual changes, but there are a couple really nice bugfixes in there. Let’s dig in!

  • Fewer methods on List will force eager evaluation of said list (some of them used List.end indirectly, which forced eagerness erroneously)
  • The combinations method of List will now also contain the empty List in the full Powerset.
  • The X and Z metaops (“Cross” – the cartesian product – and “Zip” respectively) will now correctly refrain from flattening itemized things, such as Arrays.
  • The reduction metaoperator now recognizes operators with list infix precedence like the X and Z metaops. This makes things like [Z+] work as expected on lists of lists.
  • Unary hyperops will now only distribute among the outermost level of nested list structures. Additionally, there are now “deepmap” and “duckmap” methods in addition to “flatmap”. “duckmap” first tries to apply the operator directly to any object in the list. If that fails, it’ll fall back to descending into substructures.
  • A whole bunch of spectest failures on Rakudo MoarVM that related to unicode properties have been fixed and we’ve now passed Rakudo Parrot in number of passing test cases. Rakudo Parrot is now at approximately 99.8%, Rakudo Moar is at 99.83% and Rakudo JVM is still the baseline of 100% against which we compare. As soon as the moar-conc branches get merged into their respective master branches, we expect the number of spectest passes to go up yet another bit.
  • On the topic of Rakudo MoarVM: the “moar-support” branch of panda has landed and you can now install modules for Rakudo MoarVM using panda. Unfortunately there is at least one known pre-compilation bug that prevents URI among others to be installed. You can install panda for MoarVM by running the “bootstrap.pl” or “rebootstrap.pl” scripts with perl6-m.
  • On the topic of module installation, FROGGS has been working hard on pushing the “eleven” branches forwards. Those aim to implement the much improved module installation, versioning and loading handling as specified in Synopsis S11.
  • I’ve built a fast path into the “string flattening” routine of MoarVM, that causes string concatenation, joining and repetition to become quite a bit faster. Ideally, the flattening will be gone completely soon, but the speed boost is nice to have until then. These operations still suffer from very bad asymptotical behavior.
  • Mouq and lue have been continuing their work on Pod6 and documentation.
  • Mouq also added shorthands @<foo> for @($<foo>) and %<foo> for %($<foo>) and @1 for @($1) as well as %1 for %($1). These are now useful, since $<foo> will – just like the sigil suggests – be itemized.
  • raydiak and smls have been spending time prettying up the style of the rakudo documentation website some more.

An interesting thing to point out is that the first 6 items on this list have been done by TimToady, who has left his self-imposed exile from the implementor’s side of Perl 6. Jnthn didn’t have much time during the week, so I have no big achievements to report for the multithreading support on MoarVM.


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-03-03 17:33:42

Even though the Rakudo compiler release 2014.02 was just last Thursday, there have already been a few interesting changes.

  • for-loops no longer eagerly gobble up all of the list you pass to it before starting to run. This makes for loops properly lazy in all regards.
  • in a set of branches called “moar-conc”, jnthn and JimmyZ and I have been abstracting away the JVM specific code from the concurrency classes and partially implemented the corresponding things on MoarVM. Threads, reentrant mutexes, and blocking queues are available already. ThreadPoolScheduler works, but is currently lacking its time-based scheduling functionality. The next step towards having “start” and “await” work is the Promise class itself.
  • Mouq and lue have been working on improving Pod6 support both in Rakudo’s parser and Pod::To::HTML. Among other things, lue started work on a POD to Pod6 converter that will at some point convert all the synopses to Pod6 – at least as an experiment.
  • Mouq has also contributed a bunch of documentation to doc.perl6.org, namely IO::FileTestable, IO::Handle, Mix, MixHash, Mixy, QuantHash, Set, SetHash, and Setty. moritz started the regexes reference document.
  • Coke has triaged a whole bunch of bugs in RT. There are now a bunch of tests tagged “testneeded”, which is a good starting point to get into Perl 6 development. We are aware that you are likely to receive a warning about a possible Cross-Site Request Forgery resulting from this Link. We are looking for a sensible solution to this, but as long as the warning is about “A malicious attacker may be trying to modify or access a search on your behalf.”, you can use the link “click here to resume your request” to get to the search results.

Raiph has pointed out, that there are also IRC summaries of #perl6 and #moarvm. Select a day from the list and click “summary” to find it. Usually the summary for a day is available by the next day. Another thing to look at for finding out what Perl 6ers are up to would be the Perl realm on questhub.io. There are also Perl5ers there, but you can somewhat easily tell which quests belong to Perl 6 and which do to Perl5.

What’s cooking?

I’ve been distracted over the Weekend, so I didn’t pick up too many hints on what our community members are up to right now, and the freenode netsplits haven’t been helping either.

One thing that I did hear is that jnthn wants to have a look at NativeCall on MoarVM soon.

Additionally, now that the JVM has NativeCall, we are going to be working on the Rakudo * infrastructure to support multiple backends properly. It’s likely, that Rakudo * 2014.03 will be available with Rakudo-JVM. It’s not quite as likely, but still possible that Rakudo * MoarVM will arrive in the same month.

Things you could do

Not only are there now a bunch of “testneeded” bugs in RT right now (see above), but a friend told me about JVM’s “SwitchPoint” functionality, which could be used to strongly hint to the JVM JIT that all the instanceof usages in Ops.java are very likely to succeed and very unlikely to fail. Investigating whether or not this will give a noticable performance improvement and then actually implementing it sanely across the whole Ops.java file would be a nice beginner’s project.

Another thing that’d make a decent beginner’s project would be to come up with some sane benchmarks to contribute to the perl6-bench project. It would be enough to implement the benchmarks in one of NQP or Perl 6, but if you want to port it to Perl5 as well, that would be nice. Making sure the richards benchmark works properly on nqp-moar and porting it to Perl 6 and/or Perl5 would be much appreciated.

Nobody has yet signed up as the release manager for this month’s Rakudo * release, so maybe you’d like to help? The whole process is described step-by-step with mostly copy-paste-able code snippets in that same document, and the people of #perl6 would be eager to answer any questions you come up with.

Something somewhat easier to do is to start a test suite for numeric formatting – things like floats and rats in combination with “say”, “print” and “printf”, as well as the .fmt method. Optionally, you can also fix the bugs in Rakudo, NQP, or MoarVM that you find. An approach that automatically generates either the test file or individual tests may be a good way to tackle this project.


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-02-24 14:16:56

On behalf of the Parrot team, I'm proud to announce Parrot 6.1.0, also
known as "Black-collared Lovebird". Parrot (http://parrot.org/) is a
virtual machine aimed at running all dynamic languages.

Parrot 6.1.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/6.1.0/), or by
following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 6.1.0 News:
- Build
+ Improve auto::libffi probe with -fstack-protector-all or
-fstack-protector-strong with recent GCC and OpenBSD's toolchains
- Documentation
+ Replace perldoc by a new podextract.pl [GH #1028, #973,
#520], which fixes problems with 'sudo make install' generated ops
pods as root.

Warnings:
- Latest nqp does not support the new packfile API yet,
replacing EvalPMC.
- This release of Parrot fails to build out-of-the-box under Bison 3,
e.g. on Fedora 20. For workarounds
see https://github.com/parrot/parrot/issues/1031

The SHA256 message digests for the downloadable tarballs are:
87d25119c73acdb26f89ac4c68d73f3d996451ada51f3cb2cd4878b6f0e0a34c
parrot-6.1.0.tar.gz
bb1294ad2a7d5b3c4688fc736fb775e94ecfe35fdc072a2631c2080eb5f366f7
parrot-6.1.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Esp. cPanel for the time and Erin Schoenhals for
sponsoring her old Macbook G4 powerpc to update the native_pbc's.
I've also updated all the missing old documentations on parrot.github.io
Our next scheduled release is 18 Mar 2014.

Enjoy!

--
Reini Urban
http://cpanel.net/ http://www.perl-compiler.org/

Perl 6 Announce | perl.perl6.announce | 2014-02-18 22:48:26

On behalf of the Parrot team, I'm proud to announce Parrot 6.1.0, also known
as "Black-collared Lovebird". Parrot (http://parrot.org/) is a
virtual machine aimed
at running all dynamic languages.

Parrot 6.1.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/6.1.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 6.1.0 News:
- Build
+ Improve auto::libffi probe with -fstack-protector-all or
-fstack-protector-strong with recent GCC and OpenBSD's toolchains
- Documentation
+ Replace perldoc by a new podextract.pl [GH #1028, #973,
#520], which fixes
problems with 'sudo make install' generated ops pods as root.

Warnings:
- Latest nqp does not support the new packfile API yet,
replacing EvalPMC.
- This release of Parrot fails to build out-of-the-box under Bison 3,
e.g. on Fedora 20. For workarounds see
https://github.com/parrot/parrot/issues/1031

The SHA256 message digests for the downloadable tarballs are:
87d25119c73acdb26f89ac4c68d73f3d996451ada51f3cb2cd4878b6f0e0a34c
parrot-6.1.0.tar.gz
bb1294ad2a7d5b3c4688fc736fb775e94ecfe35fdc072a2631c2080eb5f366f7
parrot-6.1.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Esp. cPanel for the time and Erin Schoenhals for
sponsoring her old Macbook G4 powerpc to update the native_pbc's.
I've also updated all the missing old documentations on parrot.github.io
Our next scheduled release is 18 Mar 2014.

Enjoy!

Perl 6 Announce | perl.perl6.announce | 2014-02-18 21:56:35

Good Monday to you!

As I predicted in last week’s post, this week’s tuits have mostly gone into making MoarVM “moar awesome”:

  • my untested varint serialization/deserialization code, which helped reduce the size for pretty much every integer that was serialized, turned out to be shoddy, so nwc10 made some tests and wrote the corresponding fixes, such that pre-compiled modules that have big-ish integer numbers in them don’t blow up in strange ways any more
  • in the same vein, we figured out that we were mistakenly using doubles for the nqp::radix operation that is in charge of parsing integers into integers. now it returns an array of integers, which doesn’t have any precision problems any more. We have JimmyZ to thank for that
  • jnthn has started and finished the I/O refactor, which also quickly led to client sockets, then server sockets. Panda can now get its project list off the ‘net on MoarVM
  • In addition, there was an STable Reposession related bug that was fixed yesterday and now panda is almost ready! Just some test failures or something left ;)
  • I’ve merged in my experiments with python plugins to gdb, resulting in a pretty-printer for MVMString while debugging as well as a nursery and gen2 analysis command that will show what Reprs appear how often  and things like that. It also shows how fragmented the pages of the gen2 are. Using that, I found a way to reduce the startup memory usage by about 2 megabytes. Hopefully I’ll be able to push that a lot further in the future!
  • small boxed integers between 0 and 15 (inclusive) are now cached, since they were overwhelmingly common in the nursery and gen2.

In other news, raydiak and smls have put in some work to make doc.perl6.org prettier. It’ll get even prettier in the future! And there’s a website for moarvm now, too. We mostly needed that to prevent people from accidentally downloading the tarballs from github, which unfortunately don’t contain some of the 3rdparty code.

Arnsholt has almost finished NativeCall for the JVM, the only thing missing now is the nqp::nativecallrefresh op which is needed to force a re-read of C-level memory to flush out some stale caches sometimes.

Spectest wise, moar has gained 0.1% on the jvm implementation, but that’s the only notable change.

What’s cooking this week?

I just turned my attention to syntax highlighting, IDE support, tooling etc. Turns out the description language Kate uses is pretty decent and is supported by QtCreator. QtCreator itself is apparently a thoroughly plugin-driven IDE. Since I’m somewhat comfortable with C++ and Qt, I may be spending some time building support for Perl6 into QtCreator. The syntax highlighting part, however, is now hoelzro’s job, as he’s already experienced :)

I’ve got a little bit of work on the JVM interop code-gen that’s not polished enough to be merged yet. In addition to that, I’m trying to build an optimization for NQP that would help inline some blocks, reduce code size and shorten object lifetimes hopefully.

Unfortunately I haven’t paid enough attention to my fellow devs this week, so I don’t know what other cool projects are in the pipeline — I blame being sick this week — but I’ve come up with a different section to make up for the gap:

How can I get involved?

I’m meaning to have a section like this often-ish. Maybe not every week, but hopefully at least once a month. Here I’d like to highlight some “low hanging fruit” that potentially anyone could pick up and get a feel for Perl 6 Development:

  • If you know some C, you could have a look at some of the outstanding inefficiencies in MoarVM:
    • Strings are currently forced to use 32bits per codepoint even if they could fit into 8bits per codepoint. The reason for that is the hash implementation we use. It doesn’t have a way to differentiate between 32bit/codepoint strings and 8bit/codepoint strings, so the same actual string can have two hashes, which messes up very early during compilation of NQP.
    • Looking at the strings found in the nursery, the empty string comes up surprisingly often. Maybe a boxed empty string could be cached somewhere. Also, maybe P6str needs to learn how to share MVMString?
    • Since I’m already talking about strings so much, if you’re interested in that topic, you can have a look at the implementation of “ropes”, that is “a string made up of a tree of string parts”. There’s a possibly wrong implementation of rope pretty-printing in moar-gdb.py, which you could test & improve.
    • There’s an op called “nqp::locallifetime” that causes registers that hold locals to be freed early if the code-gen already knows they won’t be needed any more. MoarVM currently just no-ops. Could be relatively easy to implement, since the “freeing up locals” code already exists for leaving Blocks and Stmt nodes.
  • We don’t have terribly much stuff that uses sockets yet. I’d almost call them untested (except for the spectests). So if you’re interested in some protocol, why not whip up an implementation, see how well the sockets hold up? They now work on all three of our backends.
  • If you’re interested in learning how the code-gen works in nqp, you can implement loop labels on JVM and Moar, or take the parrot implementation of loop labels on NQP and port it over to rakudo. You can find FROGGS on the irc and ask him questions
  • the list of GSoC ideas for this year

I hope I could give you a nice overview of what happened in the last 7 days. Soon I’ll do my first Rakudo Compiler Release, so maybe the weekly changes article will be skipped (you could just read the changelog of the release) or it would be a bit shorter than usual and only point out the things that were done between thursday and sunday.

Have an appropriate amount of fun! :)


Weekly changes in and around Perl 6 | Weekly changes in and around Perl 6 | 2014-02-17 16:23:14

Announce: Rakudo Star Release 2014.01

A useful, usable, “early adopter” distribution of Perl 6

On behalf of the Rakudo and Perl 6 development teams, I’m happy to
announce the January 2014 release of “Rakudo Star”, a useful and usable
distribution of Perl 6. The tarball for the January 2014 release is
available from http://rakudo.org/downloads/star/. A Windows .MSI
version of Rakudo star is available in the downloads area as well.

In the Perl 6 world, we make a distinction between the language
(“Perl 6″) and specific implementations of the language such as
“Rakudo Perl”. This Star release includes release 2014.01 of the
Rakudo Perl 6 compiler, version 5.9.0 of the Parrot Virtual
Machine
, plus various modules, documentation, and other resources
collected from the Perl 6 community.

https://github.com/rakudo/rakudo/blob/nom/docs/announce/2014.01.md

Some of the new features added to this release include:

  • The eval sub and method are now spelled EVAL
  • Numeric.narrow to coerce to narrowest type possible
  • Can now supply blocks with multiple arguments as sequence endpoints
  • Method calls and hash/list access on Nil give Nil

This release also contains a range of bug fixes, improvements to error
reporting and better failure modes.

Please note that this release of Rakudo Star does not support the JVM
nor the MoarVM backends from the Rakudo compiler. While the other backends
mostly implement the same features as the Parrot backend, many bits are
still missing, most prominently the native call interface.
We hope to provide a JVM-based and MoarVM-based Rakudo Star releases soon.

There are some key features of Perl 6 that Rakudo Star does not yet
handle appropriately, although they will appear in upcoming releases.
Some of the not-quite-there features include:

  • advanced macros
  • threads and concurrency (in work for the JVM backend)
  • Unicode strings at levels other than codepoints
  • interactive readline that understands Unicode
  • non-blocking I/O
  • much of Synopsis 9 and 11

There is an online resource at http://perl6.org/compilers/features
that lists the known implemented and missing features of Rakudo and
other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the
programmer that a given feature isn’t implemented, but there are many
that we’ve missed. Bug reports about missing and broken features are
welcomed at rakudobug@perl.org.

See http://perl6.org/ for links to much more information about
Perl 6, including documentation, example code, tutorials, reference
materials, specification documents, and other supporting resources. A
draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in
the release tarball.

The development team thanks all of the contributors and sponsors for
making Rakudo Star possible. If you would like to contribute, see
http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org
mailing list, or join us on IRC #perl6 on freenode.

rakudo.org | rakudo.org | 2014-01-31 16:58:10

This month’s Rakudo compiler was cut today, and there’s a bunch of good stuff in there. In this post I’ll take a quick look at what’s been done.

MoarVM Support

This is the first Rakudo compiler release to have support for building and running on MoarVM, a new VM being built especially for Perl 6 and NQP (the Perl 6 subset a sizable chunk of Rakudo is written in). MoarVM support is not quite up to the level of the JVM and Parrot backends yet. It passes less specification tests than either of them – though it’s getting awfully close (Rakudo on MoarVM passes over 99% of the specification tests that Rakudo on the JVM – the current leader – does). Thus, you can actually run a heck of a lot of Perl 6 code just fine on it already. I used it recently in a pair programming session and we only hit one bug in the couple of hours we were using it.

The fast-path for signature binding that I mentioned in my previous post has also been put in place. It did, as hoped, lead to a fairly dramatic speedup. The workload of building Rakudo’s built-ins and running the specification test suite was also a good basis for doing some GC tuning, which led to further improvements. By this point, on my box, Rakudo on MoarVM now has:

  • The lowest startup time of any Rakudo backend
  • The shortest spectest time of any Rakudo backend
  • For the CORE.setting build and spectests, the smallest memory footprint of any Rakudo backend

Other Rakudo developers have reported similar findings. I need more time to look into the exact numbers, but it would appear that Rakudo on MoarVM is also the fastest to build. CORE.setting build time is roughly competitive with on the JVM now (but how roughly seems to vary quite widely – I think it depends on what JVM or even version is being used), but startup time for NQP on MoarVM is rather lower, meaning that those parts of the build go by faster.

The focus for the next month or two will be getting into a position where we can produce a Rakudo Star release that uses MoarVM. This means digging through the last 1% of failing spectests and dealing with them, finishing the work of getting Panda (our module installer) to work with Rakudo on MoarVM, and then hunting bugs that keep us from running the modules. Getting NativeCall working will also be a notable task, although given we already have a NativeCall in C working against 6model (the one we built for Parrot), there is a lot of prior art this time – unlike on the JVM.

On performance – we’re not even scratching the surface of what’s possible. MoarVM’s design means it has a lot of information to hand to do a good amount of runtime specialization and optimization, but none of this is implemented yet. I aim to have a first cut of it in place within the next few months. Once we have this analysis and specialization framework in place, we can start thinking about things such as JIT compilation.

Rakudo on JVM Improvements

Probably the best news in this release for anybody working with Rakudo on JVM is that the gather/take stack overflow bug is now fixed. It was a fun one involving continuations and a lack of tailcall semantics in an important place, but with doing the MoarVM implementation of continuations in my recent past, I was in a good place to hunt it down and get a fix in. A few other pesky issues are resolved, including a regex/closure interaction issue and sometimes sub-optimal line number reporting.

The other really big piece of JVM-specific progress this month has been arnsholt++ continuing to work on the plumbing to get us towards full NativeCall support for JVM. This month, a number of the big missing pieces landed. NativeCall working, and the modules that depend on it working, is the last big blocker for a Rakudo Star on JVM release, and it’s now looking quite possible that we’ll see that happen in the February one.

General Rakudo Improvements

While a lot of energy went on the things already mentioned, we did get some nice things in place that are independent of any of the particular backend: improvements to the Nil type, the sequence operator, sets and bags, adverb syntax parsing, regex syntax errors, aliased captures in regexes, and numerics. MoarVM’s rather stricter interpretation of closure semantics than we’ve had in place on other backends has also led to various code-gen fixes, which may lead to better performance in certain scenarios across the board too (one of those, “I know it probably should but I didn’t benchmark” situations).

I’d like to take a moment to thank everyone who contributed to this month’s release. This month had the highest Rakudo contributor count in a good while – and I’m hopeful we can maintain and exceed it in the months to come.


Jonathan Worthington | 6guts | 2014-01-24 01:55:47

May your pleasures be many, your troubles be few.
-- Cast of "Hee Haw"

On behalf of the Parrot team, I'm proud to announce Parrot 6.0.0, also known
as "Red-necked Amazon". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 6.0.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/supported/6.0.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 6.0.0 News:
- Core
- Build
- Documentation
+ Fixed bad IPv6 examples in pdd22_io, thanks to Zefram++ [GH#1005]
- Tests
+ Fixed failure in t/configure/062-sha1.t.
+ Updated to Unicode 6.3 (libicu5.2): U+180e Mongolian Vowel Separator
is no whitespace anymore [GH #1015]
- Community


The SHA256 message digests for the downloadable tarballs are:
e150d4c5a3f12ae9d300f019bf03cca58d8e8051dd0b934222b4e4c91160cd54 parrot-6.0.0.tar.gz
6cb9223ee389a36588acf76ad8ac85e2224544468617412b1d7902e5eb8bd39b parrot-6.0.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 18 Feb 2014.

Enjoy!

Perl 6 Announce | perl.perl6.announce | 2014-01-22 17:38:28

Almost exactly a month ago, I reported that Rakudo on MoarVM could do “Hello, world”. Despite the general slow-down as folks took Christmas and New Year breaks (myself very much included), progress has been very good. Here are the key things to know:

  • All of the Rakudo sanity tests pass on Rakudo on MoarVM
  • At present, Rakudo on MoarVM passes over 92% of the specification tests that the current leader (Rakudo on the JVM) does
  • We’re on course to have the January compiler release be the first Rakudo release with some level of MoarVM support; I’m hopeful we’ll be passing over 95% of the specification tests Rakudo on the JVM does by that point

The work has been taking place in the moar-support branch. We didn’t merge it yet, but that should happen within the next week or so.

MoarVM itself has been improving steadily, also. Here are a few of the features that have landed in the last weeks:

  • Block exit handlers, used to implement LEAVE and friends
  • Continuations, used to make gather/take properly lazy
  • Updated the Unicode database to Unicode 6.3
  • Sized native arrays
  • State variables

However, another extremely important bit of work has been taking place that is focused on stability. Nicholas Clark has been conducting garbage collection torture tests, by now down to forcing a GC run every single object allocation and using memory protection to catch illegal accesses to moved objects. Most of the things we call GC bugs are not actually inside the garbage collector implementation, but rather are other places in the codebase where mistakes have been made that break invariants that must be upheld for the GC to do its job correctly. I’ve not been counting, but I’d estimate that a bit over a dozen bugs have been turned up by this work so far – bugs that would have been a real pain to find if they had just happened to crop up some weeks, months or years down the line in some user’s code. At this point, NQP can be built and tested under the toughest torture test that exists so far without showing any issues. The Rakudo build is the current torture subject. I’m incredibly happy this work is taking place; it means that by the time Rakudo on MoarVM starts getting used more widely, we can be reasonably confident that users are unlikely to run into GC-related issues.

So, what’s the path from here? Here’s what I’m hoping for in the coming month:

  • We reach the point of passing more than 99% of the spectests that Rakudo on the JVM does
  • Progress towards Panda being able to run and install some modules (not sure we’ll get all the way there, but I hope we’ll get reasonably close)
  • Dealing the with very-slow-path signature binding. If you try anything on Rakudo on MoarVM, you’ll discover it’s slow. This is thanks to a hot path being implemented in a slow way, in the name of getting stuff correct before optimizing. As soon as we’re beyond the 99% point with the spectests, I’ll dig into this. It should make a dramatic difference. I’m aiming to do some work in this area to make things better for Rakudo on JVM also.
  • Whatever fixes are needed to get the Rakudo build and sanity tests surviving the GC torture

I’ll be back with some updates in a couple of weeks, to let you know how we’re getting on. :-)


Jonathan Worthington | 6guts | 2014-01-08 22:50:56

We need a solution that makes us need less vodka. — jnthn

There are many extremely simple and elegant software solutions out there. But there are also those special moments, when you realize that something is more complex than you thought, and that the complexity is most likely essential.

Character encodings are the prototypical example for me. Certainly datetime handling qualifies as well.

Reaching the realization that there is that extra essential complexity, comes (at least for me) with a sinking feeling as I get used to the idea of living with that complexity forever.

With me so far? Something seemed quite easy, wrapped up, ready to go home for the day, but then all this extra complexity rears its head. And it's never going away.

I started writing this blog post because I realized that a certain snag in role handling in Rakudo doesn't have a URL, and it really should. So, without fanfare, here's the situation:

my $x;

role R {
    method foo {
        say $x;
    }
}

class C does R {
}

$x = "OH HAI";
C.new.foo;

I think we all agree that this should print OH HAI. Good? Good. Nothing up my sleeve, no hidden mirrors or escape hatches — it does print OH HAI. Relax. Take a deep breath.

Ready? Because after you learn this, there's no going back. The world will forever be more complicated and, with some luck, you'll be having that sinking feeling.

Ok, so. Just a few simple facts:

  • Blocks have lexpads. Think of a lexpad as "all the variables declared in that block, along with the values they contain." I count four blocks in the code above. From smallest to biggest, they are: the method inside the role, the class block, the role block, and finally the entire code which gets a "mainline" block even though that block doesn't have any curly braces.

  • Blocks are linked through OUTER links. A small block nested inside a larger block has the larger block as its OUTER. More precisely, it's the lexpads that are linked. I think the literature refers to these as "parent" blocks, but in Perl 6 we taboo that word and use OUTER for lexical lookup (and CALLER for dynamic lookup).

  • There's often more than one lexpad per block. This one is hard to swallow. But think of a recursive factorial function: sub fac($N) { $N ?? fac($N - 1) * $N !! 1 }. As this function calls itself, it's going to need a fresh lexpad with each call. (Or it'll clobber the distinct $N values. Let's call these lexpads runtime lexpads.

  • A block always has a static lexpad, and then one or more runtime lexpads. Consider this code: class C { method foo { my $x = 42; method bar { say $x } } }; C.bar;. In Rakudo, it prints (Any), not 42 as you might think. Why not 42? Because C.foo has never run. In fact, the (Any) value of $x is coming from C.foo's static lexpad, because that's the only lexpad C.foo has. (Interesting historical note: it took us a while to get this right in Rakudo. Used to be you could make variable lookups that didn't reach the static lexpad, but instead caused a Null PMC Access or similar. Ah, the pain.)

  • Roles are created at compile time. This one shouldn't come at a surprise. But we need it for the pressing agony up ahead.

  • Classes are composed at compile time. Yep, same. Unless, you know, you're doing high-level MOP-ery. Which we're not in this code.

Let's recap what we know by applying it to the code. There's the variable $x. We know we will find it in the static lexpad of the mainline, because it's declared on the top level and everything has a static lexpad. Does it also have a runtime lexpad? Yes, it does, because the mainline starts running after compilation is over. Will we find $x in several runtime lexpads? No, only the one.

Now, we ask ourselves the question: which lexpad is C.foo referring to?

"Of course, it's the runtime lexpad", we reply, innocent to the fact that the trap has already shut around us and there's no way out. See, it has to be the runtime lexpad, because the sane thing for the program to do is to print OH HAI, and that value is certainly stored in the runtime lexpad.

But no. It's not possible. It can't. There's no way. Because roles are created at compile time, before there is a runtime lexpad! The role method has no choice: it's bound to the static lexpad, because at that point, that's all there is.

And there we are. The trap has now closed. There's no way to both (a) do what the user expects, and (b) keep the internal model nice and free of weird exceptions.

Since we like (a), we ditch (b) and create an exception in Rakudo. It's called a fixup, it's installed during role creation, and it makes sure that whenever the block surrounding the role is entered, the role rebinds its OUTER to that block's fresh lexpad.

Simple it ain't. Nor is it pretty. But it makes the user happy.

The reason I started thinking about this is that we run into the same kind of problem with macros, and the same kind of fixup will probably be needed there.

More to the point, at the point where this need-for-a-fixup starts showing up in different parts of the architecture, it's time to give it a name and perhaps think of a uniform way to address this. That's where jnthn's quote from the start of the post originates — we need a solution that isn't worse than the problem, and that we can reason about without having to scale the Ballmer peak.

Carl Masak | Strangely Consistent | 2014-01-04 16:35:06

(這是 Day 24 – Advent Ventures 的中譯,作者是 Larry Wall。)


人類歷險百萬年到了今夕,
平安夜裡且讓時間暫停,
我們充滿疑問的冒險
再次等待答案的降臨。

我們到了嗎?

粘菌奮鬥十億年到了矽晶,
幸好(大致上)已忘卻不幸的往昔
以及無情天擇留下的血痕,
我們再次銘記未來的回憶。

我們到了嗎?

這個月是 Perl 降臨以來的第 26 年,
(包括 13 歲的天才小妹)
我們的小家庭一次用 24 個故事,
慶祝回家路上的種種勝利。

我們到了嗎?爹地,我們到那兒了嗎?

我們跟隨先聖,雖然徘徊但從未迷失:
我們跟隨亞伯拉罕,尋找尚未出現的城市;
我們跟隨神行客,守護將重新分配未來的哈比人
我們跟隨法師和巫士、學者和探險家、聖人和科學家。

我在星空下漫步尋思……我是否已經迷失……

無論智者或哈比人,我們總在黑暗裡啟程,
摸索前行,迎向新鮮空氣、一絲希望,
經過怪獸和深淵,追尋封印的星光,
找到那片樂土,讓疲憊者得以休息,讓悲傷者得以療養。

等等,你們什麼意思,我不能去?——甘道夫

我們所有人都必須在沙漠裡徘徊四十年,
埋葬酸民和鹼民的屍骨,
讓他們的孩子有一天能跨過約旦河
進入滿是牛奶、蜂蜜和漂亮手機的新天地!

等等,你們什麼意思,我不能去?——摩西

我們把古老傳說帶進未來,
在雜沓的路途上吟誦,
背包裡裝滿史詩,口袋裡塞滿故事,
把自己的軼事傳聞留在身後。

有些好故事你還沒聽過呢。

所以孩子們,除了古老的傳說,也要收拾新的工具,
輕巧而強大的工具能幫助你們,也能幫助你們來幫助我們。
最輕巧的工具、最強大的工具,就是思想,
一定要多收拾一些。我會在這裡等。

我在這兒。就是說我還沒到那兒。快!

挑選一些好朋友,也讓一些好朋友挑選你。
輪流耐心等待、急速奔走,
懷著希望前進、或絕望地爬行,
再次不屈不撓地站起來。或稀裡糊塗地,這也行。

有時你是故事的主角,但並非隨時都是。

相信你的旅程,為你帶來新的同伴;
相信新的同伴,為你帶來旅程的所需;
準備好面對永遠的邂逅,與暫時的告別。
(沒人能為暫時的邂逅、永遠的告別做好準備。)

我還不確定我是否想抵達那裡。

感謝旅途賜給你今日的夥伴,人生本就是苦樂參半。
欣賞悲欣交集的歌曲、酸甜交織的啤酒。
享受戰鬥的痛楚、擁抱的甘美。
對了,還有,享受因此得到的瘀傷,但別傷得太重。

為你還沒有抵達那裡而慶幸。

歡迎我的朋友們來這裡,我們還沒有到那裡。
歡迎來到這個家族,量子疊加出無數歡樂、悲傷和憧憬。
歡迎和我們一起不斷努力,盜取更多普羅米修斯之火。
這火燒得真快,卻永遠填不滿生命的篝火坑。

我們更暖和了嗎?

他們說:給人一把火……先等一下,有個即時新聞……
這真是胡扯:天火現在自由開源了⁈ 好吧,呃……
部落客聽到雲端天使彈出視窗、放聲歡唱?
嗯……最好看看新聞小幫手……稍等一下……等等等等……

連線品質真是混帳……快收到了……

嘿,你知道嗎⁈ 物理學家琢磨出來了。
整個宇宙剛剛順利編譯完成……
現在,他們正在找人對這玩意兒進行偵錯;
嘿,我知道,我只要用 Perl 6 測試套件就行了。

【……現在你有成千上萬的問題……】

你說什麼?

(大聲)健全測試 #1 合格了嗎?結果如何?

前路漫漫而修遠,
穿越河流與森林,
你走陽關道,我走獨木橋,
我們都在通往應許樂土的路上。

TimToady 得到賜福,開始指揮 Perl 朝聖者的合唱。】

我們向錫安山進發,
美麗動人的錫安山
我們要登上錫安山,美啊——

你不能去那裡。

等等,你們什麼意思,我不能去?

錯誤修正 #1:殺掉所有的蹩腳詩人。

噢……胡扯……

Audrey Tang | Pugs | 2013-12-28 15:23:24

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the December 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the December 2013 release is available from http://rakudo.org/downloads/star/. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.12 of the Rakudo Perl 6 compiler, version 5.9.0 of the Parrot Virtual Machine, plus various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

  • The Whatever Star (*) now works inside chain operators like comparisons.
  • Private attributes from roles are now visible in the classes they apply to.
  • Memory and speed improvements for ListIter and List.combinations, respectively.
  • Improvements to the execution of regexes.

This release also contains a range of bug fixes, improvements to error
reporting and better failure modes.

Please note that this release of Rakudo Star does not support the JVM
backend from the Rakudo compiler. While the JVM backend mostly implements
the same features as the Parrot backend, many bits are still missing,
most prominently the native call interface.
We hope to provide a JVM-based Rakudo Star release soon.

The following notable features have been deprecated or modified from previous
releases due to changes in the Perl 6 specification, and are planned to be
removed or changed as follows:

  • All unary hyper ops currently descend into nested arrays and hashes. In the future, those operators and methods that are defined “nodal” will behave like a one-level map.

There are some key features of Perl 6 that Rakudo Star does not yet
handle appropriately, although they will appear in upcoming releases.
Some of the not-quite-there features include:

  • advanced macros
  • threads and concurrency (in work for the JVM backend)
  • Unicode strings at levels other than codepoints
  • interactive readline that understands Unicode
  • non-blocking I/O
  • much of Synopsis 9 and 11

There is an online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo and
other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at rakudobug@perl.org.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

rakudo.org | rakudo.org | 2013-12-25 18:49:45

Here at the end of megayears of human adventure,
we schedule a silent night to stop time,
once again awaiting the next advent
of the answer to our questionable venture.

    Are we there yet?

After a gigayear or so of slogging from slime to silicon
mercifully forgetting (most of) the unmerciful past,
and the scars left by unnatural nature upon our pedigree,
we now remember to remember the future once more.

    Are we there yet?

So in this month, 26 year-moments after the advent of Perl,
(including 13 year-eternities of precocious brat sisterhood)
our little family celebrates, 24 tales at a time,
its victories in the struggle to find our way home.

    Are we there yet, Daddy, are we there yet?

We follow after all those who wander but are not lost:
We follow Abraham, looking for a city because it isn't there yet;
We follow Strider, guarding the hobbits who will redistribute the future;
We follow Magi and wizard, scholar and explorer, saint and scientist.

    I wonder as I wander out under the sky...whether I'm lost yet...

But wise man or hobbit, we must all take that journey in the dark,
groping ahead for the path to better air and a little hope,
following the encapsulated starlight past monsters and chasms
out to a land where the weary can rest, and be healed of grief.

    "Wait, what do you mean, I can't go there?" —Gandalf

We must all wander in this desert for forty years,
burying the bones of naysayers and yaysayers alike,
so that their children can someday cross the Jordan
into a land flowing with milk and honey and fancy new phones!

    "Wait, what do you mean, I can't go there?" —Moses

We carry these old stories to the future,
cadences to chant over the confusion of the road,
backpacks full of epics, pockets stuffed with tales,
leaving our own litter of anecdotal evidence behind us.

    You haven't heard some of the good ones yet.

So kids, along with the old stories, pack a few new tools,
light but powerful tools that will help you and help you help us.
The lightest tools, the most powerful tools, are ideas,
so pack lots and lots of 'em. I'll wait here while you do.

    I'm here yet. Which means I'm not there yet. Hurry up!

Pick some good friends, and let some good friends pick you.
Take turns waiting patiently, running impatiently,
or walking hopefully, crawling hopelessly,
standing up yet again defiantly. Or woozily, that works too.

    Be the protagonist some of the time, yet not all of the time.

Trust your journey to provide you with new companions;
trust your new companions to provide you with your journey.
Be prepared to say your eternal hellos and temporary goodbyes.
(No one's ever ready for the temporary hellos and eternal goodbyes.)

    And I'm not sure I want to be there quite yet.

Enjoy the companions your journey gives today, for life is bittersweet.
Enjoy the bittersweet songs and the bittersweet beer.
Enjoy the bitter fights and sweet hugs.
And, yes, enjoy the resulting bruises, but not too much.

    Enjoy knowing that you're not there yet.

Welcome, my friends, to the here, and to the not-there-yet.
Welcome to the clan's quantum superposition of joy and grief and longing.
Welcome to our ongoing effort to steal more of that Promethean fire
that burns too fast yet never fast enough to fit the firepits of our lives.

    Are we getting warmer yet?

As they say, "Give a man a fire..." Hold that thought, some breaking news...
This just in: Fire from heaven is now free and open-sourced?! Well, huh...
Seems a blogger heard some angels singing popup advent adverts in the cloud?
Hmm...better do some fact checking...hang in there...tum tiddly tum...

    Darn flakey connection...almost there...

Well, hey, whaddya know?! The physicists figured it out.
The whole universe has just finished compiling without error...
Now they're looking for someone to debug the silly thing;
Hey, I know, I'll just use the Perl 6 test suite.

    [...now you have thousands of problems...]

Did you say something?

    (louder) Does sanity test #1 pass yet? What's the output?

The road goes ever on and on,
Over the river and through the woods,
You take the high road, and I'll take the low road,
We're all bound for the Promised Land.

[TimToady gets blessed and starts directing the choir of Perl Pilgrims.]
We're marching to Zion,
Beautiful, beautiful Zion,
We're marching upward to Zion, that beautif—

    You can't go there.

Wait, what do you mean, I can't go there?

    Bugfix #1: kill all the bad poets.

Aw...crap...

Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-24 00:00:13

Most languages or libraries that provide a generic sort routine allow you to specify a comparator, that is a callback that tells the sort routine how two given elements compare. Perl is no exception.

For example in Perl 5, which defaults to lexicographic ordering, you can request numeric sorting like this:

 use v5;
 my @sorted = sort { $a <=> $b } @values;

Perl 6 offers a similar option:

 use v6;
 my @sorted = sort { $^a <=> $^b }, @values;

The main difference is that the arguments are not passed through the global variables $a and $b, but rather as arguments to the comparator. The comparator can be anything callable, that is a named or anonymous sub or a block. The { $^a <=> $^b} syntax is not special to sort, I have just used placeholder variables to show the similarity with Perl 5. Other ways to write the same thing are:

 my @sorted = sort -> $a, $b { $a <=> $b }, @values;
 my @sorted = sort * <=> *, @values;
 my @sorted = sort &infix:«<=>», @values;

The first one is just another syntax for writing blocks, * <=> * use * to automatically curry an argument, and the final one directly refers to the routine that implements the <=> "space ship" operator (which does numeric comparison).

But Perl strives not only to make hard things possible, but also to make simple things easy. Which is why Perl 6 offers more convenience. Looking at sorting code, one can often find that the comparator duplicates code. Here are two common examples:

 # sort words by a sort order defined in a hash:
 my %rank = a => 5, b => 2, c => 10, d => 3;
 say sort { %rank{$^a} <=> %rank{$^b} }, 'a'..'d';
 #          ^^^^^^^^^^     ^^^^^^^^^^  code duplication

 # sort case-insensitively
 say sort { $^a.lc cmp $^b.lc }, @words;
 #          ^^^^^^     ^^^^^^  code duplication

Since we love convenience and hate code duplication, Perl 6 offers a shorter solution:

 # sort words by a sort order defined in a hash:
 say sort { %rank{$_} }, 'a'..'d';

 # sort case-insensitively
 say sort { .lc }, @words;

sort is smart enough to recognize that the code object code now only takes a single argument, and now uses it to map each element of the input list to new values, which it then sorts with normal cmp sort semantics. But it returns the original list in the new order, not the transformed elements. This is similar to the Schwartzian Transform, but very convenient since it's built in.

So the code block now acts as a transformer, not a comparator.

Note that in Perl 6, cmp is smart enough to compare strings with string semantics and numbers with number semantics, so producing numbers in the transformation code generally does what you want. This implies that if you want to sort numerically, you can do that by forcing the elements into numeric context:

 my @sorted-numerically = sort +*, @list;

And if you want to sort in reverse numeric order, simply use -* instead.

The unary sort is very convenient, so you might wonder why the Perl 5 folks haven't adopted it yet. The answer is that since the sort routine needs to find out whether the callback takes one or two arguments, it relies on subroutine (or block) signatures, something not (yet?) present in Perl 5. Moreover the "smart" cmp operator, which compares number numerically and strings lexicographically, requires a type system which Perl 5 doesn't have.

I strongly encourage you to try it out. But be warned: Once you get used to it, you'll miss it whenever you work in a language or with a library that lacks this feature.


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-23 00:00:18

Perl 6 has a very healthy relationship to operators. "An operator is just a funnily-named subroutine", goes the slogan in the community.

In practice, it means you can do this:

sub postfix:<!>($N) {
    [*] 2..$N;
}
say 6!;    # 720

Yeah, that’s the prototypical factorial example. I promise I won’t do it any more in this post. I swear the only reason we don’t have factorial as a standard operator in the language, is so that we can impress people by defining it.

Anyway, so a postfix operator ! is really just a "funnily-named" subroutine postfix:<!>. Similarly, we have prefix and infix operators, like prefix:<-> and infix:<*>. They’re all just subroutines. I wrote about that before, so I’m not going to hammer on that point.

Operators have different precedence (like infix:<*> binds tighter than infix:<+>)…

$x + $y * $z   # compiler sees $x + ($y * $z)
$x * $y + $z   # compiler sees ($x * $y) + $z

…and different associativity (like infix:</> associates to the left but infix:<=> associates to the right).

$x / $y / $z   # compiler sees ($x / $y) / $z
$x = $y = $z   # compiler sees $x = ($y = $z)

But I wrote about that before too, at quite some length, so I’m not going to revisit that topic.

No, today I just want to talk about the operator categories in general. I think Perl 6 does a nice job of describing the operator types themselves. I don’t see that in many other languages.

Here are all the different types:

 type            position           syntax
======          ==========         ========

prefix          before a term        !X
infix           between two terms    X ! Y
postfix         after a term         X!

circumfix       around               [X]
postcircumfix   after & around       X[Y]

A lot of other languages give you the ability to define your own operators. Many of them provide approaches which are hackish afterthoughts, and only allow you to override existing operators. Some other languages do approach the problem head-on, but end up simplifying the language to decrease the complexity of defining new operators.

Perl 6 approaches, and solves, the problem, head-on. You get all of the above operators, you can refine or override old ones, and you can talk within the language about things like precedence and associativity.

All that is rather nice. But, as usual, Perl 6 goes one step further and starts classifying metaoperators, too.

What’s a metaoperator? That’s our name for when you can extend an operator in a certain way. For example, many languages allow some kind of not-equal operator like infix:<!=>. Perl 6 has another one for string non-equality: infix:<ne>. And then maybe you want to do smart non-matching: infix:<!~~>. And so on — pretty soon you catch on to the pattern: you may, at one point or another, want to invert the result of any boolean matcher. So that’s what Perl 6 provides.

Here are a few examples:

normal op     negated op
=========     ==========
eq            !eq     ( synonym of ne )
~~            !~~
<             !<      ( synonym of >= )
before        !before

Because this particular metaoperator attaches itself before an infix operator, it gets the name infix_prefix_meta_operator:<!>. I was going to say "and yes, you can add your own user-defined metaoperators too", but currently no implementation supports that. Maybe sometime in the future.

There are many other categories of metaoperators. For example the "multiplication reducer" [*] that we used in the factorial example at the top (which takes a list of numbers and multiplies them all together) is really an infix:<*> surrounded by the metaop prefix_circumfix_meta_operator:sym<[ ]>. (It’s "prefix" because it goes before the list of numbers.)

Luckily, we don’t have meta-meta-ops. Already thinking about the metaops is quite a challenge sometimes. But what I like about Perl 6 and the approach it takes to grammar and operators is this: someone did think about it, long and hard. The resulting system has a pleasing simplicity to it.

Perl 6 is very well-suited for building parsers for languages. As one of its neatest tricks, it takes this expressive power and directs it towards the task of defining itself. As a Perl 6 user, you’re given not just an excellent toolbox, but a well-organized workshop to adapt and extend to fit your needs.


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-22 00:02:04

In today’s post, we’ll go through the basics of Perl 6′s subroutine signatures, which allow us to declare our parameters instead of coding them as we do in Perl 5.

In Perl 5, arguments to a sub are accessible via the @_ array, so if you want to access an argument to a sub, you can either access the array directly (which is typically only done where speed is a concern)…

use v5;
sub greet {
    print "Hello, " . @_[0] . "\n";
}

Or, more commonly, pull off any arguments as lexicals:

use v5;
sub greet {
    my $name = shift @_;
    print "Hello, $name\n";
}

Positionals

Perl 6 lets you declare required positional arguments for your subs as part of the signature:

use v6;
sub greet($name) {
    print "Hello, $name\n";
}

Inside the sub, $name is available as a readonly alias to the passed in variable.

By default, parameters are required. If you try to call this function with no parameters as greet, you’ll get a compile time error:

===SORRY!===
CHECK FAILED:
Calling 'greet' requires arguments (line 1)
Expected: :($name)

You can make the parameter optional by adding a ? to the signature. Then you’ll have to examine the parameter to see if a defined value was passed in. Or, you can declare a default value to be used if none is passed in:

use v6;
sub guess($who, $what?) {
    # manually check to see if $what is defined
}

sub dance($who, $dance = "Salsa") {
    ...
}
dance("Rachael")
dance("Rachael", "Watusi")

Another way to handle optional arguments is to use multis. Multis are subs that can have multiple implementations that differ by their signature. When invoking a multi, the argument list used in the invocation is used to determine which version of the multi to dispatch to.

use v6;
multi sub dance($who, $dance) {
    say "$who is doing the $dance";
}
multi sub dance($who) {
    dance($who, "Salsa");
}

Types

The parameters in the previous section allow any type of value to be passed in. You can declare that only certain types are valid:

sub greet(Str $name) {
    say "hello $name";
}

Now hello("joe") will work as it did before. But hello(3) will generate a compile time error:

===SORRY!===
CHECK FAILED:
Calling 'greet' will never work with argument types (int) (line 1)
Expected: :(Str $name)

In addition to any of Perl 6 builtin types or user-built classes, you can also add programmatic checks inline in the signature with where clauses.

use v6;
multi odd-or-even(Int $i where * %% 2) { say "even" };
multi odd-or-even(Int $i) { say "odd"};

Note that we didn’t have to add a where clause to the second sub. The multi-dispatch algorithm will prefer the more detailed signature when it matches, but fallback to the less descriptive version when it doesn’t.

You can even use literal values as arguments. This style allows you to move some of your logic to the multi dispatcher.

use v6;
multi fib(1) { 1 }
multi fib(2) { 1 }
multi fib(Int $i) { fib($i-1) + fib($i-2) }

say fib(10);

Note that this isn’t very efficient… yet. Once the is cached trait for subs is implemented, we can use that to provide a builtin analog to Perl 5′s Memoize module.

Named

Perl 6 provides for named parameters; if the positionals are analogous to an array of parameters, the named arguments are like a hash. You use a preceding colon in the argument declaration, and then pass in a pair for each parameter when invoking the sub.

use v6;
sub doctor(:$number, :$prop) {
    say "Doctor # $number liked to play with his $prop";
}
doctor(:prop("cricket bat"), :number<5>);
doctor(:number<4>, :prop<scarf>);

Note that the order the parameters are passed in doesn’t matter for named parameters.

If you have a variable of the same name in the calling scope, you can simplify the calling syntax to avoid creating an explicit pair.

use v6;
my $prop = "fez";
my $number = 11;
doctor(:$prop, :$number)

You can also use different names internally (using the expanded pair syntax) for the named arguments. This allows you to use different (simplified or sanitized) versions for the caller.

use v6;
sub doctor(:number($incarnation), :prop($accoutrement)) {
    say "Doctor # $incarnation liked to play with his $accoutrement";
}
my $number = 2;
my $prop = "recorder";
doctor(:$number, :$prop)

Slurpy

To support functions like sprintf, we need to be able to take a variable number of arguments. We call these args “slurpy”, since they “slurp” up the arguments.

# from rakudo's src/core/Cool.pm
sub sprintf(Cool $format, *@args) {
    ...
}

When invoking this sub, the first argument must be of type Cool (kind of a utility supertype), and all the remaining positional arguments are slurped up into the @args variable. The preceding * indicates the slurp.

Symmetrically, we can also slurp up named arguments into a hash.

# from rakudo's src/core/control.pm
my &callwith := -> *@pos, *%named {
    ...
}

This snippet introduces to the pointy block syntax for anonymous subs. The -> begins the declaration, followed by the signature (without parentheses), and finally the sub definition in the block. The signature here shows all the positionals ending up in *@pos, and all the named arguments in %named.

This also shows that positionals and named arguments can be combined in the same sub.

Methods

Method and sub declarations are virtually identical. All the parameters mentioned so far are usable in methods.

The main difference is that methods can be passed an invocant (the object associated with the method call), and when you define the sub, you have the opportunity to name it. It must be the first parameter if present, and is marked with a trailing :.

use v6;
method explode ($self: $method) {...}

Note that there is no comma separating these the invocant and the first positional. In this context, the colon functions as a comma.

Parameter Traits

Each parameter can additionally specify a trait that changes the behavior of that parameter:

  • is readonly - this is the default behavior for a parameter; the sub cannot modify the passed in value.
  • is rw - the argument can be modified. Forces the argument to be required.
  • is copy - the sub gets a modifiable copy of the original value.

More Information

Check out Synopsis #6 for more information, and the roast test suite for more examples – any of the directories starting with S06-.


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-21 00:03:22

The past year saw the (at least partial) implementation of several variable traits. But before we get into that, the past year also saw Nil getting implemented closer to spec in Rakudo. If you read the spec, you will know that Nil indicates the absence of a value. This is different from being undefined, as we saw in an earlier blogpost this year. So what does that mean?

$ perl6 -e 'my $a = 42; say $a; $a = Nil; say $a'
42
(Any)

So, assigning Nil will reset a scalar container to its default value, which (by default) is the type object of that container. Which is (Any) if nothing is specifically specified.

Enter the “is default” variable trait

If you want to specify the default value of a variable, you can now use the “is default” variable trait:

$ perl6 -e 'my $a is default(42); say $a; say $a.defined’
42
True

So oddly enough, even though we haven’t assigned anything to the variable, it now has a defined value. So let’s assign something else to it, and then assign Nil:

$ perl6 -e 'my $a is default(42) = 69; say $a; $a = Nil; say $a'
69
42

Of course, this is a contrived example.

“is default” on arrays and hashes

It gets more interesting with arrays and hashes. Suppose you want a Bool array where you can only switch “off” elements by assigning False? That is now possible:

$ perl6 -e 'my Bool @b is default(True); say @b[1000]; @b[1000]=False; say @b[1000]'
True
False

Of course, type checks should be in place when specifying default values.

$ perl6 -e 'my Bool $a is default(42)'
===SORRY!=== Error while compiling -e
Type check failed in assignment to '$a'; expected 'Bool' but got 'Int’

Note that this type check is occurring at compile time. Ideally, this should also happen with arrays and hashes, but that doesn’t happen just yet. Patches welcome!

Using is default on arrays and hashes interacts as expected with :exists (well, at least for some definition of expected :-):

$ perl6 -e 'my @a is default(42); say @a[0]; say @a[0].defined; say @a[0]:exists'
42
True
False

Note that even though each element in the array appears to have a defined value, it does not necessarily exist. Personally, I would be in favour of expanding that to scalar values as well, but for now :exists does not work on scalar values, just on slices.

It’s not the same as specifying the type

What’s wrong with using type objects as default values? Well, for one it doesn’t set the type of the variable. Underneath it is still an (Any) in this case:

$ perl6 -e 'my $a is default(Int) = "foo"; say $a'
foo

So you don’t get any type checking, which is probably not what you want (otherwise you wouldn’t take all the trouble of specifying the default). So don’t do that! Compare this with:

$ perl6 -e 'my Int $a = "foo"'
Type check failed in assignment to '$a'; expected 'Int' but got 'Str'

Which properly tells you that you’re doing something wrong.

Nil assigns the default value

Coming back to Nil: we already saw that assigning it will assign the default value. If we actually specify a default value, this becomes more clear:

$ perl6 -e 'my $a is default(42) = 69; say $a; $a = Nil; say $a'
69
42

In the context of arrays and hashes, assigning Nil has a subtly different effect from using the :delete adverb. After assigning Nil, the element still exists:

$ perl6 -e 'my @a is default(42) = 69; say @a[0]; @a[0] = Nil; say @a[0]:exists'
69
True

Compare this with using the :delete adverb:

$ perl6 -e 'my @a is default(42) = 69; @a[0]:delete; say @a[0]; say @a[0]:exists'
42
False

One could argue that assigning Nil (which assigns the default value, but which is also specced as indicating an absence of value) should be the same as deleting. I would be in favour of that.

Argh, I want to pass on Nil

The result of a failed match, now also returns Nil (one of the other changes in the spec that were implemented in Rakudo in the past year). However, saving the result of a failed match in a variable, will assign the default value, losing its Nilness, Nilility, Nililism (so much opportunity for creative wordsmithing :-).

$ perl6 -e 'my $a = Nil; say $a'
(Any)

It turns out you can specify Nil as the default for a variable, and then It Just Works™

$ perl6 -e 'my $a is default(Nil) = 42; say $a; $a = Nil; say $a'
42
Nil

Introspection

You can use the VAR macro to introspect variables to find out its default value, without having to do something as destructive as assigning Nil to it:

$ perl6 -e 'my @a is default(42); say @a.VAR.default'
42

This also works on system variables like $/:

$ perl6 -e 'say $/.VAR.default'
Nil

And can also be used to interrogate other properties:

$ perl6 -e 'say $/.VAR.dynamic’
True

But this gets us into material for a blog post for another day!

Conclusion

The “is default” variable trait allows you to manipulate the default value of variables and elements in arrays and hashes. Together with the new meaning of Nil, it is a powerful means of mangling initial values of variables to your satisfaction.


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-20 00:00:26

Several days back, we took a look at promises and channels. Promises provided a synchronization mechanisms for asynchronous things that produced a single result, while channels were ideal for setting up producer/consumer style workflows, with producers and consumers able to work in parallel. Today we’ll take a look at a third mechanism aimed at introducing asynchrony and coping with concurrency: supplies!

Synchronous = Pull, Asynchronous = Push

One of the most important features of promises is the then method. It enables one or more things to be run whenever the asynchronous work the promise represents is completed. In a sense, it’s like a publish/subscribe mechanism: you call then to subscribe, and when the work is done then the notification is published. This happens asynchronously. You don’t sit around waiting for the promise to complete, but instead just say what to do when this takes place. Thinking about the way data flows, values are pushed along to the next step of the process.

This is rather different to things like sub and method calls. There, we make the call, then we block until it has completed. Iterating over a lazy list is the same: you can’t progress with the iteration until the next value is available. Thus, iteration is really about pulling stuff from a source. So, if a promise can be thought of as something that can push a single values out as it becomes available, do we have something that can push a whole stream of values outwards, as they are produced over time?

Supplies! We do!

A while back, it was realized that the observer pattern is the mathematical dual of the iterator pattern. Why is this exciting? Quite simply, because it means that all the things you can sensibly do with something you can iterate (map, grep, zip, etc.), you can also sensibly do with something you can observe. Out of this was born reactive programming, and the Rx (Reactive Extensions) library, which has now been ported to many platforms. In Perl 6, we’re providing support for this in core.

The basics

Let’s start out simple. First, we create a new Supply:

my $measurements = Supply.new;

We can then tap the supply, passing a closure that should be called whenever a value is made available:

$measurements.tap(-> $value {
    say "Measured: $value";
});

Finally, we produce some values:

$measurements.more(1.5);
$measurements.more(2.3);
$measurements.more(4.6);

On each of these calls, the closure we tapped the supply with is invoked. Just as we can call then many times, so we can tap many times too:

$measurements.tap(-> $value {
    say "Also measured: $value";
});

Now, when we produce a value:

$measurements.more(2.8);

Both of the closures tapping the supply will be called. Note that tap returns an object which can be used to express you’re no longer interested in the supply, essentially turning that tap off.

Note that we didn’t introduce any asynchrony so far. However, supplies are built for it. You can safely have multiple threads supplying values. By default, the supplying thread is used to execute the taps.

Enter the combinators!

Since supplies are essentially a thread-safe observer implementation, we can define many of the same things on them as we’re used to having on lists. For example, imagine we just wanted to tap high measurements. Well, we just re-use knowledge from how we’d filter a list: using grep!

$measurements.grep(* > 4).tap(-> $value {
    say "HIGH: $value";
});

Calling grep on a supply produces another supply, just as calling grep on a list gives another list. We could, if we wished, store it in a variable and tap this derived supply many times, grep it again, map it, etc.

Supply factories

There are ways to get supplies besides simply creating them directly. The Supply class has various factory methods that create various interesting kinds of supply, while introducing asynchrony. For example, interval gives a supply that, when tapped, will produce an ascending integer once per time interval.

my $secs = Supply.interval(1);
$secs.tap(-> $s { say "Started $s seconds ago" });
sleep 10;

Factories can also help map between paradigms. The Supply.for method produces a supply that, when tapped, will iterate the specified (potentially lazy) list and push the values out to the tapper. It does the iteration asynchronously. While it’s not implemented yet, we’ll be able to define a similar mechanism for taking a Channel and tapping each value that is received.

Crossing the streams

Some of the most powerful – and tricky to implement – combinators are those that involve multiple supplies. For example, merge gives a single supply whose values are those of the two other supplies it tapped, and zip pairs together values from two different supplies. These are tricky to implement because it’s entirely possible that two different threads will be supplying values. Thankfully, though, we just need to manage this once inside of the supplies implementation, and save those using them from worrying about the problem! In a sense, combinators on lists factor out flow control, while combinators on supplies factor out both flow control and synchronization. Both let us program in a more declarative style, getting the imperative clutter out of our code.

Let’s bring all of this together with an example from one of my recent presentations. We simulate a situation where we have two sets of readings coming in: first, measurements from a belt, arriving in batches of 100, which we need to calculate the mean of, and second another simpler value arriving once every 5 seconds. We want to label them, and get a single supply merging these two streams of readings together. Here’s how it can be done:

my $belt_raw = Supply.interval(1).map({ rand xx 100 });
my $belt_avg = $belt_raw.map(sub (@values) {
    ([+] @values) / @values
});
my $belt_labeled = $belt_avg.map({ "Belt: $_" });
my $samples = Supply.interval(5).map({ rand });
my $samples_labeled = $samples.map({ "Sample: $_" });
my $merged = $belt_labeled.merge($samples_labeled);
$merged.tap(&say);
sleep 20;

Notice how it’s not actually that much harder than code that maps and greps lists we already had – and yet we’re managing to deal with both time and concurrently arriving data.

The future

Supplies are one of the most recently implemented things in Rakudo, and what’s done so far works on Rakudo on the JVM. With time, we’ll flesh out the range of combinators and factories, keep growing our test coverage, and deliver this functionality on Rakudo on MoarVM too, for those who don’t want to use the JVM.


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-19 11:26:39

Today’s example constructs a grammar for tracking playing cards in a single deal. We’ll say it’s poker with one or more players and that each player is being dealt a hand that contains exactly five cards.

There is, however, the need to detect duplicate cards. We’ll need some way of tracking cards both within each card-hand and between hands.

A simple Card Game Grammar

To start with, here’s the basic grammar (no duplicate checks yet):

grammar CardGame {

    rule TOP { ^ <deal> $ }

    rule deal {
        <hand>+ % ';'
    }

    rule hand { [ <card> ]**5 }
    token card {<face><suit>}

    proto token suit {*}
    token suit:sym<♥>  {<sym>}
    token suit:sym<♦>  {<sym>}
    token suit:sym<♣>  {<sym>}
    token suit:sym<♠>  {<sym>}

    token face {:i <[2..9]> | 10 | j | q | k | a }
}

say CardGame.parse("2♥ 5♥ 7♦ 8♣ 9♠");
say CardGame.parse("2♥ a♥ 7♦ 8♣ j♥");

The  top-level rule consists of a deal. The deal consists of one or more hands separated by ';'. Each hand consists of 5 playing cards.

Each card is represented by a face, one of: a (ace), j (jack), q (queen) or k (king), or 2 - 10. This is followed by a suite: ♥ (hearts) ♦ (diamonds) ♣ (clubs) or ♠ (spades).

[We could have used the playing cards characters, newly introduced in Unicode 6.0, but these aren't widely supported yet].

As expected, the first cut of the grammar cheerily parses any hand:

say CardGame.parse("a♥ a♥ 7♦ 8♣ j♥");
# one hand, duplicate a♥
say CardGame.parse("a♥ 7♥ 7♦ 8♣ j♥; 10♥ j♥ q♥ k♥ a♥");
# two hands, duplicate j♥

Detecting Duplicates

We start by adding a Perl 6 variable declaration to the grammar. This will be used to track cards:

rule deal {
    :my %*PLAYED = ();
    <hand>+ % ';'
}

This declares %*PLAYED [1]. The '%*' twigil  indicates that it’s a hash '%' and that’s dynamically scoped '*'.

Dynamic scoping is not only for subroutine and method calls [1]. It also works seamlessly with grammar rules, tokens and actions.

Being dynamically scoped, %*PLAYED is available to callees of the deal rule; the hand token, and its callee, the card token.

It’s also available to any actions, that then get called. So we can track and report on duplicates by creating an action class with a method for the card token:

class CardGame::Actions {
    method card($/) {
       my $card = $/.lc;
       say "Hey, there's an extra $card"
           if %*PLAYED{$card}++;
   }
}

my $a = CardGame::Actions.new;
say CardGame.parse("a♥ a♥ 7♦ 8♣ j♥", :actions($a));
# "Hey there's an extra a♥"
say CardGame.parse("a♥ 7♥ 7♦ 8♣ j♥; 10♥ j♥ q♥ k♥ a♦",
                   :actions($a));
# "Hey there's an extra j♥"

And that might be all that’s needed  for tracking and reporting on duplicates. There’s a pretty good separation between the declarative grammar and procedural actions, with just one dynamically scoped hash variable.

Disallowing Duplicates

But I had a situation where I wanted duplicate checking to be a parsing constraint. Parsing needed to fail when duplicates were encountered.

I achieved this by moving the duplicate check grammar side:

token card {<face><suit>
    <?{
        # only allow each card to appear once
        my $card = $/.lc;
        say "Hey, there's an extra $card"
            if %*PLAYED{$card};

        ! %*PLAYED{$card}++;
     }>
}

This has introduced a code assertion between the <?{ and }>  [2]. The rule succeeds when the code evaluates to a True value. The card token thus fails when the same card is detected more than once in a single deal.

say CardGame.parse("2♥ 7♥ 2♦ 3♣ 3♦");
# legitimate, parses

say CardGame.parse("a♥ a♥ 7♦ 8♣ j♥");
# fails with message: Hey, there's an extra a♥

say CardGame.parse("a♥ 7♥ 7♦ 8♣ j♥; 10♥ j♥ q♥ k♥ a♦");
# fails with message: Hey, there's an extra j♥

[This last technique works fully on rakudo/parrot today and should be available on the jvm in the near future.]

Discussion/Conclusion

One thing to be careful of with this type of technique is back-tracking (trying of alternatives). If, for instance, the grammar was modified in such a way that the card token could be called more than once for single input card, then we might erroneously report a duplicate. It’s still possible to track, but becomes a bit more involved. The simplest answer is to keep the grammars as simple as possible and minimize back-tracking.

If in any doubt,  please consider using one or more of the Grammar::Debugger, Grammar::Tracer [3] or the debugger [4] modules [5] to track what’s going on. You can also insert debugging code into tokens or rules as closures: { say "here" } [6].

That the exercise for today; a simple Perl 6 Grammar to parse playing-cards in a card-game, but with duplicate checks using either actions or code assertions.


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-18 00:00:53

If you’re anything like me, you’ve read last year’s advent calendar posts with delight, squeeing a little bit at the neat things you could express so easily with Perl 6. You might have wondered – like I have – if all those sundry features are just shiny bells and whistles that were added on top of an okay language. And you might be wondering how to find out more, dig deeper, and maybe even contribute.

As you can tell from the fact that I’m writing this article, I’ve decided to become a contributor to the Perl 6 effort. I’m going to tell you how I got interested and then involved (and inloved) with Perl 6.

Before my involvement with Perl 6 I mostly used Python. However beautiful and flexible Python is, I worked on a project where it was a poor fit and writing proper test code was exceptionally uncomfortable. Thus, I often made mistakes – misused my data structures, passed values of incorrect types to methods – that I felt the language should be able to detect early without sacrificing flexibility. The “gradual typing” approach of Perl 6 sounded like a very good fit to me.

Having a friend show me bits and pieces of Perl 6 quickly led to looking at the Advent Calendar last year. I also joined the IRC channel and asked a whole bunch of questions. Not having done any Perl 5 programming before made figuring out the nooks and crannies of Perl 6 syntax a bit harder than I would have liked, especially when using the Perl 6 book. Fortunately, the IRC channel was always active and eager to help.

After having learnt a bit more about Perl 6, I quickly started helping out here and there. In part because I’ve already enjoyed doing that for PyPy – which is implemented in a subset of Python, much like Rakudo is implemented in NQP – but also because I kept hitting little problems and bugs.

So, how do you actually get to contributing?

Well, starting small is always good. Come to the IRC channel and ask for “low hanging fruit”. Check out bugs blocking on test coverage or easy tickets from the Perl 6 bug tracker. Try porting or creating one of the most wanted modules. Experiment with the language and look what bugs you hit. One thing that’s always good and often not very hard is fixing “LTA error messages”; those are error messages that are “less than awesome”.

At some point you’ll find something you’d like to fix. For me, the first thing was making spectests for bugs that were already fixed, but didn’t have a test yet. After that, giving a more helpful error message when a programmer tries to concatenate strings with . instead of ~. Then came fixing the output of nested Pair objects to have proper parenthesis. And then I dug deep into the Grammar, Actions and World classes to implement a suggestion mechanism for typos in variables, classes and subs.

The fact that most of your potential contributions will likely be done either in Perl 6 code or at least in NQP code makes it rather easy to get started, since even NQP is fairly high-level. And if you work through the materials from the Rakudo and NQP Internals Workshop 2013, you’ll get a decent head start in understanding how Rakudo and NQP work.

Whenever you get stuck, the people in #perl6 – including me, of course – will be happy to help you out and give advice along the way.

Let’s try to tackle a simple “testneeded” bug. this bug about parsing :$<a> pair shorthand syntax looks simple enough to write a test case for.

Here’s the executive summary:

The :$foo syntax has already been introduced a few days ago by the post on adverbly adverbs. The syntax here is a combination of the shorthand notation $<foo> to mean $/<foo>, which refers to the named match group “foo” from the regex we just matched.

So :$<foo> is supposed to give the pair "foo" => $<foo>. It didn’t in the past, but someone came along and fixed it. Now all we need to do is create a simple test case to make sure Rakudo doesn’t regress and other implementations don’t make the same mistake.

In the last comment of the discussion, jnthn already wrote a much shortened test case for this bug, hidden behind a “show quoted text” link:

'abc' ~~ /a $<a>=[\w+]/; say :$<a>.perl

Since sometimes it happens that such a test case causes a compilation error on one of the implementations, it’s useful to be able to “fudge” the whole thing in one go. That means writing something like #?niecza skip in front of a block. That’s why we wrap our whole little test in curly braces, like so:

# RT #76998
{
   my $res = do { 'abc' ~~ /a $<a>=[\w+]/; :$<a> };
   ok $res ~~ Pair, ':$<a> returns a pair';
   ok $res.key eq 'a', 'its key is "a"';
   ok $res.value ~~ Match:D, 'the pair's value is a defined match object';
}

So what have we done here? We executed the code in a do block, so that we can grab the value of the last piece of the block. The next three lines inspect the return value of our code block: Is it a Pair object? Is its key “a”, like expected? Is its value a defined Match object instance? (See day 2′s post about the :D smiley).

Also, we wrapped the whole test in a big block and put a comment on top to mention the ticket ID from the bug tracker. That way, it’s easier to see why this test exists and we can reference the commit to the perl6/roast repository in the ticket discussion when we close it (or have someone close it for us).

Next up, we have to “bump up the plan”, which means we go to the beginning of the test file and increase the line that calls “plan” with a number by 3 (which is exactly how many more ok/not ok outputs we expect our test case to generate).

Then we can either fork perl6/roast on github and make a pull-request from our change or submit a git format-patch created patch via IRC, the bug tracker, or the mailing lists.

Note that I didn’t commit anything I wrote here yet. If you’re faster than the other readers of this blog, you can feel free to go ahead and submit this. If you’re not, feel free to select a different ticket to improve upon.

Finally, if you apply for a commit bit – meaning the right to directly commit to the repositories on github – make sure to mention I sent you to get 50% off of your first 10 commits; what an incredible deal!


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-17 00:12:35

use v6;
my $thing = "123abc";
say try $thing + 1; # this will fail

{
    use v5;
    say $thing + 1 # will print 124
}

Slangs are pretty interesting things in natural languages, so naturally they will be pretty awesome in computer languages as well. Without it the cross-language communication is like talking through a thin pipe, like it is when calling C functions. It does work, but calling functions is not the only nor the most comfortable thing out there.

The example above shows that we create a variable in Perl 6 land and use it in a nested block, which derives from another language. (This does only work if the nested language is capable of handling the syntax, a dollar-sigilled variable in this case.)
We use this other language on purpose: it provides a feature that we need to solve our task.
I hope that slangs will pop up not just to provide functionality to solve a given problem, but also help in writing the code in a way that fits the nature of that said problem.

How does that even work?

The key is that the module that lets you switch to the slang provides a grammar and appropriate action methods. That is not different from how Perl 6 is implemented itself, or how JSON::Tiny works internally.
The grammar will parse all statements in our nested block, and the actions are there to translate the parsed source code (text) into something a compiler can handle better: usually abstracted operations in form of a tree, called AST.

The v5 slang compiles to QAST, which is the name of the AST that Rakudo uses. The benefit of that approach is that this data structure is already known by the guts of the Rakudo compiler. So our slang would just need to care about translating the foreign source code text into something known. The compiler takes this AST then and maps it to something the underlying backend understands.
So it does not matter if we’re running on Parrot, on the JVM or something else, the slang’s job is done when it produced the AST.

A slang was born.

In March this year at the GPW2013, I felt the need for something that glues both Perl 6 and Perl 5 together. There were many nice people that shared this urge, so I investigated how to do so.
Then I found a Perl 5 parser in the std repository. Larry Wall took the Perl 6 parser years ago and modified it to be Perl 5 conform. The Perl 6 parser it is based on is the very same that Rakudo is built upon. So the first step was to take this Perl 5 grammar, take the action methods from Rakudo, and try to build something that compiles.
(In theory this is all we needed: grammar + action = slang.)

I can’t quite remember whether it took one week or two, but then there was a hacked Rakudo that could say “Hallo World”. And it already insisted on putting parens around conditions for example. Which might be the first eye catcher for everyone when looking at both languages.
Since then there was a lot of progress in merging in Perl 5′s test suite, and implementing and fixing things, and making it a module rather than a hacked standalone Rakudo-ish thing.

Today I can proudly say that it passes more than 4600 of roughly 45000 tests. These 4600 passing tests are enough so you can play with it and feed it simple Perl 5 code. But the main work for the next weeks and months is to provide the core modules so that you can actually use a module from CPAN. Which, after all, was the main reason to create v5.

What is supported at the moment?

  • all control structures like loops and conditions
  • functions like shift, pop, chop, ord, sleep, require, …
  • mathematical operations
  • subroutine signatures that affect parsing
  • pragmas like vars, warnings, strict
  • core modules like Config, Cwd and English

The main missing pieces that hurt are:

Loop labels for next LABEL, redo LABEL and last LABEL will land soon in rakudo and v5. The other missing parts will take their time but will happen :o).

The set goals of v5:

  • write Perl 5 code directly in Perl 6 code, usually as a closure
  • allow Perl 6 lexical blocks inside Perl 5 ones
  • make it easy to use variables declared in an outer block (outer means the other language here)
  • provide the behaviour of Perl 5 operators and built-ins for v5 blocks only, nested Perl 6 blocks should not be affected
  • and of course: make subs, packages, regexes, etc available to the other language

All of the statements above are already true today. If you do a numeric operation it will behave differently in a v5 block than a Perl 6 block like the example at the top shows. That is simply because in Perl 6 the + operator will dispatch to a subroutine called &infix:<+>, but in a v5 block it translates to &infix:<P5+>.

Oversimplified it looks a bit like this:

Perl 6/5 code:

1 + 2;
{
    use v5;
    3 + 4
}

Produced AST:

- QAST::CompUnit
    - QAST::Block 1 + 2; { use v5; 3 + 4 }
        - QAST::Stmts 1 + 2; { use v5; 3 + 4 }
            - QAST::Stmt
                - QAST::Op(call &infix:<+>) +
                    - QAST::IVal(1)
                    - QAST::IVal(2)
            - QAST::Block { use v5; 3 + 4 }
                - QAST::Stmts  use v5; 3 + 4 
                    - QAST::Stmt
                        - QAST::Op(call &infix:<P5+>)
                            - QAST::IVal(3)
                            - QAST::IVal(4)

The nice thing about this is that you can use foreign operators (of the used slang) in your Perl 6 code. Like &prefix:<P5+>("123hurz") would be valid Perl 6 code that turn a string into a number even when there are trailing word characters.

To get v5 you should follow its README, but be warned, at the moment this involves recompiling Rakudo.

Conclusion: When was the last time you’ve seen a language you could extend that easily? Right. I was merely astonished how easy it is to get started. Next on your TODO list: the COBOL slang. :o)


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-16 08:57:22

Consider the humble integer.

my $number-of-dwarfs = 7;
say $number-of-dwarfs.WHAT;   # (Int)

Integers are great. (Well, many of them are.) Computers basically run on integers. Also, God made the integers, and everything else is basically cheap counterfeit knock-offs, smuggled into the Platonic realm when no-one’s looking. If they’re so important, we’d expect to have lots of good ways of writing them in any respectable programming language.

Do we have lots of ways of writing integers in Perl 6? We do. (Does that make Perl 6 respectable? No, because it’s a necessary but not sufficient condition. Duh.)

One thing we might want to do, for example, is write our numbers in other number bases. The need for this crops up now and then, for example if you’re stranded on Binarium where everyone has a total of two fingers:

my $this-is-what-we-humans-call-ten = 0b1010;   # 'b' for 'binary', baby
my $and-your-ten-is-our-two = 0b10;

Or you may find yourself on the multi-faceted world of Hexa X, whose inhabitants (due to an evolutionary arms race involving the act of tickling) sport 16 fingers each:

my $open-your-mouth-and-say-ten = 0xA;    # 'x' is for, um, 'heXadecimal'
my $really-sixteen = 0x10;

If you’re really unlucky, you will find yourself stuck in a file permissions factory, with no other means to signal the outer world than by using base 8:

my $halp-i'm-stuck-in-this-weird-unix-factory = 0o644;

(Fans of other C-based languages may notice the deviation from tradition here: for your own sanity, we no longer write the above number as 0644 — doing so rewards you with a stern (turn-offable) warning. Maybe octal numbers were once important enough to merit a prefix of just 0, but they ain’t no more.)

Of course, just these bases are not enough. Sometimes you will need to count with a number of fingers that’s not any of 2, 8, or 16. For those special occasions, we have another nice syntax for you:

say :3<120>;       # 15 (== 1 * 3**2 + 2 * 3**1 + 0 * 3**0)
say :28<aaaargh>;  # 4997394433
say :5($n);        # let me parse that in base 5 for you

Yes, that’s the dear old pair syntax, here hijacked for base conversions from a certain base. You will recognize this special syntax by the fact that it uses colonpairs (:3<120>), not the "fat arrow" (3 => 120), and that the key is an integer. For the curious, the integer goes up to 36 (at which point we’ve reached ‘z’ in the alphabet, and there’s no natural way to make up more symbols for "digits").

I once used :4($_) to translate DNA into proteins. Perl 6 — fulfilling your bioinformatics dreams!

Oh, and sometimes you want to convert into a certain base, essentially taking a good-old-integer and making it understandable for a being with a certain amount of fingers. We’ve got you covered there, too!

say 0xCAFEBABE;            # 3405691582
say 3405691582.base(16);   # CAFEBABE

Moving on. Perl 6 also has rationals.

my $tenth = 1/10;
say $tenth.WHAT;   # (Rat)

Usually computers (and programming languages) are pretty bad at representing numbers such as a tenth, because most of them are stranded on the planet Binarium. Not so Perl 6; it stores your rational numbers exactly, most of the time.

say (0, 1/10 ... 1);            # 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
say (1/3 + 1/6) * 2;            # 1
say 1/10 + 1/10 + 1/10 == 0.3;  # True (\o/)

The rule here is, if you give some number as a decimal (0.5) or as a ratio (1/2), it will be represented as a Rat. After that, Perl 6 does its darnedest to strike a balance between representing numbers without losing precision, and not losing too much performance in tight loops.

But sometimes you do reach for those lossy, precision-dropping, efficient numbers:

my $pi = 3.14e0;
my $earth-mass = 5.97e24;  # kg
say $earth-mass.WHAT;      # (Num)

You get floating-point numbers by writing things in the scientific notation (3.14e0, where the e0 means "times ten to the power of 0"). You can also get these numbers by doing some lossy calculation, such as exponentiation.

Do keep in mind that these are not exact, and will get you into trouble if you treat them as exact:

say (0, 1e-1 ... 1);             # misses the 1, goes on forever
say (0, 1e-1 ... * >= 1);        # 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 (oops)
say 1e0/10 + 1/10 + 1/10 == 0.3; # False (awww!)

So, don’t do that. Don’t treat them as exact, that is.

You’re supposed to be able to write lists of numbers really neatly with the good old Perl 6 quote-words syntax:

my @numbers = <1 2.0 3e0>;
say .WHAT for @numbers;          # (IntStr) (RatStr) (NumStr)

But this, unfortunately, is not implemented in Rakudo yet. (It is implemented in Niecza.)

If you’re into Mandelbrot fractals or Electrical Engineering, Perl 6 has complex numbers for you, too:

say i * i == -1;            # True
say e ** (i * pi) + 1;      # 0+1.22464679914735e-16i

That last one is really close to 0, but the exponentiation throws us into the imprecise realm of floating-point numbers, and we lose a tiny bit of precision. (But what’s a tenth of a quadrillionth between friends?)

my $googol = EVAL( "1" ~ "0" x 100 );
say $googol;                            # 1000…0 (exactly 100 zeros)
say $googol.WHAT;                       # (Int)

Finally, if you have a taste for the infinite, Perl 6 allows handing, passing, and storage of Inf as a constant, and it compares predictably against numbers:

say Inf;              # Inf
say Inf.WHAT;         # (Num)
say Inf > $googol;    # True
my $max = -Inf;
$max max= 5;          # (max= means "store only if greater")
say $max;             # 5

Perl 6 meets your future requirements by giving you alien number bases, integers and rationals with excellent precision, floating point and complex numbers, and infinities. This is what has been added — now go forth and multiply.


Perl 6 Advent Calendar | Perl 6 Advent Calendar | 2013-12-15 10:56:36

In Perl 6 every block acts as a try exception if there is a CATCH block in it. Inside the CATCH block one can use a default block to catch all the exceptions.

Perl 6 Maven | Perl 6 Maven | 2013-12-08 21:52:02

Another release of Rakudo Star. Let's see how can we get start using it on Mac OSX.

Perl 6 Maven | Perl 6 Maven | 2013-12-07 11:52:02

I’ve been rather snowed under with work in November, thus the absence of posts here. Sorry about that. It turns out the sales folks at $dayjob did their job a little too well, and besides my own heavy teaching load I had to step in to rescue a few other things. Anyway, that’s done now, and I’m back to having a bit more time. (It’s not that I had absolutely no time at all since the last post. It was just more important to spend the bits I had clearing blockers in the way of others rather than writing blog posts. :-)) So, some quick bits of news.

Concurrency

At the Austrian Perl Workshop, I worked with lizmat on getting the Perl 6 concurrency spec aligned with the things I’d been working on for Rakudo on JVM. Happily, this had some good consequences. Firstly, Larry did a load of work on it, vastly improving naming of things and giving some syntactic relief in various areas where I’d just done simple function and method APIs. Secondly, both lizmat and timotimo dug in to bring the implementation in line with these spec changes, doing some other improvements to boot. So, now the bus number on the Rakudo concurrency support has increased. You can find the latest spec here. Also, you can see my Nordic Perl Workshop slides about it.

Rakudo on MoarVM progress

Last time I wrote here, we had NQP support for MoarVM pretty much complete, and were ready to dig into working on getting Rakudo running on MoarVM. The first step was to get the core of the compiler itself building. This wasn’t a terrible amount of work, since the majority of it is just NQP code. There’s more of it than is found in the NQP compiler, but that aside it doesn’t do too much that’s new. Next came the Perl 6 MOP, which is written in NQP. Things went very smoothly with that. Beyond there, things got more interesting.

The next big piece to make work was the BOOTSTRAP. This uses the MOP to start piecing together the key types at the heart of Perl 6, doing enough so we can write the rest of the built-ins in Perl 6 itself. Most of it is one huge BEGIN block. Here there were various unimplemented things, plus some of the VM-specific bits of Rakudo needed porting. And beyond that lay…the setting. When I did the JVM port, we had around 14,000 lines of built-ins there. These days it’s closer to 17,000 – and that’s excluding the concurrency stuff. Since compiling the setting actually involves running little bits of Perl 6 code, thanks to traits and BEGIN blocks, getting through it means making quite a lot of things work. We got there a week or so back.

That still didn’t give us “Hello, world”, however. The setting does various bits of initialization work as it loads, which naturally hit more things that didn’t yet work. Finally, yesterday, we reached the point where setting loading worked and we could do “Hello, world”. Actually, once the setting loaded, this worked right off.

Of course, there’s still lots to do from here. The next step will be to work on the sanity tests. There’s a couple of large and complex features that will need some porting work; of note, gather/take will need doing. Also there are a couple of stability and algorithmic things that need taking care of in MoarVM itself. Building CORE.setting is easily the biggest workout it’s had so far, and naturally it highlighted a couple of issues. And, of course, beyond the sanity tests like the spectests…

Better use of invokedynamic, and other optimization

I’ve got a talk at Build Stuff next week on invokedynamic, the instruction added to the JVM in JDK7 to support those implementing languages of a more dynamic nature So, in the last week I spent some time tweaking our usage of it to get some further wins in various areas, and to make sure I was nicely familiar with it again in time for my talk. That work got merged into the mainline development branches today. I did some other optimizations along the way that are a win for all backends, too; lizmat noted a 12.5% decrease in spectest time. Not Bad.


Jonathan Worthington | 6guts | 2013-12-06 23:40:05

## A useful, usable, "early adopter" distribution of Perl 6

On behalf of the Rakudo and Perl 6 development teams, I'm happy to
announce the November 2013 release of "Rakudo Star", a useful and usable
distribution of Perl 6. The tarball for the November 2013 release is
available from <http://rakudo.org/downloads/star/>. A Windows .MSI
version of Rakudo star will usually appear in the downloads area
shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language
("Perl 6") and specific implementations of the language such as
"Rakudo Perl". This Star release includes [release 2013.11] of the
[Rakudo Perl 6 compiler], version 5.9.0 of the [Parrot Virtual
Machine], plus various modules, documentation, and other resources
collected from the Perl 6 community.

[release 2013.11]:
https://github.com/rakudo/rakudo/blob/nom/docs/announce/2013.11.md
[Rakudo Perl 6 compiler]: http://github.com/rakudo/rakudo
[Parrot Virtual Machine]: http://parrot.org

Some of the new features added to this release include:

+ Order::Increase/Decrease are deprecated. Please use Order::Less/More.
+ Leading whitespace is ignored for :sigspace
+ Better null pattern detection in regexes
+ improved run()/shell(), these return Proc::Status-objects now
+ The "gethostname" function implemented
+ Performance optimization: unfold junctions in 'when' clauses

This release also contains a range of bug fixes, improvements to error
reporting and better failure modes.

Please note that this release of Rakudo Star does not support the JVM
backend from the Rakudo compiler. While the JVM backend mostly
implements the same features as the Parrot backend, many bits are still
missing, most prominently the native call interface.
We hope to provide a JVM-based Rakudo Star release soon.

The following notable features have been deprecated or modified from
previous releases due to changes in the Perl 6 specification, and are
planned to be removed or changed as follows:

* All unary hyper ops currently descend into nested arrays and
hashes. In the future, those operators and methods that are
defined "nodal" will behave like a one-level map.

There are some key features of Perl 6 that Rakudo Star does not yet
handle appropriately, although they will appear in upcoming releases.
Some of the not-quite-there features include:

* advanced macros
* threads and concurrency (in work for the JVM backend)
* Unicode strings at levels other than codepoints
* interactive readline that understands Unicode
* non-blocking I/O
* much of Synopsis 9 and 11

There is an online resource at <http://perl6.org/compilers/features>
that lists the known implemented and missing features of Rakudo and
other Perl 6 implementations.

In many places we've tried to make Rakudo smart enough to inform the
programmer that a given feature isn't implemented, but there are many
that we've missed. Bug reports about missing and broken features are
welcomed at <rakudobug@perl.org>.

See <http://perl6.org/> for links to much more information about
Perl 6, including documentation, example code, tutorials, reference
materials, specification documents, and other supporting resources. A
draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in
the release tarball.

The development team thanks all of the contributors and sponsors for
making Rakudo Star possible. If you would like to contribute, see
<http://rakudo.org/how-to-help>, ask on the <perl6-compiler@perl.org>
mailing list, or join us on IRC \#perl6 on freenode.

Perl 6 Announce | perl.perl6.announce | 2013-11-24 15:54:54

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the November 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the November 2013 release is available from http://rakudo.org/downloads/star/. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.11 of the Rakudo Perl 6 compiler, version 5.9.0 of the Parrot Virtual Machine, plus various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

  • Order::Increase/Decrease are deprecated. Please use Order::Less/More.
  • Leading whitespace is ignored for :sigspace
  • Better null pattern detection in regexes
  • improved run()/shell(), these return Proc::Status-objects now
  • The “gethostname” function implemented
  • Performance optimization: unfold junctions in ‘when’ clauses

This release also contains a range of bug fixes, improvements to error reporting and better failure modes.

Please note that this release of Rakudo Star does not support the JVM backend from the Rakudo compiler. While the JVM backend mostly implements the same features as the Parrot backend, many bits are still missing, most prominently the native call interface. We hope to provide a JVM-based Rakudo Star release soon.

The following notable features have been deprecated or modified from previous releases due to changes in the Perl 6 specification, and are planned to be removed or changed as follows:

  • All unary hyper ops currently descend into nested arrays and hashes. In the future, those operators and methods that are defined “nodal” will behave like a one-level map.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:

  • advanced macros
  • threads and concurrency (in work for the JVM backend)
  • Unicode strings at levels other than codepoints
  • interactive readline that understands Unicode
  • non-blocking I/O
  • much of Synopsis 9 and 11

There is an online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at rakudobug@perl.org.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

rakudo.org | rakudo.org | 2013-11-24 14:29:02

Great quotations are the wisdom of the tribe.
They bridge time and space.
They connect the living and the dead.

The Talmud says the right quotation at the right moment is
like "bread to the Famished."
May you be Fed.
-- from "Sunbeams: A Book of Quotations" [2nd ed], by Sy Safransky
(Inadvertently creating a meta-meta-quote)

On behalf of the Parrot team, I'm proud to announce Parrot 5.10.0, also known
as "Sun Conure". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 5.10.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/5.10.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 5.10.0 News:
- Core
+ Fixed bareword method names check for " in imcc [GH #1015]
+ Moved eval from eval.pmc to HLLCompile and use new packfile API.
This is a prerequisite for --target=pbc in the HLLCompiler [GH #937]
+ Merged branch 'new-packfile-api' [GH #937]
= Removed Eval PMC
= IMCCompiler now returns PackfileView instead of Eval.
= Added Parrot_pf_single_sub_by_tag() to packfile API.
= Added first_sub_in_const_table() to PackfileView PMC as a
stopgap until properly tagged subs are generated.
- Build
+ Removed wrong -Wlogical-op exception for imcparser.c [GH #1015]
+ Fixed parsing for OpenGL 4.1 on OS X Mavericks. [GH #1016]
- Documentation
- Tests
- Community


The SHA256 message digests for the downloadable tarballs are:
417d504ccf557528d179c9f25df4f8430d5ec1e703ea63126d722452bfd38be3 parrot-5.10.0.tar.gz
6030f72adccdb577a8e781e3d81c52dc60d68c6a9e2be626db3cff69e1f36ce5 parrot-5.10.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 17 Dec 2013.

Enjoy!

Perl 6 Announce | perl.perl6.announce | 2013-11-22 13:35:57

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the October 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the October 2013 release is available from http://rakudo.org/downloads/star/. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.10 of the Rakudo Perl 6 compiler, version 5.9.0 of the Parrot Virtual Machine, plus various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

  • postcircumfix {} and [] are now implemented as multi subs rather than multi methods.
  • Add support for “is DEPRECATED”, making it easy for early adopters to stay current.
  • Track multiple spec changes for various container classes.
  • Greatly reduce object creation during Regex parsing.
  • Various portability fixes.
  • qx// and run() now auto-quote correctly.
  • Allow #`[...]-style comments in regexes.
  • unlink() behaves like P5′s, it deletes write-protected files on Windows.

This release also contains a range of bug fixes, improvements to error reporting and better failure modes.

Please note that this release of Rakudo Star does not support the JVM backend from the Rakudo compiler. While the JVM backend mostly implements the same features as the Parrot backend, many bits are still missing, most prominently the native call interface. We hope to provide a JVM-based Rakudo Star release soon.

The following notable features have been deprecated or modified from previous releases due to changes in the Perl 6 specification, and are planned to be removed or changed as follows:

  • postcircumfix:<[ ]> and postcircumfix:<{ }> are now multi-subs rather than multi-methods. Both at_pos and at_key remain as methods.

  • All unary hyper ops currently descend into nested arrays and hashes. In the future, those operators and methods that are defined “nodal” will behave like a one-level map.

  • The Str.ucfirst builtin is deprecated; it will be replaced by Str.tc. In this Rakudo Star release, use of Str.ucfirst will actually generate a warning upon first usage.

  • In the next release, leading whitespace in rules and under :sigspace will no longer be converted to <.ws>. For existing regexes that expect this conversion now and in the future, add a <?> in front of leading whitespace to replicate the old behavior. (Adding <?> today will have no adverse effects on your code.)

There are some key features of Perl 6 that Rakudo Star does not yet
handle appropriately, although they will appear in upcoming releases.
Some of the not-quite-there features include:

  • advanced macros
  • threads and concurrency
  • Unicode strings at levels other than codepoints
  • interactive readline that understands Unicode
  • non-blocking I/O
  • much of Synopsis 9 and 11

We are also aware of these issues in the current release, which will be dealt with in a future relase:

  • Whatever-currying of *<>, *{}, and *[] is currently broken.

  • You may experience some test failures running the rakudo spectest. S16-filehandles/filestat.t will fail if you have Rakudo Star on a drive that does not record access times. S05-mass/charsets.rakudo.parrot will fail due to the tests being outdated.

There is an online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the
programmer that a given feature isn’t implemented, but there are many
that we’ve missed. Bug reports about missing and broken features are
welcomed at rakudobug@perl.org.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in
the release tarball.

The development team thanks all of the contributors and sponsors for
making Rakudo Star possible. If you would like to contribute, see
http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org/a> mailing list, or join us on IRC #perl6 on freenode.

rakudo.org | rakudo.org | 2013-10-29 19:37:21

Recently I read the course material of the Rakudo and NQP Internals Workshop, and had an idea for a small optimization for the regex engine. Yesterday night I implemented it, and I'd like to walk you through the process.

As a bit of background, the regex engine that Rakudo uses is actually implemented in NQP, and used by NQP too. The code I am about to discuss all lives in the NQP repository, but Rakudo profits from it too.

In addition one should note that the regex engine is mostly used for parsing grammar, a process which involves nearly no scanning. Scanning is the process where the regex engine first tries to match the regex at the start of the string, and if it fails there, moves to the second character in the string, tries again etc. until it succeeds.

But regexes that users write often involve scanning, and so my idea was to speed up regexes that scan, and where the first thing in the regex is a literal. In this case it makes sense to find possible start positions with a fast string search algorithm, for example the Boyer-Moore algorithm. The virtual machine backends for NQP already implement that as the index opcode, which can be invoked as start = index haystack, needle, startpos, where the string haystack is searched for the substring needle, starting from position startpos.

From reading the course material I knew I had to search for a regex type called scan, so that's what I did:

$ git grep --word scan
3rdparty/libtommath/bn_error.c:   /* scan the lookup table for the given message
3rdparty/libtommath/bn_mp_cnt_lsb.c:   /* scan lower digits until non-zero */
3rdparty/libtommath/bn_mp_cnt_lsb.c:   /* now scan this digit until a 1 is found
3rdparty/libtommath/bn_mp_prime_next_prime.c:                   /* scan upwards 
3rdparty/libtommath/changes.txt:       -- Started the Depends framework, wrote d
src/QRegex/P5Regex/Actions.nqp:                     QAST::Regex.new( :rxtype<sca
src/QRegex/P6Regex/Actions.nqp:                     QAST::Regex.new( :rxtype<sca
src/vm/jvm/QAST/Compiler.nqp:    method scan($node) {
src/vm/moar/QAST/QASTRegexCompilerMAST.nqp:    method scan($node) {
Binary file src/vm/moar/stage0/NQPP6QRegexMoar.moarvm matches
Binary file src/vm/moar/stage0/QASTMoar.moarvm matches
src/vm/parrot/QAST/Compiler.nqp:    method scan($node) {
src/vm/parrot/stage0/P6QRegex-s0.pir:    $P5025 = $P5024."new"("scan" :named("rx
src/vm/parrot/stage0/QAST-s0.pir:.sub "scan" :subid("cuid_135_1381944260.6802") 
src/vm/parrot/stage0/QAST-s0.pir:    push $P5004, "scan"

The binary files and .pir files are generated code included just for bootstrapping, and not interesting for us. The files in 3rdparty/libtommath are there for bigint handling, thus not interesting for us either. The rest are good matches: src/QRegex/P6Regex/Actions.nqp is responsible for compiling Perl 6 regexes to an abstract syntax tree (AST), and src/vm/parrot/QAST/Compiler.nqp compiles that AST down to PIR, the assembly language that the Parrot Virtual Machine understands.

So, looking at src/QRegex/P6Regex/Actions.nqp the place that mentions scan looked like this:

    $block<orig_qast> := $qast;
    $qast := QAST::Regex.new( :rxtype<concat>,
                 QAST::Regex.new( :rxtype<scan> ),
                 $qast,
                 ($anon
                      ?? QAST::Regex.new( :rxtype<pass> )
                      !! (nqp::substr(%*RX<name>, 0, 12) ne '!!LATENAME!!'
                            ?? QAST::Regex.new( :rxtype<pass>, :name(%*RX<name>) )
                            !! QAST::Regex.new( :rxtype<pass>,
                                   QAST::Var.new(
                                       :name(nqp::substr(%*RX<name>, 12)),
                                       :scope('lexical')
                                   ) 
                               )
                          )));

So to make the regex scan, the AST (in $qast) is wrapped in QAST::Regex.new(:rxtype<concat>,QAST::Regex.new( :rxtype<scan> ), $qast, ...), plus some stuff I don't care about.

To make the optimization work, the scan node needs to know what to scan for, if the first thing in the regex is indeed a constant string, aka literal. If it is, $qast is either directly of rxtype literal, or a concat node where the first child is a literal. As a patch, it looks like this:

--- a/src/QRegex/P6Regex/Actions.nqp
+++ b/src/QRegex/P6Regex/Actions.nqp
@@ -667,9 +667,21 @@ class QRegex::P6Regex::Actions is HLL::Actions {
     self.store_regex_nfa($code_obj, $block, QRegex::NFA.new.addnode($qast))
     self.alt_nfas($code_obj, $block, $qast);
 
+    my $scan := QAST::Regex.new( :rxtype<scan> );
+    {
+        my $q := $qast;
+        if $q.rxtype eq 'concat' && $q[0] {
+            $q := $q[0]
+        }
+        if $q.rxtype eq 'literal' {
+            nqp::push($scan, $q[0]);
+            $scan.subtype($q.subtype);
+        }
+    }
+
     $block<orig_qast> := $qast;
     $qast := QAST::Regex.new( :rxtype<concat>,
-                 QAST::Regex.new( :rxtype<scan> ),
+                 $scan,
                  $qast,

Since concat nodes have always been empty so far, the code generators don't look at their child nodes, and adding one with nqp::push($scan, $q[0]); won't break anything on backends that don't support this optimization yet (which after just this patch were all of them). Running make test confirmed that.

My original patch did not contain the line $scan.subtype($q.subtype);, and later on some unit tests started to fail, because regex matches can be case insensitive, but the index op works only case sensitive. For case insensitive matches, the $q.subtype of the literal regex node would be ignorecase, so that information needs to be carried on to the code generation backend.

Once that part was in place, and some debug nqp::say() statements confirmed that it indeed worked, it was time to look at the code generation. For the parrot backend, it looked like this:

    method scan($node) {
        my $ops := self.post_new('Ops', :result(%*REG<cur>));
        my $prefix := self.unique('rxscan');
        my $looplabel := self.post_new('Label', :name($prefix ~ '_loop'));
        my $scanlabel := self.post_new('Label', :name($prefix ~ '_scan'));
        my $donelabel := self.post_new('Label', :name($prefix ~ '_done'));
        $ops.push_pirop('repr_get_attr_int', '$I11', 'self', %*REG<curclass>, '"$!from"');
        $ops.push_pirop('ne', '$I11', -1, $donelabel);
        $ops.push_pirop('goto', $scanlabel);
        $ops.push($looplabel);
        $ops.push_pirop('inc', %*REG<pos>);
        $ops.push_pirop('gt', %*REG<pos>, %*REG<eos>, %*REG<fail>);
        $ops.push_pirop('repr_bind_attr_int', %*REG<cur>, %*REG<curclass>, '"$!from"', %*REG<pos>);
        $ops.push($scanlabel);
        self.regex_mark($ops, $looplabel, %*REG<pos>, 0);
        $ops.push($donelabel);
        $ops;
    }

While a bit intimidating at first, staring at it for a while quickly made clear what kind of code it emits. First three labels are generated, to which the code can jump with goto $label: One as a jump target for the loop that increments the cursor position ($looplabel), one for doing the regex match at that position ($scanlabel), and $donelabel for jumping to when the whole thing has finished.

Inside the loop there is an increment (inc) of the register the holds the current position (%*REG<pos>), that position is compared to the end-of-string position (%*REG<eos>), and if is larger, the cursor is marked as failed.

So the idea is to advance the position by one, and then instead of doing the regex match immediately, call the index op to find the next position where the regex might succeed:

--- a/src/vm/parrot/QAST/Compiler.nqp
+++ b/src/vm/parrot/QAST/Compiler.nqp
@@ -1564,7 +1564,13 @@ class QAST::Compiler is HLL::Compiler {
         $ops.push_pirop('goto', $scanlabel);
         $ops.push($looplabel);
         $ops.push_pirop('inc', %*REG<pos>);
-        $ops.push_pirop('gt', %*REG<pos>, %*REG<eos>, %*REG<fail>);
+        if nqp::elems($node.list) && $node.subtype ne 'ignorecase' {
+            $ops.push_pirop('index', %*REG<pos>, %*REG<tgt>, self.rxescape($node[0]), %*REG<pos>);
+            $ops.push_pirop('eq', %*REG<pos>, -1, %*REG<fail>);
+        }
+        else {
+            $ops.push_pirop('gt', %*REG<pos>, %*REG<eos>, %*REG<fail>);
+        }
         $ops.push_pirop('repr_bind_attr_int', %*REG<cur>, %*REG<curclass>, '"$!from"', %*REG<pos>);
         $ops.push($scanlabel);
         self.regex_mark($ops, $looplabel, %*REG<pos>, 0);

The index op returns -1 on failure, so the condition for a cursor fail are slightly different than before.

And as mentioned earlier, the optimization can only be safely done for matches that don't ignore case. Maybe with some additional effort that could be remedied, but it's not as simple as case-folding the target string, because some case folding operations can change the string length (for example ß becomes SS while uppercasing).

After successfully testing the patch, I came up with a small, artifical benchmark designed to show a difference in performance for this particular case. And indeed, it sped it up from 647 ± 28 µs to 161 ± 18 µs, which is roughly a factor of four.

You can see the whole thing as two commits on github.

What remains to do is implementing the same optimization on the JVM and MoarVM backends, and of course other optimizations. For example the Perl 5 regex engine keeps track of minimal and maximal string lengths for each subregex, and can anchor a regex like /a?b?longliteral/ to 0..2 characters before a match of longliteral, and generally use that meta information to fail faster.

But for now I am mostly encouraged that doing a worthwhile optimization was possible in a single evening without any black magic, or too intimate knowledge of the code generation.

Update: the code generation for MoarVM now also uses the index op. The logic is the same as for the parrot backend, the only difference is that the literal needs to be loaded into a register (whose name fresh_s returns) before index_s can use it.

Moritz Lenz | Perlgeek.de | 2013-10-19 09:22:52

Try to remember the kind of September
When life was slow and, oh, so mellow.
Try to remember the kind of September
When grass was green and grain was yellow.
-- El Gallo in "The Fantasticks"
(World's Longest Running Musical)

On behalf of the Parrot team, I'm proud to announce Parrot 5.9.0, also known
as "Fantastick Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 5.9.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/supported/5.9.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 5.9.0 News:
- Core
+ Don't complain about utf8 multibyte characters split by chunk boundaries
+ Fixed readall SEGV at eof, e.g. when calling readall twice [GH #981]
+ Fixed SEGV when .const not found. [GH #996]
+ Added PackfileView-compatible methods to EvalPMC. [GH #937] (Temporary)
- Build
+ Fixed MacOSX -Wno-long-double warning [GH #980]
- Testing
+ Sorted hash keys to allow better diffs between runs of `make fulltest`.
- Branches
+ The new-packfile-api branch with the final removal of the EvalPMC
is ready and will most likely land in 5.10.0
- Documentation
+ Fixed and improved the documentation about Parrot_str_new
- Community
+ Zefram reported 31 issues. Thanks a lot!
+ All three of our GSoC students completed their projects, and passed
their finals with flying colors. Congratulations! Their weekly reports
are available here:
= Saurabh Kumar - "Update parrot-libgit2 to latest libgit2 release"
http://www.parrot.org/tags/parrot-libgit2
= Paweł Murias - "A Javascript backend for Rakudo"
http://parrot.org/blog/1850
= Denis Boyun - "Improve Web UI of App::Parrot::Create"
http://parrot.org/blog/1843


The SHA256 message digests for the downloadable tarballs are:
52d2e7c33ba9b2c3ed692738af8a27b13c838d45790cee0609e116c724d3d7b4 parrot-5.9.0.tar.gz
b4704231d90ddec827f45f945b9ad13dd4f9dc7cc9bc0cfc97ab6e30ec1c38ca parrot-5.9.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 19 Nov 2013.

Enjoy!

Perl 6 Announce | perl.perl6.announce | 2013-10-16 06:31:47

I thought it was high time for a round-up of news about the Perl 6 related stuff that I’ve been working on – as usual, with the help of many others. So, here goes!

MoarVM Backend in NQP master

In the last progress update, I mentioned that MoarVM could host NQP and pass most of the NQP language test suite. We got to that point by using a cross-compiler running on Parrot to compile the NQP sources into MoarVM bytecode. Fast forward a month, and we’ve reached the point of having NQP bootstrapped on MoarVM. What’s the difference? Simply, that you can use NQP running on MoarVM in order to build an NQP from source. This means that both bytecode generation and serialization are now working. Since some of the source files we’re compiling are also fairly large (4000 lines), it’s also been a fairly good hardening exercise; many bugs have been fixed.

The MoarVM support has now been added to the NQP repository. Just as you only need a JVM to build NQP for the JVM, you also only need MoarVM to build an NQP for MoarVM. That is, it’s bootstrapped and can stand alone now. NQP monthly releases from here on will thus come with support for three backends: Parrot, JVM and MoarVM.

Better still, NQP on MoarVM now passes the full NQP test suite. It also does so faster than any other backend. This doesn’t mean MoarVM is faster for all things; if you write NQP code with a tight integer loop, or something long running, the JVM’s JIT will kick in and typically come out ahead after the slow start. MoarVM just gets moving a lot faster than the JVM, which is what matters for running lots of short-lived tests.

This means we’re now ready to start work on getting Rakudo running on MoarVM. I don’t have any predictions just yet on precisely when we might land this; the rate of progress in the next couple of weeks, as we dig into it, will provide an indication. Reaching the point of being able to bootstrap NQP on MoarVM is a significant achievement along the way, though. While NQP is both smaller and simpler than full Perl 6, it still requires being able to execute Perl 6 grammars, eval code, do a whole range of OO-related things (including classes, roles and meta-programming), perform multiple dispatch, handle BEGIN time, and so on. These are all things that a full Perl 6 needs, so it’s very good to have them in place.

Allocating Less Cursors

Allocation profiling work by hoelzro and timotimo (I hope I remembered right; correct me if not!) indicated that both NQP and Rakudo were allocating a huge number of Cursor objects. So what are these? They are objects that keep track of parsing state. They are created as we enter a rule or token in the grammar, thrown away if it fails, or typically placed into a tree of cursors if it passes (though sometimes we only care for pass/fail and so quickly throw it away again). Naturally, we’re going to allocate quite a few of these, but 2.14 million of them being allocated while parsing Rakudo’s CORE.setting seemed decidedly excessive.

I decided to spend some time trying to understand where they all came from. NQP itself tends to be a bit easier to analyze than Rakudo, so I started there, and added some instrumentation to record the number of times each production rule or token was hit and led to allocation of a Cursor. I then used this to gather statistics on a fairly large source file from the NQP build. It started out at 284,742 Cursor allocations.

The first big win from this came when I realized that a huge number of these Cursors were just allocated and returned, to indicate failure. A Cursor is born in a failed state, and many places were just creating them and returning them without any further work, to say “I failed to match”. Thing is, failed Cursor objects all look about the same. The only thing they differ by is type, and for each Cursor type we have a ParseShared instance. Thus, it wasn’t too hard to create a shared “fail Cursor” and just look it up in a bunch of places, rather than allocating. That shaved off over 50,000 allocations. A related realization about improving MARKED and MARKER led to another 30,000 or so allocations chopped. Note that none of this really leads to overall reduced memory usage; all of these objects were quickly collectable. But allocation makes GC run more often, and thus carries some amount of runtime cost.

Further examination of the statistics showed hotspots that seemed unreasonable. The problem wasn’t that they allocated a Cursor needlessly, but rather than we should never have been calling the grammar rule in question so many times. This led to a number of grammar optimizations. In one case, just making sure that a certain rule got a declarative prefix meant the LTM mechanism could very quickly rule it out, saving 15,000 calls to the production rule, and thus cursors. The other big discovery was that the ident built-in was not having an NFA properly generated for it. This is notable because ident features in various paths in the grammar, and thus meant we were failing to quickly rule out a lot of impossible paths through the grammar using the NFA (which is the cheap way to rule things out). With a few other tweaks, all told, we were down to 172,424 Cursor allocations, just 60% of what we started out with.

The statistics for Rakudo’s CORE.setting showed we started out doing 2,196,370 Cursor allocations. Some of the fixes above also aided Rakudo directly, making a good cut into this number. However, analysis of the statistics revealed there were more wins to be had. So far, we managed to bring it down to 1,231,085 – a huge reduction. There’s likely a lot more wins to be had here, but I think we’ve picked all of the large, low-hanging fruit by now.

Rakudo Debugger on JVM

I spent some time getting the Rakudo Debugger to also work on the JVM. I’ve still got some work to do on making it easily buildable, and so it can be installed with Panda on the JVM like it can on Parrot. But the rest of the work is done, and I was able to step through programs, set breakpoints, examine variables and so forth. So, that’s another of the “things missing on the JVM port” mostly gone.

Rakudo on JVM spectest

I also did a little work to fix more of the spectests that pass on Rakudo Parrot, but fail on Rakudo JVM. The result: 99.9% of the spectests that pass on Rakudo Parrot also pass on Rakudo JVM. To put an exact number on it, that’s 27 tests different. Around half are Unicode related, which isn’t so surprising since we’ve borrowed Java strings for the time being; in the end, we’ll need to do an NFG implementation on the JVM. The others still need to be hunted down. I think it’s fair to say that you’re rather unlikely to run into this 0.1% in day-to-day usage of Rakudo on the JVM, however. In fact, the thing you’re most likely to miss today is NativeCall – which arnsholt has been working on, and I plan to join in with during the coming days.

All in all, things are coming along very nicely with Rakudo’s JVM support. It was just 11 months ago – not even a year – that I started to build a little bit of infrastructure so we’d be able to create a tree-like data structure in NQP and get JVM bytecode made from it. By now, thanks to the contributions of many besides me, Rakudo runs on the JVM and is very capable there. When it comes to concurrency, it’s the most capable Rakudo you can get. And, once it gets past the slow startup, it’s usually the fastest. It’ll be interesting to see where we stand on these things in another six months or a year’s time. Here’s hoping for another productive time ahead!


Jonathan Worthington | 6guts | 2013-10-11 21:31:37

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the September 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the September 2013 release is available from http://rakudo.org/downloads/star/. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.09 of the Rakudo Perl 6 compiler, version 5.5.0 of the Parrot Virtual Machine, plus various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

  • candidate argument to bless removed (per spec change)
  • @a.VAR.name and %h.VAR.name implemented
  • The $var.++ and $var.() syntaxes work
  • basics of tr/// implemented
  • Sets/Bags/KeySet/KeyBag now up to spec, except for the empty set symbol ‘∅’

This release also contains a range of bug fixes, improvements to error reporting and better failure modes.

Please note that this release of Rakudo Star does not support the JVM backend from the Rakudo compiler. While the JVM backend mostly implements the same features as the Parrot backend, many bits are still missing, most prominently the native call interface. We hope to provide a JVM-based Rakudo Star release soon.

The following features have been deprecated or modified from previous releases due to changes in the Perl 6 specification, and are planned to be removed or changed as follows:

  • postcircumfix:<[ ]> and postcircumfix:<{ }> will become
    multi-subs rather than multi-methods IN THE NEXT RELEASE of Rakudo
    Star. Both at_pos and at_key will remain methods.

  • All unary hyper ops currently descend into nested arrays and
    hashes. In the future, those operators and methods that are
    defined “nodal” will behave like a one-level map.

  • The Str.ucfirst builtin is deprecated; it will be replaced by
    Str.tc. In the next Rakudo Star release, use of Str.ucfirst will actually
    generate a warning upon first usage.

  • Leading whitespace in rules and under :sigspace will no longer be
    converted to <.ws>. For existing regexes that expect this
    conversion, add a <?> in front of leading whitespace to make it
    meta again.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:

  • advanced macros
  • threads and concurrency
  • Unicode strings at levels other than codepoints
  • interactive readline that understands Unicode
  • non-blocking I/O
  • much of Synopsis 9 and 11

There is an online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at rakudobug@perl.org.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

rakudo.org | rakudo.org | 2013-09-26 13:13:08

A little over a month ago, lizmat++ contacted my employer, Edument AB, to discuss using our training services to deliver a 2-day workshop aimed at current and potential Rakudo and NQP contributors. For those of you who don’t know, aside from working on Perl 6 I also work teaching/mentoring on various topics (typically software architecture, TDD, and advanced C# programming). The goal was for me to spend a couple of days explaining a bunch of topics – including NQP, grammars, QAST, 6model, and bounded serialization – to help people know where to start or to help them get into unfamiliar areas of the code.

The workshop took place this last weekend in Frankfurt. I’d expected we might have 5-6 people sign up; in the end, we had around 15 people attending! The workshop involved a combination of teaching and exercises, with plenty of chances to ask questions. Happily, there were tasty lunches and dinners too (though I can’t take any of the credit for this side of things). I greatly enjoyed teaching the course; I like both working on Perl 6 and doing my teaching work at Edument, and being able to do both at once was delightful! :-)

One aim of developing the course was to help with the state of documentation of the NQP and Rakudo internals. Typically at Edument, we only give out our material to those attending a delivery of our courses, and even then not the original source! However, for this course, we’ve made an exception and released the course material under a Creative Commons license. So, while I hope that we’ll be able to put together future “live” deliveries of the material for other new (or potential) contributors, this means it will now always be available to the community at large. :-)

I hope this will prove a valuable resource for those contributing or interested in contributing to Perl 6 development, and would like to take a moment to thank Edument for once again being supportive of my work on the Perl 6 project!


Jonathan Worthington | 6guts | 2013-09-17 15:39:52

A while back, I wrote here to introduce the MoarVM project, which aims to build a Virtual Machine specially tailored for the needs of NQP and Rakudo. Since then, a lot of progress has been made and the project has attracted a number of new contributors. So, it’s time for a round-up of the latest MoarVM news.

MoarVM Hosts NQP

Back when I first introduced MoarVM, we had a partly complete cross-compiler that used NQP on Parrot to parse NQP source code and build its AST, then turned it into MoarVM bytecode. This enabled us to cross-compile some of the NQP test suite to MoarVM. Since then, we’ve been improving the cross-compiler so it is able to translate NQP itself into MoarVM bytecode, as well as making the VM capable of doing the things that NQP requires to run.

In the last week, we’ve reached the point of having a self-hosted NQP on MoarVM that passes most of the NQP test suite. What does this really mean? That MoarVM is capable enough not only to run programs written in NQP, but also to run the NQP compiler (meaning it has pretty complete Perl 6 grammar support), its own backend code-generator and to be able to evaluate the result of this compilation.

We’re down to 5 files from t/nqp that either fail outright or fail some tests. I expect we’ll be able to get these fixed within the next week, and also make progress on the serialization tests. Beyond this, it will be time to start working on getting this self-hosted NQP able to build itself. At this point, the cross-compiler will no longer be needed, and the MoarVM backend code will migrate to the main NQP repository, joining the Parrot and JVM backends.

How slow is it?

I’ve been pushing pretty hard for us to focus on getting things working before getting things fast; for now keeping the code easy to work on and implementing features on the path to having a Rakudo on MoarVM is far, far more important. That said, there is already an encouraging result. While it’s not quite a fair test given the five of the 77 test files that do not run to completion, currently MoarVM can run the NQP test suite in 16s on my main development machine, This compares to 21s on Parrot (which means we come out ahead at startup time, execution time, or a mixture). While it’s not incredibly faster (though of course running lots of small test files is startup-dominated), it isn’t a bad starting point given we’ve barely scratched the surface of what we’ll be able to do in terms of optimization (for example, no JIT yet, known algorithmic issues in handling of GC inter-generational pointers, and comparatively little time invested in optimization all over the VM). By the way, thanks to its much greater start-up time, NQP on the JVM takes a huge 139s. However, much like in the story of the tortoise and the hare, leave things to run for long enough and the JVM backend still typically comes out ahead, sometimes very significantly so.

On memory usage, to compile and run a simple “hello, world” in NQP, MoarVM’s memory usage clocks in at just over half that which Parrot uses, which in turn clocks in at about half that of the hungry JVM. :-)

While the JVM backend doesn’t come out looking too awesome here resource usage wise unless you’re doing something long-running, it’s worth remembering that the JVM port of Rakudo is just a couple of months old, and the whole porting effort less than a year old. In that sense, we’ve a lot of room for improvement yet.

Goodbye APR, welcome libuv

Originally, MoarVM was using the Apache Portable Runtime for IO, multi-platform thread support and a handful of other things. By now, we’ve moved over to using libuv. This isn’t because there was anything inherently wrong with the APR, and while we’ve ended up not using it, I’ve come out with a positive impression of it. So, thanks to the APR developers! The primary reason we moved to using libuv instead was to ensure we can provide support for asynchronous I/O on a relatively wide variety of platforms. We were also using a mix of the atomic primitives from the APR and from libatomic_ops; we’ve now standardized completely on libatomic_ops for this.

Build and Dependencies

There’s a bunch of stuff I’m decent at doing, but build systems is not one of them. My first pass was a case of, “make something work, and hope somebody replaces it.” Happily, that has now happened, so the build system is in much better shape. We’ve also got things in place for building MoarVM as a shared library, and for pulling in varius dependencies while keeping them at enough of a distnace that we should be able to depend on system versions of them (trying to balance easy source builds with keeping future packagers happier).

Contributors

Since becoming a public project, MoarVM has grown its contributor base. In total, 16 authors now show up in the commit log and I’d say we’re at around 5 core developers who commit notable things relatively often. I’m happy that people have felt inspired to contribute, and that things are progressing fairly swiftly now (I hoped we’d be a little further by this point, but my tuit supply over the summer was not what I’d hoped). Currently, it looks like the October NQP release will be the first to include MoarVM support, and from there we’ll focus on getting Rakudo onto MoarVM.


Jonathan Worthington | 6guts | 2013-09-08 18:15:01