This is the fifth progress report from the Wide Finder Project; an aggregation of what other people have been saying.
Misguided? · The opinion continues to be heard, here and there, that the project is misguided; that a well-tuned regexp engine will run at I/O speed or better, thus the whole thing is I/O bound, and not suitable for speedup via parallelization.
The numbers I see coming out of the Ruby and Perl versions of the code lead me to disagree, so I’m going to keep poking around. And even if this point of view turns out to be right, that in itself would be a valuable finding: “Typical sequential logfile-processing tasks are not particularly helped by the use of many-core technologies.”
Bryan O’Sullivan took a run at the problem with Haskell, written up in What the heck is a Wide Finder, anyway?
See Also · Tom Preston-Werner is thinking about Calling Erlang from Ruby. Patrick Logan discovers Apparently Fast Erlang File Read and Regexp. The Semergence blog has I Second That Emotion, about Erlang file I/O.