This final article looks at FileList, Pathmap, CLEAN, CLOBBER, and passing arguments. These are not super important for beginners right away, but they will certainly come in very handy at a later point—invaluable really.

Topics

• Passing Arguments
• FileList
• Pathmap
• Clean & Clobber
• For the Road

Passing Arguments

You have two options to pass arguments into Rake tasks. You can either do it by using Bash variables or by making use of Rake’s syntax itself.

ENV Variable

In case you haven’t played with Bash before—or Bash sounds like gobbledegook to you—let’s take five and start from the beginning. 

Bash in your shell offers two sorts of variables: global (aka environment) variables and local ones. Both are written in uppercase. The environment variables are global ones, which means they are available in all shells and don't vanish when you close one—unlike local Bash variables, which are only available in the current shell. 

Environment variables can contain data that can be used by multiple applications and are often used as a handy way to share configuration settings. In contrast to that, local Bash variables are just that, local. 

In our context of using Rake, you have the ability to access both via Ruby and in effect pass variables from the command line.

FYI

Just as a little aside, if you type env or ENV in your shell, you’ll get access to a whole bunch of environment variables. I redacted the list, but for a better understanding of what environment variables are and what they include, I encourage you to run it for yourself.

Shell

env

Output

TERM_PROGRAM=Apple_Terminal
TERM=screen-256color
SHELL=/bin/bash
TMUX=/private/var/folders/4z/3np9k5ks62b1xpbn_w_lmrgh0000gr/T/tmux-504/default,4146,0
EDITOR=vim
LANG=en_US.UTF-8
TMUX_PANE=%1
is_vim=echo "#{pane_current_command}" | grep -iqE "(^|\/)g?(view|n?vim?x?)(diff)?$"
...
...
...

If you want to see a list of local Bash variables, you can run set.

Shell

( set -o posix ; set ) | less

The set command gives you a lot more output, but the above shows you the relevant bits right away.

Ruby’s ENV Class Method

Ruby offers a way to use environment and local Bash variables alike via a hash-like accessor. For our needs, when we pass a variable to a Rake task, it’s going to be a local Bash variable, which you can find in the list of variables running set or a variation of it. Ruby can read it out using ENV['VARIABLE'].

Shell

rake prepare_book BOOKTITLE='Confessions of a unicorn'

What I want to make clear, though, is that this variable won’t get added to the ENV list that your system uses—the stuff that you saw calling env from the shell. To add it to that list, you would need to “export” it. This is another story, but I thought I should make this clear.

Some Rakefile

task :prepare_book do
  book_title = ENV['BOOKTITLE'] || 'Working Title'
  puts "Do something with the #{book_title}"
end

In this task definition, you can see how we prepared to accept or incorporate the variable passed to the task invocation. Ruby’s ENV[BASHVARIABLE] does all the heavy lifting. If BOOKTITLE had been a global environment variable, though, we could have accessed it inside this task definition as well with this syntax.

Rake Parameter Syntax

The second approach is using pure Rake syntax. You simply pass variables into square braces. That approach is better, and you can keep things more isolated. Why involve Bash if Rake is perfectly capable of handling this? Plus you don’t have any Bash variables floating around that way. If you want to pass multiple arguments into a task, it’s a lot more elegant as well.

Shell

rake "create_mi6_agent[James, Bond, 007]"

Some Rakefile

task :create_mi6_agent, [:first_name, :last_name, :number] do |t, args|
  puts "Number #{args.number} is commander #{args.first_name} #{args.last_name}."
end

When you pass in more arguments than you have defined in your task, you can simply access them via args. args.extras displays an array of all the additionally passed-in parameters. args.to_a shows you all the parameters—in an array as well, of course.

FileList

In previous examples, we have been manually collecting lists of files that need some transformation. That’s tedious, right? FileList is one of those niceties that makes Rake a powerful tool. It’s just too easy to define a glob pattern for the files you need and have it automatically be up to date when you add or delete files from that destination. With that at our disposal, filtering lists can be as straightforward or as sophisticated as we need. Regular expressions are just the tip of the iceberg—although very handy, of course.

Needing lists of files to process is very common for build tools, and making it easy to deal with them is one of the strengths of Rake. FileList makes your Rakefile smaller, smarter, and capable of handling an arbitrary number of files that you don’t need to manage. You can leave Rake in charge. 

So what is a FileList exactly?  Think of it as an array of files that match the given pattern. It’s a specialized Ruby Array that is focused on processing lists of files—storing them as strings. Once collected, they are ready for you to iterate over and to apply transformations.

Some Rakefile

image_list = FileList['images/*.png']

=> ["images/jim-weirich.png", "images/zen-rake.png"]

Managing these files by hand is one sure way to build on sand. And, of course, Rake checks the timestamps of this list and rebuilds only files that are out of date. A FileList is lazy as well. It doesn’t grab files until they are needed. If you have a bunch of file lists, they behave very sane and smart because of that. The lists that don’t get actively used are taking it easy without hitting the file system. It’s more efficient that way.

As you can see below, you can provide multiple glob patterns for the list as well.

Some Rakefile

image_list = FileList['images/*.png', 'images/*.jpg']

=> ["images/happy-jim.jpg", "images/jim-weirich.png", "images/zen-rake.png"]

With large sets of files, exclusions come in very handy—for example, if we want to filter out temporary files, backup files from editors, Git files, or certain directories that are unneeded. In short, exclusion rules are for files that you don’t want in your build.

articles = Rake::FileList.new('_posts/**/*.{markdown, md}') do |files|
  files.exclude('/_posts/drafts/*.{markdown, md}')
end

=> ["_posts/published/2016/2016-02-02-some-article.md", "_posts/published/2015/2015-12-12-another-article.markdown"] 

We can pass the files of the FileList through its initializer, which accepts a list of file masks. You process any exclusions within the block. We simplified the list of desired file extensions via {markdown, md} to keep things DRY. Also, you can chain these exclusions as much as you need to. Here we could even conveniently check if the files included in the FileList are empty (zero?) and exclude these from the array that way.

articles = Rake::FileList.new('_posts/**/*.md') do |files|
  files.exclude('/_posts/drafts/*.{markdown, md}')
  files.exclude('_posts/~*')
  files.exclude do |file|
    File.zero?(file)
  end
end

We are basically providing multiple glob patterns for collecting only files we need into the FileList. For whatever reason, you can also go the opposite way and include files into a FileList.

FL = FileList['images/*.png']
FL.include('images/private/*.jpg)

Pathmap

It’s the secret weapon of Rake and shows its true power by enabling you to manipulate file paths. It can be called on a list of files via FileList or on single files as well. Don’t forget that it works on strings, though. It’s part of an extension of Ruby’s String class.

Let’s play with a simple file and change a simple extension. We could do this with the handy ext method, of course.

Some Ruby File

"/mi6/q/secret_gadgets.xml".ext("html")

# => '/mi6/q/secret_gadgets.html'

The ext method lets us replace a file extension rather easily.  Let’s see what we can do with this file when we play around with pathmap, though. I think that’s the best way to show you what it has in store for you. We can achieve the same thing like this.

Some Rakefile

"/mi6/q/secret_gadgets.xml".pathmap('%X.html')

# => '/mi6/q/secret_gadgets.html'

As you can see, this is a bit more elegant. We provide pathmap with a specification of what we need from that string via %.

• %X

Using this, we get everything but the file extension. Then we simply add the extension we need. This is just scratching the surface, though. pathmap has lots of useful markers that let you be more creative. File path manipulations couldn’t be any easier with this. 

• %p

If you need the complete path.

"/mi6/q/secret_gadgets.xml".pathmap('%p')

# => "mi6/q/secret_gadgets.xml"

• %f

If you just need the name of a given path. No directories but with the file extension.

"/mi6/q/secret_gadgets.xml".pathmap('%f')

# => "secret_gadgets.xml"

• %n

If you need file name of a given path without its file extension.

"/mi6/q/secret_gadgets.xml".pathmap('%n')

# => "secret_gadgets"

• %d

If you need just the list of directories of a given path.

"/mi6/q/secret_gadgets.xml".pathmap('%d')

# => "mi6/q"

• %x

Extracts only the file extension.

"/mi6/q/secret_gadgets.xml".pathmap('%x')

# => ".xml"

• %s

Shows you the file separator only.

"/mi6/q/secret_gadgets.xml".pathmap('%s')

# => "/"

• %nd

If you want to specify a specific number of directories that you need. Handy for deeply nested file structures.

"/mi6/q/secret_gadgets.xml".pathmap('%1d')

# => "mi6"

"/mi6/q/secret_gadgets.xml".pathmap('%2d')

# => "mi6/q"

You can also approach it in the reverse order by using a minus.

"/mi6/q/secret_gadgets.xml".pathmap('%-2d')

# => "mi6/q"

"/mi6/q/secret_gadgets.xml".pathmap('%-1d')

# => "q"

As you can see, this one little method tackles all the various needs you can have with mapping one list of files to a different list of files. Attach it to a FileList and the magic kicks in. It really is a power tool for munging file names.

images = FileList['images/*.png']
thumbs = images.pathmap('thumbs/%n-thumbs%x')

Here, for example, we are taking a list of images and mapping them to new filenames by extracting the filenames and adding a -thumbs suffix plus the extracted file extension while putting them in a thumbs directory. I’m sure you will find very good use for pathmap.

Clean & Clobber

We want to be able to return a project to a pristine state. For that matter, a FileList is not only handy for prepping files that are to be transformed, but also makes it easy to collect files that you want to have cleaned after you are done with your tasks. CLEAN and CLOBBER are actually FileLists as well—they just have two very specific jobs to handle—Deletion.

Some Rakefile

require 'rake/clean'

CLEAN.include('*.intermediate_files')
CLOBBER.include('*.intermediate_files', 'built_files/*')

These two tasks are dumb, of course, and you need to feed them file lists via our handy include. When you run rake clean or rake clobber, these collected files will disappear. Since this is an optional module, you need to require it first in your Rakefile. The nice thing about CLEAN and CLOBBER is that they give you a central place to handle cleaning your build files. Sure, you could manually write Rake tasks yourself to handle this, but both CLEAN and CLOBBER solve it for you without reinventing the wheel.

We are not putting everything in a clean task because it’s handy that you can differentiate between intermediate and build files. Let’s say we needed to create HTML files in order to create final PDF versions of our Markdown files. We would include the HTML files into our CLEAN list. Both the .html and the final .pdf files would go into CLOBBER. Conceptually, the CLOBBER list is supposed to remove everything in both lists.

Why do we care about these build files? Sometimes you want to rebuild everything and wipe out old files to get a brand new build. Therefore you need a way to delete all the files that were generated while keeping the source files that are needed for the build—most often files that are under version control. It’s easy for these lists to get out of date when you solve this manually. Therefore, handling them like our good old friend FileList makes this process much more effective.

For the Road

• Components 

Rake, at its essence, is for managing tasks, of course. Break them down to their most useful component pieces and build them up in order to create bigger tasks. Think OOP! The same goes for your files. Rails makes this very easy for you via tasks/lib. In other projects, you can create a directory called rakelib and build your Rake components in there. Rake loads the Rakefile and rakelib/*.rake files automatically.

• Using rake --dry-run

If you need to run a task that is potentially destructive in some sense and you want to rather check first what this task would do, you can sort of sandbox the task. You will see the log of what it's doing without the file operations.

• KISS

Keep it simple! Rake is smart about doing the minimal amount possible. So should you be. The nice thing about Rake is that it provides a great DSL without giving you much rope to harm yourself by reinventing the wheel needlessly.

• Namespaces

Namespaces are cheap and keep you from running into conflicting task names. This is especially important if you have Rakefiles that are coming from different sources—and from multiple developers.

task :fight_bad_dude do
  ...
end

namespace :bond do
  task :fight_bad_dude
    ...
  end
end

• File manipulations

Make use of FileUtils and stay away from shell file manipulations inside your tasks. It’s a bit dirty when Rake makes them already directly available to you.

• Ruby commands

You can run Ruby files within Rake files. That might come in handy every once in a while.

task :some_task do
  ruby 'ruby_program.rb'
end

• Dynamically generated tasks

Use rules if you have a lot of files instead of dynamically generated tasks. Why? Run rake -P and you will get a list all of them. That can get out of hand very, very quickly. Besides that, not using rules is often simply lacking elegance. You might not have reduced the pattern to its core yet. Most importantly, this will make reuse easier while being DRY, of course.

• Lists

Instead of defining collections for files yourself—and also updating this list—we'd rather let Rake be in charge of that. As we have seen, collecting files for your tasks is not at all complicated in Rake. 

• Ruby

Make use of Ruby methods for more complex stuff. Extract methods for reuse wherever you can. Just because we are writing code in Rake files, it should not prevent us from proper encapsulation and OOP.

• Searching tasks

Shell

rake -T secret_service_agent

This, for example, will search for rake tasks with “secret_service_agent” in them. It matches the task name but not the description.

rake -W create_mi6_agent

This shows us where the task create_mi6_agent is defined.

Final Thoughts

Rake is a powerful task management and execution engine. Open-source software at its best, if you ask me. At first, I was really surprised to learn how many downloads it has amassed over the last couple of years. That this little build tool is the most popular Gem to date and has over 100 million downloads seems crazy. 

But when you look deeper into what it has to offer, it becomes crystal clear in a jiffy what a master software writer Jim Weirich really was—a true Ruby hero we all should admire for his work, legacy, and passion. I’ll leave you with a nice video interview where Jim discusses Rake. There are tons of other videos of his talks available online. Go watch all of them!