On Clojure

February 17, 2010

Managing namespaces

Filed under: General, Libraries — khinsen @ 2:38 pm

One aspect of Clojure that I have not been quite happy with is namespace management. In a bigger project that consists of several namespaces, I usually end up having nearly identical :use and :require clauses in the initial ns form. These clauses set up the project-specific set of symbols that I want to work with. Individual namespaces sometimes add symbols for their specific needs, of course. What bothers me is that I have to repeat the :use and :require clauses, often with :exclude or :only options with many symbols, in every single namespace. And of course I often forget a copy when updating my symbol set. Therefore I decided to look at how namespaces work in more detail, and try to find a better way to manage symbols in namespaces.

For those who don’t want to read all the explanations, my solution (still a bit experimental, for the moment), is in my nstools library, which is also on Clojars.

As most Clojure programmers know, a namespace maps symbols to vars. Vars are mutable storage locations with well defined concurrency semantics, but this is not the topic of this post – see the documentation for details. But a namespace is not a simple map. To start with, a namespace stores two maps: one from symbols to their values, and one from namespace aliases to namespaces. Aliases are usually created using a (:require ... :as ...) clause in the ns form that opens a namespace. They are used in namespace-qualified symbols before the slash, as a shorthand for the full namespace name. Since aliases are used before the slash and namespace-local symbols are used after the slash (or in an unqualified name with no slash at all), there is no conflict between the two. It is thus possible to use the same symbol both as an alias and as a regular symbol in the same namespace.

The main symbol-to-value map is also not quite as simple as it seems. The values it stores are not always vars. A symbol can also have a Java class as its value. A symbol-to-class entry is created using import or using the :import clause in ns. A submap containing only the symbol-to-class entries of the namespace map can be obtained by calling ns-imports. Finally, the symbol-to-var entries can be divided up into two categories: those that refer to vars in the same namespace (created by def and the many macros based on it), and those that refer to vars in some other namespace. The latter are created with use or the :use clause of ns, and the submap of these symbols can be obtained by calling ns-refers. The first category, a submap of symbols to vars defined in the same namespace, is the return value of ns-interns.

There is one more subtlety, and an undocumented one as far as I know: Two symbols, ns and in-ns, are put in the namespace map when the namespace is created, and can’t be removed (using ns-unmap) nor redefined. This makes sense because they refer to a macro and a function needed to create new namespaces and to switch namespaces. Having them in every namespace (referring to vars in clojure.core) ensures that it is always possible to get out of the current namespace.

Next, let’s look at how namespaces are set up in Clojure. Pretty much all the namespace management functionality is available through the standard ns form with its various clauses and options. The one exception is removing symbols, which can be done only by calling ns-unmap explicitly. The ns form first switches to the namespace it defines, creating it if necessary. The second step is to add references to all public vars defined in namespace clojure.core. This step can be modified by specifying a :refer-clojure clause that lists the symbols to include or exclude. Then ns goes through its optional clauses. A :require clause loads another namespace, but doesn’t normally modify the namespace under construction. Only if the option :as is specified, there is an impact on the namespace: an alias is added. A :use clause first does a :require and then adds all of the newly loaded namespace’s public vars to the symbol table of the current namespace. The options :exclude and :only can be used to select a subset of the public vars. Finally, an :import clause adds Java classes to the namespace’s symbol table.

The most dangerous, but also most convenient, ns clause is :use. In its basic form, it adds all public vars of another namespace to the symbol table of the namespace under construction. And once those symbols are there, they cannot be redefined in the namespace, except by first removing them using ns-unmap. The problem is that “all public vars of namespace X” is not something under your control. It’s the author of the other namespace who decides which symbols you get in your namespace. The next release of namespace X may well have a few more public definitions, and if those are in conflict with your own definitions, then your module will fail to load. Therefore, as a security measure, you should use the :only option of :use with all namespaces that are out of your control, listing explicitly the definitions that you need, in order to be certain that you don’t get more than you expect. Unfortunately, this includes clojure.core, which also grows with every new Clojure release. To be on the safe side, you should have a :refer-clojure clause with the :only in every namespace that you intend to maintain for a longer time.

So far for what I have, but what do I want? I’d like to be able to set up a namespace to my taste and then be able to use it as a basis for deriving other namespaces. With that possibility, I would define a master namespace once per project, being careful to always use the :only option in :refer-clojure and :use. All other namespaces in my project would then be based on this master namespace and only add or remove symbols for their specific local needs.

To implement this functionality, I added three new clauses to ns. The :like clause takes a namespace as its only argument and adds all symbols from that namespace that refer to vars in yet another namespace to the current namespace (make sure you read this properly; there are at least three namespaces involved here!). The :clone clause does the same but also adds the symbols defined in the other namespace. In other words, :clone is equivalent to :like followed by :use. The third new clause is :remove, whose arguments are symbols to be removed from the namespace. It is explicitly allowed to “remove” symbols that aren’t there. This creates another way to protect one’s namespace against future extensions in namespaces that are :used: simply add all symbols defined in your namespace to the :remove list.

The above paragraph contains a small lie: I didn’t add anything to ns, of course, though that’s what I would have liked to do. I made a copy of ns and added the new clauses to the copy. The copy is in namespace nstools.ns and it’s called ns+ – as explained above, I cannot call it ns. So to use nstools, you have to replace ns by ns+ and put a (use 'nstools.ns) before it.

As I said, this library is still a bit experimental. I am not sure for example if both :like and :clone are necessary. And perhaps :remove should be called :exclude. Of course, any feedback is welcome!

About these ads

9 Comments »

  1. This is really interesting; I’ve had the same complaints with copy/pasting of huge blocks of use clauses all around a project. I know there’s been talk of a huge overhaul of the ns macro for a future release; I wonder if :like or something similar could make it into that. http://www.mail-archive.com/clojure@googlegroups.com/msg20403.html

    Comment by Phil — February 17, 2010 @ 6:19 pm

  2. Great post! I’ve had some problems with this myself and wrote some helpers, but didn’t had the time to push thing that far.

    > simply add all symbols defined in your namespace to the :remove list.

    This could be automated with a function that you would call at the end of your namespace. If integrate into Clojure this could also be done automatically by the compiler, with warnings to let you know.

    Comment by Nicolas Buduroi — February 17, 2010 @ 9:49 pm

    • If modifying the behaviour of the compiler is an option, then it would be sufficient to allow redefining a symbol that currently points to a var in another namespace.

      I am not sure if that’s a good idea though. At least at the REPL level, I like the error message that prevents me from overwriting something accidentally. Perhaps the default behaviour should differ between the REPL and loading from a file.

      Comment by khinsen — February 18, 2010 @ 10:38 am

  3. Just remembered something else, what would be great is a function to reset/empty a namespace.

    Comment by Nicolas Buduroi — February 18, 2010 @ 12:10 am

  4. Seriously awesome explanation of namespaces…many thanks.

    Comment by Alex Miller — February 18, 2010 @ 6:04 am

  5. I created a ticket for this: http://www.assembla.com/spaces/clojure/tickets/272-load-ns-require-use-overhaul and will bring it up on the dev list.

    Comment by Stuart Halloway — February 18, 2010 @ 8:48 am

  6. There’s already remove-ns. Combine this with in-ns, and you get the equivalent of emptying a namespace, except that the namespace object changes, rendering aliases invalid.

    But… what would you use this for in practice? The only namespace I ever wanted to reset is “user”, from the REPL. But it’s just as easy to make a new one instead.

    Comment by khinsen — February 18, 2010 @ 10:43 am

  7. This was a really useful post when trying to make sense of namespaces!

    Comment by Martin — September 11, 2010 @ 4:03 pm

  8. [...] So far I have enjoyed working with Clojure, but I still get confused by name spaces, for the reasons… One aspect of Clojure that I have not been quite happy with is namespace management. In a bigger project that consists of several namespaces, I usually end up having nearly identical :use and :require clauses in the initial ns form. These clauses set up the project-specific set of symbols that I want to work with. Individual namespaces sometimes add symbols for their specific needs, of course. What bothers me is that I have to repeat the :use and :require clauses, often with :exclude or nly options with many symbols, in every single namespace. And of course I often forget a copy when updating my symbol set. Therefore I decided to look at how namespaces work in more detail, and try to find a better way to manage symbols in namespaces. [...]

    Pingback by Namespaces in Clojure are confusing | Smash Company — September 10, 2012 @ 7:57 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Shocking Blue Green Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: