Planet JM

Category: Submerged

  • Sparse Directories, Now With Exclusion

    “cmpilato ❤ sparse directories”

    If I had a dollar for every time I’ve typed that… well, you and I could at least spring for some Fazoli’s fast-food Italian. Okay, I admit that the emotion doesn’t always drive me to public expression, but that in no way diminishes my fondness for this feature.

    Introduced in Subversion 1.5, sparse directory support is one of a few features from that release (besides merge tracking and foreign repository merges) that I’ve fully integrated into my day-to-day activities. I dig organization. I tend to keep a pretty neat home directory. But I routinely work on several different pieces of software, and at any given time, I’m tracking several different development branches in each of those pieces of software. Were I not using sparse directories, my “projects” directory would look something like this:

    $ ls ~/projects
subversion/		      svnbook/		 viewvc-1.0.7/
subversion-1.5.x/	      thotkeeper/	 viewvc-1.0.x/
subversion-1.6.x/	      thotkeeper-0.3.0/  viewvc-1.1.0-beta1/
subversion-http-protocol-v2/  viewvc/		 viewvc-1.1.x/
$

    On the positive side of things, I could quickly update all my working copies by simply running svn update ~/projects/*.

    But those of you who have command-line completion as hard-wired into your habits as I do will immediately notice that so many common directory prefixes does a useless completion environment make. And not using common prefixes? Well that’s just barbaric.

    Fortunately, sparse directories has given me a whole new perspective on working copy organization. Now, my projects directory contains (gasp!) only projects, and looks like this:

    $ ls ~/projects
subversion/  svnbook/  thotkeeper/  viewvc/
$

    Those directories are sparse checkouts of the root directories of their respective project repositories. Beneath them are (at least) “trunk” directories, probably “branch” and some of its children, and maybe even “tags” with some of its children in certain cases. svn up ~/projects/* still works, and my working copy topology matches that of the repositories.

    I won’t go into the details of how sparse checkouts works here — I’ve already documented the Subversion 1.5 implementation of them in the second edition of Version Control With Subversion (which you can read at http://svnbook.red-bean.com/en/1.5/svn.advanced.sparsedirs.html). The point of this blog post is to tell you about how Subversion 1.6 improves this feature in a key way.

    In Subversion 1.6 (slated for release Any Day Now), the --set-depth parameter to svn update has grown a new value — exclude. This value tells Subversion to exclude the target from the working copy, immediately and until further notice. Prior to Subversion 1.6, if a branch I was working on was no longer of interest to me, I couldn’t easily remove it from my working copy. If I simply deleted it, it would return the next time I updated the working copy. If I svn delete‘d it, the branch remained as a local modification forever. (Unless, of course, I accidentally committed it, which brought a whole different sort of trouble to my doorstep. Angry peers. Pitchforks and torches. It was a mess — you don’t want to go there.) The new exclusion mechanism in Subversion 1.6 is the Right Way To Do It.

    Say I no longer care about what’s going on some directory of one my project working copies. Maybe I don’t care about the Subversion project’s website any more. Well, with this new exclusion feature, I can tell Subversion to remove that directory:

    $ cd ~/projects/subversion/trunk
$ svn update --set-depth=exclude www
D         www
$ ls www
ls: cannot access www: No such file or directory
$

    Done deal. When I update my working copy in the future, I will not receive any changes aimed at that www directory. If I later decide that I once again care about that directory, I can “resubscribe” to it again:

    $ svn update --set-depth=infinity www
A    www
A    www/links.html
A    www/testing-goals.html
…
A    www/tigris-permissions.html
A    www/webdav-usage.html
Updated to revision 36292.
$

    Note that if you exclude a versioned directory that has some unversioned files in it, or some files with local modifications, Subversion handles this situation gracefully. All the files that aren’t safe to delete, Subversion leaves around, and of course leaves any intermediate directories required to reach those files, too.

    I hope this enhancement serves you as well as it has served me. What do you think? How are you using sparse directories to better organize your life?

  • Subversion with Apache and LDAP: Updated

    My previous blog entry discussing Subversion, Apache and LDAP is nearing two years old. It was written when Apache 2.0.x was still the mainstream and when Apache 2.2.x was released, changes in the LDAP modules and their respective configuration directives has left my previous entry very confusing for those wanting to use Apache 2.2.x. The purpose of the Definitive Guide is to provide a single location for questions for Apache 2.0.x and 2.2.x, while also providing more depth about things to consider when building your Apache-based Subversion server using LDAP for authentication.

    The Configuration

    For those of you that just want to get to the point, where you can copy and paste and move on, here you go:

    Example Apache 2.2.x Configuration Snippet

    # Load Apache LDAP modules
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

# Load Subversion Apache Modules
LoadModule dav_svn_module     modules/mod_dav_svn.so # Use full path to SUBVERSION_HOME/bin/mod_dav_svn.so on Windows
LoadModule authz_svn_module   modules/mod_authz_svn.so # Use full path to SUBVERSION_HOME/bin/mod_authz_svn.so on Windows

# Work around authz and SVNListParentPath issue
RedirectMatch ^(/repos)$ $1/

# Enable Subversion logging
CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION

<Location /repos/>
  # Enable Subversion
  DAV svn

  # Directory containing all repository for this path
  SVNParentPath /subversion/svn-repos

  # List repositories colleciton
  SVNListParentPath On

  # Enable WebDAV automatic versioning
  SVNAutoversioning On

  # Repository Display Name
  SVNReposName "Your Subversion Repository"

  # Do basic password authentication in the clear
  AuthType Basic

  # The name of the protected area or "realm"
  AuthName "Your Subversion Repository"

  # Make LDAP the authentication mechanism
  AuthBasicProvider ldap

  # Make LDAP authentication is final
  AuthzLDAPAuthoritative on

  # Active Directory requires an authenticating DN to access records
  AuthLDAPBindDN "CN=ldapuser,CN=Users,DC=your,DC=domain"

  # This is the password for the AuthLDAPBindDN user in Active Directory
  AuthLDAPBindPassword ldappassword

  # The LDAP query URL
  AuthLDAPURL "ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)"

  # Require a valid user
  Require valid-user

  # Authorization file
  AuthzSVNAccessFile /subversion/apache2/auth/repos.acl
</Location>

    Example Apache 2.0.x Configuration Snippet

    # Load Apache LDAP modules
LoadModule ldap_module modules/mod_ldap.so
LoadModule auth_ldap_module modules/mod_auth_ldap.so

# Load Subversion Apache Modules
LoadModule dav_svn_module     modules/mod_dav_svn.so # Use full path to SUBVERSION_HOME/bin/mod_dav_svn.so on Windows
LoadModule authz_svn_module   modules/mod_authz_svn.so # Use full path to SUBVERSION_HOME/bin/mod_authz_svn.so on Windows

# Work around authz and SVNListParentPath issue
RedirectMatch ^(/repos)$ $1/

# Enable Subversion logging
CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION

<Location /repos/>
  # Enable Subversion
  DAV svn

  # Directory containing all repository for this path
  SVNParentPath /subversion/svn-repos

  # List repositories colleciton
  SVNListParentPath On

  # Enable WebDAV automatic versioning
  SVNAutoversioning On

  # Repository Display Name
  SVNReposName "Your Subversion Repository"

  # LDAP Authentication is final
  AuthLDAPAuthoritative on

  # Do basic password authentication in the clear
  AuthType Basic

  # The name of the protected area or "realm"
  AuthName "Your Subversion Repository"

  # Active Directory requires an authenticating DN to access records
  AuthLDAPBindDN "CN=ldapuser,CN=Users,DC=your,DC=domain"

  # This is the password for the AuthLDAPBindDN user in Active Directory
  AuthLDAPBindPassword ldappassword

  # The LDAP query URL
  AuthLDAPURL "ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)"

  # Require authentication
  Require valid-user

  # Authorization file
  AuthzSVNAccessFile /subversion/apache2/auth/repos.acl
</Location>

    (The configurations above were for pointing to an Active Directory (AD) server.

    Understanding the Configuration

    So…the above Apache configurations are what I personally use when building an Apache-based server. Obviously there are changes that need to be made depending on the environment in but for now, it’s a great start. To make the best of this opportunity, let’s talk about the miscellaneous parts of the configuration.

    SVNListParentPath and Subversion’s authz

    One of the first problems people run into when building an Apache-based Subversion server is when they want to have mod_dav_svn serve a list of repositories. Everything works fine until they enable Subversion’s authorization (authz) support. What happens is the server will be configured properly and secured properly but when you go to the repository collection list, which in our case is http://localhost/repos, you are forbidden to view the collection even if you have access. Well, with the RedirectMatch closer to the top of the configuration, you fix this issue. How you might be asking and the reason is that when you enable authz, you must have a trailing slash at the end of the collection url. With the RedirectMatch, we automatically redirect urls to the collection listing when there is no trailing slash. Problem solved.

    Custom Subversion Logging

    Subversion uses Apache’s WebDAV support for providing access to its repositories when using Apache. Unfortunately, when you look at Apache’s access logs to try and see your Subversion usage, you end up with a lot of WebDAV communication being logged and you only see a portion of the actual client/server communication. This is because mod_dav_svn uses Apache subrequests and Apache does not log subrequests. Even if it did, turning the Subversion communication in the Apache access log into something meaningful would be nearly impossible. That being said, the configuration above has been setup to use one of Subversion’s features: Apache Logging which takes the guess work out.

    Subversion Configuration

    The other Subversion-specific parts of the Apache configuration are pretty self-explanitory. To summarize what is enabled with the above:

    • SVNListParentPath: Enables the ability to browse the location root and get a list of repositories being served by that url base
    • SVNAutoversioning: Enables the use of WebDAV clients to make changes to the repository contents without using a Subversion client
    • SVNParentPath: Enables serving N number of repositories for the url base
    • SVNReposName: Enables you to put in your own text to be visible in the web browser when browsing your repository contents via the built-in repository browser provided by mod_dav_svn
    • AuthzSVNAccessFile: Tells Subversion’s mod_authz_svn module where to find the authz file.

    For more details about the Subversion-specific Apache directives, and a list of even more ways you can configure your Apache-based Subversion server, view the mod_dav_svn and the mod_authz_svn documentation.

    LDAP Configuration

    The LDAP portion of the Apache configuration is where most people run into problems. That being said, we’ll spend a little more time explaining the Apache LDAP configuration. The most important thing to note is the subtle differences between Apache 2.0.x and Apache 2.2.x:

    Apache 2.0.x           | Apache 2.2.x
-----------------------------------------------
AuthLDAPAuthoritative  | AuthzLDAPAuthoritative
AuthLDAPBindDN         | AuthLDAPBindDN
AuthLDAPBindPassword   | AuthLDAPBindPassword
AuthLDAPURL            | AuthLDAPURL
                       | AuthBasicProvider

    You should note that the Apache LDAP module names have also changed between Apache 2.0.x and 2.2.x. Now that we see the naming changes, let’s talk about how to properly use these Apache directives to get the LDAP-based authentication you’re looking for. (I will be using the Apache 2.2.x names for the Apache directives. If you’re still using Apache 2.0.x, please refer to the table above for how to take my documentation and apply it to Apache 2.0.x.)

    • AuthzLDAPAuthoritative: Tells Apache whether or not a failed authentication request can be passed to other Apache modules
    • AuthLDAPBindDN: The distinguished name of the user account that Apache will use to connect to the directory system to perform its user authentication
    • AuthLDAPBindPassword: The password for the user account configured via the AuthLDAPBindDN directive
    • AuthLDAPURL: This is a url that tells where the directory server is, where to look for users at, what user attribute is used to identify a user and other miscellaneous things specific to the LDAP query syntax (More on this later.)
    • AuthBasicProvider: This tells Apache which authentication module you want to use for Basic authentication

    All of the directives above are pretty straight forward except for the AuthLDAPURL directive. This directive we will discuss in more detail below. For any other Apache configuration questions, please resort to the Apache Documentation for your respective Apache version.

    The LDAP Query URL

    For most, the AuthLDAPURL directive is the most challenging to understand. There is good reason for this. That one directive actually consists of 6+ pieces of information that will be different for each Subversion server. Let’s break our example AuthLDAPURL into its pieces and discuss the importance, and nuances, of each.

    For simplicity, here is the url again, in its entirety: ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)

    • Url scheme: [ldap] This is nothing more than a url scheme. It will usually be either ‘ldap’ or ‘ldaps’ in the event that you’re using SSL for accessing your directory server.
    • Hostname: [your.domain] This is the ip address or hostname of your directory server.
    • Port: [389] This is the port the server is listening on for directory server communication.
    • Search Base: [DC=your,DC=domain] This is the distinguished name to the path in the directory tree that you want to search for users.
    • Username attribute: [sAMAccountName] This is the attribute contains the login name being used.
    • Query scope: [sub] This tells the directory server what type of query to perform.
    • Filter: [(objectClass=*)] This tells the directory server to filter the query for objects matching a particular filter

    For more details on constructing an ldap url, which is a standard and not specific to Apache, view RFC 2255.

    Working with Active Directory

    Active Directory is known as a Multi-Master Directory System. This being said, each directory server in AD does not always have all the necessary information to perform all directory server requests. The best way to handle this is to have Apache query a Global Catalog. A Global Catalog server has the ability to search at the whole forest for users. This means if you want to do domain-wide searches or larger, you need to point to a Global Catalog and you need to update your Apache configuration accordingly. When using a Global Catalog, you should be using port 3268 when performing your queries.

    Searching for Users

    In the example url above, the sAMAccountName attribute is used to identify the username. This attribute is Windows/Active Directory specific so for those of you using OpenLDAP or another option, that attribute probably will not exist. Change your attribute accordingly. An example is if you wanted to use the Common Name to login, you could specify "CN" as the attribute.

    LDAP Query Tuning

    The last thing we will talk about is the ability to use filters to make your LDAP query a little more specific. In the example url above we used "(objectClass=*)", which will search for all objects. If you know that you only want to search for a particular object type, like the "user" type, you could use "(objectClass=user)" instead.

    Conclusion

    Building an Apache-based Subversion server with LDAP as the authentication mechanism can be daunting for some. I hope this has made things easier for you.

  • Subversion 1.6.0 Release Candidate available

    The Subversion project released the first publicly available release candidate for the 1.6.0 release on Friday Feb. 20.  You can download the source for this release candidate from the Subversion project on tigris.org.  The release notes for the 1.6 release are still being assembled but you can follow the latest state of the document from the project website.

    As we did with the Subversion 1.5.0 release, CollabNet is providing binary packages of this release candidate to make it as easy as possible for the community to evaluate the release and provide their feedback.  You can download binaries for Windows and Linux right now.  We will be adding Solaris and OSX binaries soon.

    Follow this blog for more entries on specific features in Subversion 1.6.  There are several new features as you can see in the release notes.  The biggest new feature is arguably the support for tree conflicts.  This will need a separate post of its own to explain, perhaps more than one.  For those that cannot wait, I can only point to the folder with the notes the developers were keeping on the feature.  Not all of those files represent what finally went into the feature so keep that in mind and keep an eye out for more posts.

  • Subversion on Facebook

    I recently setup a page for Subversion on the popular social networking site Facebook. You can see it at http://www.facebook.com/pages/Subversion/61237832183 and — if you are a Facebook member — can "Become a Fan" of Subversion there. Fans can participate in discussions, get occasional updates about Subversion, share Subversion-related photos and videos, and be notified about Subversion-themed events. So login to Facebook, and show your support for the open source version control system that took the world by storm!

  • Subversion Revision Graph

    This past summer, via my role as project owner for the Subclipse project, I decided to participate as a mentor in the Google Summer of Code program.  Summer of Code is a great way to bring some visibility to your project as well as build relationships with other open source developers.  One of the first things you have to do when participating in Summer of Code as a mentor organization is to come up with some ideas for projects that students can work on for the summer.  It needs to be something interesting enough to attract talented students and at the same time it ought to be something where a measurable amount of work can be accomplished in just one summer.  I came up with a number of ideas, but the one I knew someone would go after, and the one I hoped would get done, was to add a revision graph feature.

    One of the top requests I hear from new Subversion users is the desire for a good revision graph feature.  Unfortunately the way that Subversion stores information in its repository is not conducive to providing a feature like this.  Specifically, Subversion does not currently track "copy-to" information.  It can tell you the history of a given item and trace that history back through all of the places it was "copied-from", but for any given path/revision pair it cannot tell you if it has been copied to another location.  This is an essential feature to create a proper revision graph, so in order to draw a decent graph, you pretty much need to have the entire history of the repository available so that you can construct the "copy-to" information yourself.  Performance would be terrible if you had to get this history every time, so a good local cache of the information is essential.  This is a fairly challenging engineering task and therefore was one that the existing Subclipse team was unlikely to ever find the time to do.  At the same time, it was a great task for a motivated student.  Combined with the ability to visualize the information in cool ways via the rich graphical library that Eclipse provides, I knew we would attract some interest.

    We were fortunate enough to attract a great student, Alberto Gimeno.  He dove right into the task when the Summer of Code program started and did a great job delivering the feature.  I was hoping we would come out of this with some interesting code that someone else might want to pick up and finish, but Alberto was able to drive this feature to a point that it was nearly ready to ship when the summer ended.  We have just spent a few months adding some Eclipse-polish to it and integrating it into Subclipse.  Today we release it, and this is what a simple graph looks like:

    Notice that the graph is capable of showing merge information when used with Subversion 1.5.  It also contains a number of nice features made available via the Eclipse Graphical Editing Framework.  Namely, the ability to zoom in/out, export to an image file as well as the ability to navigate a large graph via the Outline view.

    This feature will be included in an upcoming release of the CollabNet Desktop – Eclipse Edition.  You can get it today via the "dev-builds" update site for the CollabNet Desktop.  Download information for the CollabNet Desktop can be found at the project home here.

    Additional details on the revision graph feature can be found at the Subclipse site.

    I would like to close by once again thanking Alberto for the work he did on this feature.  I look forward to his continued involvement in this feature and the Subclipse project.  I would also thank Google for running Summer of Code and including Subclipse in the program for 2008.

  • Subversion Developer Summit – Day 1

    The core Subversion developers got together to address the current and future roadmap of Subversion. Follow along with the Subversion Developer Summit at the svn-summit project and live-blogging streams. See below for some photos.

    Img_0081jpgresize_2 See the flags of CollabNet flying high outside.

    This greets all visitors to the hotel.













    Img_0083jpgresize Subversion devs get together at the Subversion Developer Summit and do what they do best: hack Subversion. The SVN developer summit is going strong. Follow our progress here: identi.ca/svnsummit/all











    Img_0091jpgresize markphip, gstein and cmpilato belly up to the bar.

















    Img_0094jpgresize At the Subversion Roundtable. Scores of attendees filed in to hear about the latest and greatest from the core Subversion developers. There was lots of discussion about tree conflicts, restricted access, and the working copy library rewrite.











    Check back here for more news from the Subversion Developer Summit

  • Subversion Developer Summit in Full Swing

    The Subversion Developer Summit, co-located at SubConf, is now in full swing. Follow along with our daily agenda, on IRC (#subconf on irc.freenode.net), or via twitter. Got ideas for the future direction of Subversion? Submit them here.