You are browsing the archive for Submerged.

Profile photo of admin

by admin

Sparse Directories, Now With Exclusion

March 4, 2009 in Submerged, subversion

“cmpilato ❤ sparse directories”

If I had a dollar for every time I’ve typed that… well, you and I could at least spring for some Fazoli’s fast-food Italian. Okay, I admit that the emotion doesn’t always drive me to public expression, but that in no way diminishes my fondness for this feature.

Introduced in Subversion 1.5, sparse directory support is one of a few features from that release (besides merge tracking and foreign repository merges) that I’ve fully integrated into my day-to-day activities. I dig organization. I tend to keep a pretty neat home directory. But I routinely work on several different pieces of software, and at any given time, I’m tracking several different development branches in each of those pieces of software. Were I not using sparse directories, my “projects” directory would look something like this:

$ ls ~/projects
subversion/		      svnbook/		 viewvc-1.0.7/
subversion-1.5.x/	      thotkeeper/	 viewvc-1.0.x/
subversion-1.6.x/	      thotkeeper-0.3.0/  viewvc-1.1.0-beta1/
subversion-http-protocol-v2/  viewvc/		 viewvc-1.1.x/
$

On the positive side of things, I could quickly update all my working copies by simply running svn update ~/projects/*.

But those of you who have command-line completion as hard-wired into your habits as I do will immediately notice that so many common directory prefixes does a useless completion environment make. And not using common prefixes? Well that’s just barbaric.

Fortunately, sparse directories has given me a whole new perspective on working copy organization. Now, my projects directory contains (gasp!) only projects, and looks like this:

$ ls ~/projects
subversion/  svnbook/  thotkeeper/  viewvc/
$

Those directories are sparse checkouts of the root directories of their respective project repositories. Beneath them are (at least) “trunk” directories, probably “branch” and some of its children, and maybe even “tags” with some of its children in certain cases. svn up ~/projects/* still works, and my working copy topology matches that of the repositories.

I won’t go into the details of how sparse checkouts works here — I’ve already documented the Subversion 1.5 implementation of them in the second edition of Version Control With Subversion (which you can read at http://svnbook.red-bean.com/en/1.5/svn.advanced.sparsedirs.html). The point of this blog post is to tell you about how Subversion 1.6 improves this feature in a key way.

In Subversion 1.6 (slated for release Any Day Now), the --set-depth parameter to svn update has grown a new value — exclude. This value tells Subversion to exclude the target from the working copy, immediately and until further notice. Prior to Subversion 1.6, if a branch I was working on was no longer of interest to me, I couldn’t easily remove it from my working copy. If I simply deleted it, it would return the next time I updated the working copy. If I svn delete‘d it, the branch remained as a local modification forever. (Unless, of course, I accidentally committed it, which brought a whole different sort of trouble to my doorstep. Angry peers. Pitchforks and torches. It was a mess — you don’t want to go there.) The new exclusion mechanism in Subversion 1.6 is the Right Way To Do It.

Say I no longer care about what’s going on some directory of one my project working copies. Maybe I don’t care about the Subversion project’s website any more. Well, with this new exclusion feature, I can tell Subversion to remove that directory:

$ cd ~/projects/subversion/trunk
$ svn update --set-depth=exclude www
D         www
$ ls www
ls: cannot access www: No such file or directory
$

Done deal. When I update my working copy in the future, I will not receive any changes aimed at that www directory. If I later decide that I once again care about that directory, I can “resubscribe” to it again:

$ svn update --set-depth=infinity www
A    www
A    www/links.html
A    www/testing-goals.html
…
A    www/tigris-permissions.html
A    www/webdav-usage.html
Updated to revision 36292.
$

Note that if you exclude a versioned directory that has some unversioned files in it, or some files with local modifications, Subversion handles this situation gracefully. All the files that aren’t safe to delete, Subversion leaves around, and of course leaves any intermediate directories required to reach those files, too.

I hope this enhancement serves you as well as it has served me. What do you think? How are you using sparse directories to better organize your life?

Profile photo of admin

by admin

Subversion with Apache and LDAP: Updated

March 3, 2009 in Submerged

My previous blog
entry discussing Subversion, Apache and LDAP is nearing two years old.
It was written when Apache 2.0.x was still the mainstream and when Apache
2.2.x was released, changes in the LDAP modules and their respective
configuration directives has left my previous entry very confusing for
those wanting to use Apache 2.2.x. The purpose of the Definitive
Guide
is to provide a single location for questions for Apache
2.0.x and 2.2.x, while also providing more depth about things to consider
when building your Apache-based Subversion server using LDAP for
authentication.

The Configuration

For those of you that just want to get to the point, where you can copy
and paste and move on, here you go:

Example Apache 2.2.x Configuration Snippet

# Load Apache LDAP modules
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

# Load Subversion Apache Modules
LoadModule dav_svn_module     modules/mod_dav_svn.so # Use full path to SUBVERSION_HOME/bin/mod_dav_svn.so on Windows
LoadModule authz_svn_module   modules/mod_authz_svn.so # Use full path to SUBVERSION_HOME/bin/mod_authz_svn.so on Windows

# Work around authz and SVNListParentPath issue
RedirectMatch ^(/repos)$ $1/

# Enable Subversion logging
CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION

<Location /repos/>
  # Enable Subversion
  DAV svn

  # Directory containing all repository for this path
  SVNParentPath /subversion/svn-repos

  # List repositories colleciton
  SVNListParentPath On

  # Enable WebDAV automatic versioning
  SVNAutoversioning On

  # Repository Display Name
  SVNReposName "Your Subversion Repository"

  # Do basic password authentication in the clear
  AuthType Basic

  # The name of the protected area or "realm"
  AuthName "Your Subversion Repository"

  # Make LDAP the authentication mechanism
  AuthBasicProvider ldap

  # Make LDAP authentication is final
  AuthzLDAPAuthoritative on

  # Active Directory requires an authenticating DN to access records
  AuthLDAPBindDN "CN=ldapuser,CN=Users,DC=your,DC=domain"

  # This is the password for the AuthLDAPBindDN user in Active Directory
  AuthLDAPBindPassword ldappassword

  # The LDAP query URL
  AuthLDAPURL "ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)"

  # Require a valid user
  Require valid-user

  # Authorization file
  AuthzSVNAccessFile /subversion/apache2/auth/repos.acl
</Location>

Example Apache 2.0.x Configuration Snippet

# Load Apache LDAP modules
LoadModule ldap_module modules/mod_ldap.so
LoadModule auth_ldap_module modules/mod_auth_ldap.so

# Load Subversion Apache Modules
LoadModule dav_svn_module     modules/mod_dav_svn.so # Use full path to SUBVERSION_HOME/bin/mod_dav_svn.so on Windows
LoadModule authz_svn_module   modules/mod_authz_svn.so # Use full path to SUBVERSION_HOME/bin/mod_authz_svn.so on Windows

# Work around authz and SVNListParentPath issue
RedirectMatch ^(/repos)$ $1/

# Enable Subversion logging
CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION

<Location /repos/>
  # Enable Subversion
  DAV svn

  # Directory containing all repository for this path
  SVNParentPath /subversion/svn-repos

  # List repositories colleciton
  SVNListParentPath On

  # Enable WebDAV automatic versioning
  SVNAutoversioning On

  # Repository Display Name
  SVNReposName "Your Subversion Repository"

  # LDAP Authentication is final
  AuthLDAPAuthoritative on

  # Do basic password authentication in the clear
  AuthType Basic

  # The name of the protected area or "realm"
  AuthName "Your Subversion Repository"

  # Active Directory requires an authenticating DN to access records
  AuthLDAPBindDN "CN=ldapuser,CN=Users,DC=your,DC=domain"

  # This is the password for the AuthLDAPBindDN user in Active Directory
  AuthLDAPBindPassword ldappassword

  # The LDAP query URL
  AuthLDAPURL "ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)"

  # Require authentication
  Require valid-user

  # Authorization file
  AuthzSVNAccessFile /subversion/apache2/auth/repos.acl
</Location>

(The configurations above were for pointing to an Active Directory (AD)
server.

Understanding the Configuration

So…the above Apache configurations are what I personally use when
building an Apache-based server. Obviously there are changes that need
to be made depending on the environment in but for now, it’s a great
start. To make the best of this opportunity, let’s talk about the
miscellaneous parts of the configuration.

SVNListParentPath and Subversion’s authz

One of the first problems people run into when building an Apache-based
Subversion server is when they want to have mod_dav_svn serve a list of
repositories. Everything works fine until they enable Subversion’s
authorization (authz) support. What happens is the server will be
configured properly and secured properly but when you go to the
repository collection list, which in our case is http://localhost/repos,
you are forbidden to view the collection even if you have access. Well,
with the RedirectMatch closer to the top of the
configuration, you fix this issue. How you might be asking and the
reason is that when you enable authz, you must have a trailing slash
at the end of the collection url. With the RedirectMatch, we
automatically redirect urls to the collection listing when there is no
trailing slash. Problem solved.

Custom Subversion Logging

Subversion uses Apache’s WebDAV support for providing access to its
repositories when using Apache. Unfortunately, when you look at Apache’s
access logs to try and see your Subversion usage, you end up with a lot
of WebDAV communication being logged and you only see a portion of the
actual client/server communication. This is because mod_dav_svn uses
Apache subrequests and Apache does not log subrequests. Even if it did,
turning the Subversion communication in the Apache access log into
something meaningful would be nearly impossible. That being said,
the configuration above has been setup to use one of Subversion’s
features: Apache Logging which
takes the guess work out.

Subversion Configuration

The other Subversion-specific parts of the Apache configuration are
pretty self-explanitory. To summarize what is enabled with the above:

  • SVNListParentPath: Enables the ability to browse the location root and
    get a list of repositories being served by that url base
  • SVNAutoversioning: Enables the use of WebDAV clients to make changes
    to the repository contents without using a Subversion client
  • SVNParentPath: Enables serving N number of repositories for the
    url base
  • SVNReposName: Enables you to put in your own text to be visible in
    the web browser when browsing your repository contents via the
    built-in repository browser provided by mod_dav_svn
  • AuthzSVNAccessFile: Tells Subversion’s mod_authz_svn module where
    to find the authz file.

For more details about the Subversion-specific Apache directives, and
a list of even more ways you can configure your Apache-based Subversion
server, view the mod_dav_svn
and the mod_authz_svn
documentation.

LDAP Configuration

The LDAP portion of the Apache configuration is where most people run
into problems. That being said, we’ll spend a little more time
explaining the Apache LDAP configuration. The most important thing to
note is the subtle differences between Apache 2.0.x and Apache 2.2.x:

Apache 2.0.x           | Apache 2.2.x
-----------------------------------------------
AuthLDAPAuthoritative  | AuthzLDAPAuthoritative
AuthLDAPBindDN         | AuthLDAPBindDN
AuthLDAPBindPassword   | AuthLDAPBindPassword
AuthLDAPURL            | AuthLDAPURL
                       | AuthBasicProvider

You should note that the Apache LDAP module names have also changed
between Apache 2.0.x and 2.2.x. Now that we see the naming changes,
let’s talk about how to properly use these Apache directives to get
the LDAP-based authentication you’re looking for. (I will be
using the Apache 2.2.x names for the Apache directives. If you’re
still using Apache 2.0.x, please refer to the table above for how to
take my documentation and apply it to Apache 2.0.x.)

  • AuthzLDAPAuthoritative: Tells Apache whether or not a failed
    authentication request can be passed to other Apache modules
  • AuthLDAPBindDN: The distinguished name of the user account that
    Apache will use to connect to the directory system to perform its
    user authentication
  • AuthLDAPBindPassword: The password for the user account configured
    via the AuthLDAPBindDN directive
  • AuthLDAPURL: This is a url that tells where the directory server
    is, where to look for users at, what user attribute is used to
    identify a user and other miscellaneous things specific to the
    LDAP query syntax (More on this later.)
  • AuthBasicProvider: This tells Apache which authentication module
    you want to use for Basic authentication

All of the directives above are pretty straight forward except for
the AuthLDAPURL directive. This directive we will
discuss in more detail below. For any other Apache configuration
questions, please resort to the
Apache Documentation
for your respective Apache version.

The LDAP Query URL

For most, the AuthLDAPURL directive is the most
challenging to understand. There is good reason for this. That
one directive actually consists of 6+ pieces of information that
will be different for each Subversion server. Let’s break our
example AuthLDAPURL into its pieces and discuss
the importance, and nuances, of each.

For simplicity, here is the url again, in its entirety:
ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)

  • Url scheme: [ldap] This is nothing more than a url scheme.
    It will usually be either ‘ldap’ or ‘ldaps’ in the event that
    you’re using SSL for accessing your directory server.
  • Hostname: [your.domain] This is the ip address or hostname
    of your directory server.
  • Port: [389] This is the port the server is listening on for
    directory server communication.
  • Search Base: [DC=your,DC=domain] This is the distinguished name
    to the path in the directory tree that you want to search
    for users.
  • Username attribute: [sAMAccountName] This is the attribute
    contains the login name being used.
  • Query scope: [sub] This tells the directory server what type
    of query to perform.
  • Filter: [(objectClass=*)] This tells the directory server to
    filter the query for objects matching a particular filter

For more details on constructing an ldap url, which is a standard
and not specific to Apache, view RFC 2255.

Working with Active Directory

Active Directory is known as a Multi-Master Directory
System
. This being said, each directory server in AD
does not always have all the necessary information to perform all
directory server requests. The best way to handle this is to have
Apache query a Global Catalog. A Global Catalog
server has the ability to search at the whole forest for users.
This means if you want to do domain-wide searches or larger, you
need to point to a Global Catalog and you need to update your
Apache configuration accordingly. When using a Global Catalog,
you should be using port 3268 when performing your queries.

Searching for Users

In the example url above, the sAMAccountName
attribute is used to identify the username. This attribute is
Windows/Active Directory specific so for those of you using
OpenLDAP or another option, that attribute probably will not exist.
Change your attribute accordingly. An example is if you wanted to
use the Common Name to login, you could specify
"CN" as the attribute.

LDAP Query Tuning

The last thing we will talk about is the ability to use filters to
make your LDAP query a little more specific. In the example url above
we used "(objectClass=*)", which will search for all objects. If you
know that you only want to search for a particular object type, like
the "user" type, you could use "(objectClass=user)" instead.

Conclusion

Building an Apache-based Subversion server with LDAP as the
authentication mechanism can be daunting for some. I hope this
has made things easier for you.

Profile photo of admin

by admin

Subversion 1.6.0 Release Candidate available

February 23, 2009 in Submerged

The Subversion project released the first publicly available release candidate for the 1.6.0 release on Friday Feb. 20.  You can download the source for this release candidate from the Subversion project on tigris.org.  The release notes for the 1.6 release are still being assembled but you can follow the latest state of the document from the project website.

As we did with the Subversion 1.5.0 release, CollabNet is providing binary packages of this release candidate to make it as easy as possible for the community to evaluate the release and provide their feedback.  You can download binaries for Windows and Linux right now.  We will be adding Solaris and OSX binaries soon.

Follow this blog for more entries on specific features in Subversion 1.6.  There are several new features as you can see in the release notes.  The biggest new feature is arguably the support for tree conflicts.  This will need a separate post of its own to explain, perhaps more than one.  For those that cannot wait, I can only point to the folder with the notes the developers were keeping on the feature.  Not all of those files represent what finally went into the feature so keep that in mind and keep an eye out for more posts.

Profile photo of admin

by admin

Subversion on Facebook

February 20, 2009 in Submerged

I recently setup a page for Subversion on the popular social networking site Facebook. You can see it at http://www.facebook.com/pages/Subversion/61237832183 and — if you are a Facebook member — can "Become a Fan" of Subversion there. Fans can participate in discussions, get occasional updates about Subversion, share Subversion-related photos and videos, and be notified about Subversion-themed events. So login to Facebook, and show your support for the open source version control system that took the world by storm!

Profile photo of admin

by admin

Subversion Revision Graph

November 10, 2008 in Submerged

This past summer, via my role as project owner for the Subclipse project, I decided to participate as a mentor in the Google Summer of Code program.  Summer of Code is a great way to bring some visibility to your project as well as build relationships with other open source developers.  One of the first things you have to do when participating in Summer of Code as a mentor organization is to come up with some ideas for projects that students can work on for the summer.  It needs to be something interesting enough to attract talented students and at the same time it ought to be something where a measurable amount of work can be accomplished in just one summer.  I came up with a number of ideas, but the one I knew someone would go after, and the one I hoped would get done, was to add a revision graph feature.

One of the top requests I hear from new Subversion users is the desire for a good revision graph feature.  Unfortunately the way that Subversion stores information in its repository is not conducive to providing a feature like this.  Specifically, Subversion does not currently track "copy-to" information.  It can tell you the history of a given item and trace that history back through all of the places it was "copied-from", but for any given path/revision pair it cannot tell you if it has been copied to another location.  This is an essential feature to create a proper revision graph, so in order to draw a decent graph, you pretty much need to have the entire history of the repository available so that you can construct the "copy-to" information yourself.  Performance would be terrible if you had to get this history every time, so a good local cache of the information is essential.  This is a fairly challenging engineering task and therefore was one that the existing Subclipse team was unlikely to ever find the time to do.  At the same time, it was a great task for a motivated student.  Combined with the ability to visualize the information in cool ways via the rich graphical library that Eclipse provides, I knew we would attract some interest.

We were fortunate enough to attract a great student, Alberto Gimeno.  He dove right into the task when the Summer of Code program started and did a great job delivering the feature.  I was hoping we would come out of this with some interesting code that someone else might want to pick up and finish, but Alberto was able to drive this feature to a point that it was nearly ready to ship when the summer ended.  We have just spent a few months adding some Eclipse-polish to it and integrating it into Subclipse.  Today we release it, and this is what a simple graph looks like:

Notice that the graph is capable of showing merge information when used with Subversion 1.5.  It also contains a number of nice features made available via the Eclipse Graphical Editing Framework.  Namely, the ability to zoom in/out, export to an image file as well as the ability to navigate a large graph via the Outline view.

This feature will be included in an upcoming release of the CollabNet Desktop – Eclipse Edition.  You can get it today via the "dev-builds" update site for the CollabNet Desktop.  Download information for the CollabNet Desktop can be found at the project home here.

Additional details on the revision graph feature can be found at the Subclipse site.

I would like to close by once again thanking Alberto for the work he did on this feature.  I look forward to his continued involvement in this feature and the Subclipse project.  I would also thank Google for running Summer of Code and including Subclipse in the program for 2008.

Profile photo of admin

by admin

Subversion Developer Summit – Day 1

October 15, 2008 in Submerged

The core Subversion developers got together to address the current and future roadmap of Subversion. Follow along with the Subversion Developer Summit at the svn-summit project and live-blogging streams. See below for some photos.

Img_0081jpgresize_2
See the flags of CollabNet flying high outside.

This greets all visitors to the hotel.












Img_0083jpgresize
Subversion devs get together at the Subversion Developer Summit and do what they do best: hack Subversion. The SVN developer summit is going strong. Follow our progress here: identi.ca/svnsummit/all










Img_0091jpgresize
markphip, gstein and cmpilato belly up to the bar.
















Img_0094jpgresize
At the Subversion Roundtable. Scores of attendees filed in to hear about the latest and greatest from the core Subversion developers. There was lots of discussion about tree conflicts, restricted access, and the working copy library rewrite.










Check back here for more news from the Subversion Developer Summit

Profile photo of admin

by admin

Subversion Developer Summit in Full Swing

October 14, 2008 in Submerged

The Subversion Developer Summit, co-located at SubConf, is now in full swing. Follow along with our daily agenda, on IRC (#subconf on irc.freenode.net), or via twitter. Got ideas for the future direction of Subversion? Submit them here.