Subversion Best Practices: Repository Structure

Maintaining a Subversion repository can become a complex task, and implementing the right project layout from the very beginning is crucial. As Subversion doesn’t impose a strict file structure, users are free to tailor Subversion repositories to their project’s needs. Users can organize Subversion on a ‘one project per repository’ basis or create multiple projects within the same repository; and have considerable freedom when it comes to how they use Subversion’s trunk and branches. In this post, we’ll look at some guidelines and best practices on how to keep Subversion files, for users who are embarking on a new project in Subversion.

Multiple Projects: Single Repository vs. Multiple Repositories

In modern software development, it’s normal for teams to be working on multiple projects simultaneously. If this sounds like your organization, the first question you’ll need to answer is: should I set up a single repository for multiple projects, or create one repository per project? Although the experience will be slightly different for each project, there are some general benefits and drawbacks to each approach.

Single Repository

Single repositories are typically suited to organizations managing multiple, small projects that require cross references, cross tracking, etc.

Positives:

    • there is a single location where all the code is stored, even for projects you aren’t directly involved in.
    • ability to reuse common libraries.
    • lack of duplicated maintenance (e.g only one repository needs to be backed up.)
    • the ability to move data between projects more easily, and without losing any versioning information.
    • all projects share the same repository history.
    • typically less administration – new projects can be created without creating a new repository, and without the help of sysadmin.
    • you can delete entire projects without losing the record from Subversion.

 

 

Negatives:

    • Subversion uses repository-global revision numbers which apply to entire trees, not to individual files. The revision number for projects will increase in accordance with the rest of the repository, even if no changes have been made to that particular project.
    • unable to use unified logging (svn log command).
    • branching can be complex when many folders and files are involved.
    • dumping/loading one huge repository can be time-consuming.

 

 

Multiple repositories

These are typically suited to multiple large unrelated projects.

Positives

    • the ability to define different access permissions, for different users.
    • each project repository will have its own revision sequence.
    • the version number is meaningful for all projects.
    • projects have a tendency to increase in size, and numerous large projects on a single repository can become difficult to maintain. It is typically easier to manage large projects with a ‘one repository per project’ approach.
    • can tailor each repository’s structure to suit a project’s unique needs. This can include branching, tagging, naming conventions, etc.

 

 

Negatives:

    • Subversion does not support merging code between projects in different repositories, and the transplanted code will appear in the new repository with no history. This also means you cannot perform merges if you need to temporarily maintain two versions of related code.
    • different projects have different mailing lists, which can be a problem if there’s cross-over between two related, but separate, projects.

 

 

There is also the potential to have more than one repository, and to group related projects within the same repository. This allows related projects to share data, and when new revisions are added to a repository, they are more likely to be relevant to everyone who uses the repository.

Project Layout

Once you’ve decided whether to organize your projects in a single or multiple repository structure, it’s time to plan your project layout. Putting some thought into your layout in the beginning, can help you avoid having to move files around later.


An illustration of how a Subversion Repository evolves using branching, tagging and a code trunk.

Here are some best practices for getting the most out of your project layout:

    • Project root – This is the anchoring point for a project. A repository may contain one project root, or multiple roots, but each project root contains three subdirectories: /trunk, /branches, and /tags. The use of a project root is officially recommended by the Apache Subversion project.
    • Trunk – This is where you should store current release code – only! Don’t muddy the trunk directory with revisions or release names.
    • Branches – Use these to work on significant changes, variations of code etc, without disrupting the current release code.
    • Bug fixing on a branch – Branches should be created to fix major bugs; this allows bug changes to be immediately worked on without disrupting whatever work is currently underway in the trunk/development branches.
    • “Toe in the water” branches – Branches can be used as a code “sandbox” where new technology can be tested without risking the working code. If things go right, the new code can always be merged back into the trunk.
    • Tags – Should be used as “code milestones” providing a snapshot of the code at specific points in its history.
    • Tagging bug fix / development branches – when creating a code or bug fix branch, it’s useful to create a “pre” tag, and a “post” tag after the bug fix or code change has been completed
      :

 

 

http://10.2.5.2:9880/encom/tags/PRE_authchange_bug9343

http://10.2.5.2:9880/encom/tags/POST_authchange_bug9343


Ready to start a new project with Apache Subversion? Certified open source Apache Subversion binaries can be downloaded from the WANdisco website.

via: http://blogs.wandisco.com/2011/10/24/subversion-best-practices-repository-structure/

Guidance – Branching for each Sprint

metro-visual-studio-2005-128-linkThere are a lot of developers using version control these days, but a feature of version control called branching is very poorly understood and remains unused by most developers in favour of Labels. Most developers think that branching is hard and complicated. Its not!

What is hard and complicated is a bad branching strategy. Just like a bad software architecture a bad branch architecture, or one that is not adhered to can prove fatal to a project. We I was at Aggreko we had a fairly successful Feature branching strategy (although the developers hated it) that meant that we could have multiple feature teams working at the same time without impacting each other. Now, this had to be carefully orchestrated as it was a Business Intelligence team and many of the BI artefacts do not lend themselves to merging.

Today at SSW I am working on a Scrum team delivering a product that will be used by many hundreds of developers. SSW SQL Deploy takes much of the pain out of upgrading production databases when you are not using the Database projects in Visual Studio.

With Scrum each Scrum Team works for a fixed period of time on a single sprint. You can have one or more Scrum Teams involved in delivering a product, but all the work must be merged and tested, ready to be shown to the Product Owner at the the Sprint Review meeting at the end of the current Sprint.

So, what does this mean for a branching strategy?

We have been using a “Main” (sometimes called “Trunk”) line and doing a branch for each sprint. It’s like Feature Branching, but with only ONE feature in operation at any one time, so no conflicts Smile

clip_image001
Figure: DEV folder containing the Development branches.

 

I know that some folks advocate applying a Label at the start of each Sprint and then rolling back if you need to, but I have always preferred the security of a branch.

Like:

  • being able to create a release from Main that has Sprint3 code even while Sprint4 is being worked on.
  • being sure I can always create a stable build on request.
  • Being able to guarantee a version (labels are not auditable)
  • Be able to abandon the sprint without having to delete the code (rare I know, but would be a mess if it happened)
  • Being able to see the flow of change sets through to a safe release
  • It helps you find invalid dependencies when merging to Main as there may be some file that is in everyone’s Sprint branch, but never got checked in. (We had this at the merge of Sprint2)
  • If you are always operating in this way as a standard it makes it easier to then add more scrum teams in the future. Muscle memory of this way of working.

Don’t Like:

  • Additional DB space for the branches
  • Baseless merging between sprint branches when changes are directly ported
    Note: I do not think we will ever attempt this!
  • Maybe a bit tougher to see the history between sprint branches since the changes go up through Main and down to another sprint branch
    Note: What you would have to do is see which Sprint the changes were made in and then check the history he same file in that Sprint. A little bit of added complexity that you would have to do anyway with multiple teams.
  • Over time, you can end up with a lot of old unused sprint branches. Perhaps destroy with /keephistory can help in this case.
    Note: We ALWAYS delete the Sprint branch after it has been merged into Main. That is the theory anyway, and as you can see from the images Sprint2 has already been deleted.

Why take the chance of having a problem rolling back or wanting to keep some of the code, when you can just abandon a branch and start a new one?

It just seems easier and less painful to use a branch to me! What do you think?

via: http://nakedalm.com/guidance-branching-for-each-sprint/

The 10 commandments of good source control management

Ah source control, if there’s a more essential tool which indiscriminately spans programming languages without favour, I’m yet to see it. It’s an essential component of how so many of us work; the lifeblood of many development teams, if you like. So why do we often get it so wrong? Why are some of the really core, fundamentals of version control systems often so poorly understood?

I boil it down to 10 practices – or “commandments” if you like – which often break down or are not properly understand to begin with. These are all relevant to version control products of all types and programming languages of all flavours. I’ll pick some examples from Subversion and .NET but they’re broadly applicable to other technologies.

1. Stop right now if you’re using VSS – just stop it!

It’s dead. Let it go. No really, it’s been on life support for years, taking its dying gasps as younger and fitter VCS tools have rocketed past it. And now it’s really seriously about to die as Microsoft finally pulls the plug next year (after several stays of execution).

In all fairness, VSS was a great tool. In 1995. It just simply got eclipsed by tools like Subversion then the distributed guys like Git and Mercurial. Microsoft has clearly signalled its intent to supersede it for many years now – the whole TFS thing wasn’t exactly an accident!

The point is that VSS is very broadly, extensively, almost unanimously despised due to a series of major shortcomings by today’s standards. Colloquially known as Microsoft’s source destruction system, somehow it manages to just keep clinging on to life despite extensively documented glitches, shortcomings and essential functionality (by today’s standards), which simply just doesn’t exist.

Visual Source Safe 2005 boxVSS = a great big steaming pile of shit - JUST SAY NO!

2. If it’s not in source control, it doesn’t exist

Repeat this mantra daily – “The only measure of progress is working code in source control”. Until your work makes an appearance in the one true source of code truth – the source control repository for the project – it simply doesn’t exist.

Sure, you’ve got it secreted away somewhere on your local machine but that’s not really doing anyone else any good now, is it? They can’t take your version, they can’t merge theirs, you can’t deploy it (unless you’re deploying it wrong) and you’re one SSD failure away from losing it all permanently.

Once you take the mindset of it not existing until it’s committed, a whole bunch of other good practices start to fall into place. You break tasks into smaller units so you can commit atomically. You integrate more frequently. You insure yourself against those pesky local hardware failures.

But more importantly (at least for your team lead), you show that you’re actually producing something. Declining burn down charts or ticked off tasks lists are great, but what do they actually reconcile with? Unless they correlate with working code in source control, they mean zip.

3. Commit early, commit often and don’t spare the horses

Further to the previous point, the only way to avoid “ghost code” – that which only you can see on your local machine – is to get it into VCS early and often and don’t spare the horses. Addressing the issues from the previous point is one thing the early and often approach achieves, but here’ a few others which can make a significant difference to the way you work:

  1. Every committed revision gives you a rollback position. If you screw up fundamentally (don’t lie, we all do!), are you rolling back one hour of changes or one week?
  2. The risk of a merge nightmare increases dramatically with time. Merging is never fun. Ever. When you’ve not committed code for days and you suddenly realise you’ve got 50 conflicts with other people’s changes, you’re not going to be a happy camper.
  3. It forces you to isolate features into discrete units of work. Let’s say you’ve got  a 3 man day feature to build. Oftentimes people won’t commit until the end of that period because they’re trying to build the whole box and dice into one logical unit. Of course a task as large as this is inevitably comprised of smaller, discrete functions and committing frequently forces you to identify each of these, build them one by one and commit them to VCS.

When you work this way, your commit history inevitably starts to resemble a semi-regular pattern of multiple commits each work day. Of course it’s not always going to be a consistent pattern, there are times we stop and refactor or go through testing phases or any other manner of perfectly legitimate activities which interrupt the normal development cycle.

However, when I see an individual – and particularly an entire project – where I know we should be in a normal development cycle and there are entire days or even multiple days where nothing is happening, I get very worried. I’m worried because as per the previous point, no measurable work has been done but I’m also worried because it usually means something is wrong. Often development is happening in a very “boil the ocean” sort of way (i.e. trying to do everything at once) or absolutely nothing of value is happening at all because people are stuck on a problem. Either way, something is wrong and source control is waving a big red flag to let you know.

4. Always inspect your changes before committing

Committing code into source control is easy – too easy! (Makes you wonder why the previous point seems to be so hard.) Anyway, what you end up with is changes and files being committed with reckless abandon. “There’s a change somewhere beneath my project root – quick – get it committed!”

What happens is one (or both) of two things: Firstly, people inadvertently end up with a whole bunch of junk files in the repository. Someone sees a window like the one below, clicks “Select all” and bingo – the repository gets polluted with things like debug folders and other junk that shouldn’t be in there.

A commit window showing a lot of files that shouldn't be in source control

Or secondly, people commit files without checking what they’ve actually changed. This is real easy to do once you get things like configuration or project definition files where there are a lot going on at once. It makes it really easy to inadvertently put things into the repository that simply weren’t intended to be committed and then of course they’re quite possibly taken down by other developers. Can you really remember everything you changed in that config file?

A commit window showing a web.config file with unkown changes

The solution is simple: you must inspect each change immediately before committing. This is easier than it sounds, honest. The whole “inadvertently committed file” thing can be largely mitigated by using the “ignore” feature many systems implement. You never want to commit the Thumbs.db file so just ignore it and be done with it. You also may not want to commit every file that has changed in each revision – so don’t!

As for changes within files, you’ve usually got a pretty nifty diff function in there somewhere. Why am I committing that Web.config file again?

A diff window showing inadvertant changes in the web.config file

Ah, I remember now, I wanted to decrease the maximum invalid password attempts from 5 down to 3. Oh, and I played around with a dummy login page which I definitely don’t want to put into the repository. This practice of pre-commit inspection also makes it much easier when you come to the next section…

5. Remember the axe-murderer when writing commit messages

There’s an old adage (source unknown), along the lines of “Write every commit message like the next person who reads it is an axe-wielding maniac who knows where you live”. If I was that maniac and I’m delving through reams of your code trying to track down a bug and all I can understand from your commit message is “updated some codes”, look out, I’m coming after you!

The whole idea of commit messages is to explain why you committed the code. Every time you make any change to code, you’re doing it for a reason. Maybe something was broken. Maybe the customer didn’t like the colour scheme. Maybe you’re just tweaking the build configuration. Whatever it is, there’s a reason for it and you need to leave this behind you.

Why? Well there are a few different reasons and they differ depending on the context. For example, using a “blame” feature or other similar functionality which exposes who changed what and hopefully, why. I can’t remember what I was doing in the Web.config of this project 18 months ago or why I was mucking around with app settings, but because I left a decent commit message, it all becomes very simple:

A blame log with a descriptive log message

It’s a similar thing for looking at changes over time. Whether I want to see the entire history of a file, like below, or I just want to see what the team accomplished yesterday, having a descriptive paper trail of comments means it doesn’t take much more than a casual glance to get an idea of what’s going on.

A series of well formed commit messages

And finally, commit messages are absolutely invaluable when it comes to tracking down errors. For example, getting to the bottom of why the build is breaking in your continuous integration environment. Obviously my example is overtly obvious, but the point is that bringing this information to the surface can turn tricky problems into absolute no-brainers.

A descriptive commit message on a failing TeamCity build

With this in mind, here are some anti-patterns of good commit messages:

  1. Some shit.
  2. It works!
  3. fix some fucking errors
  4. fix
  5. Fixed a little bug…
  6. Updated
  7. typo
  8. Revision 1024!!

Ok, I picked these all out of the Stack Overflow question about What is the WORST commit message you have ever authored but the thing is that none of them are that dissimilar to many of the messages I’ve seen in the past. They tell you absolutely nothing about what has actually happened in the code; they’re junk messages.

One last thing about commit messages; subsequent commit messages from the same author should never be identical. The reason is simple: you’re only committing to source control because something has changed since the previous version. Your code is now in a different state to that previous version and if your commit message is accurate and complete, it logically cannot be identical. Besides, if it was identical (perhaps there’s a legitimate edge-case there somewhere), the log is now a bit of a mess to read as there’s no way to discern the difference between the two commits.

6. You must commit your own changes – you can’t delegate it

As weird as this sounds, it happens and I’ve seen it more than once, most recently just last week. What’s happening here is that the source control repository is being placed on a pedestal. For various reasons, the team is viewing it as this sanitised, pristine environment of perfect code. In order to maintain this holy state, code is only committed by a lead developer who carefully aggregates, reviews and (assumedly) tweaks and improves the code before it’s committed.

It’s pretty easy to observe this pattern from a distance. Very infrequent commits (perhaps weekly), only a single author out of a team with multiple developers and inevitably, conflict chaos if anyone else has gone near the project during that lengthy no-commit period. Very, very nasty stuff.

There are two major things wrong here: Firstly, source control in not meant to be this virginal, unmolested stash of pristine code; at least not throughout development cycles. It’s meant to be a place where the team integrates frequently, rolls back when things go wrong and generally comes together around a single common base. It doesn’t have to be perfect throughout this process, it only has to (try to) achieve that state at release points in the application lifecycle.

The other problem – and this is the one that really blow me away – is that from the developer’s perspective, this model means you have no source control! It means no integration with code from peers, no rollback, no blame log, no nothing! You’re just sitting there in your little silo writing code and waiting to hand it off to the boss at some arbitrary point in the future.

Don’t do this. Ever.

7. Versioning your database isn’t optional

This is one of those ones that everyone knows they should be doing but very often they just file it away in the “too hard” basket. The problem you’ve got is that many (most?) applications simply won’t run without their database. If you’re not versioning the database, what you end up with is an incomplete picture of the application which in practice is rendered entirely useless.

Most VCS systems work by simply versioning files on the file system. That’s just fine for your typical app files like HTML page, images, CSS, project configuration files and anything else that sits on the file system in nice discrete little units. Problem is that’s not quite the way relational databases work. Instead, you end up with these big old data and log files which encompass a whole bunch of different objects and data. This is pretty messy stuff when it comes to version control.

SQL Source Control boxWhat changes the proposition of database versioning these days is the accessibility of tools like the very excellent SQL Source Control from Red Gate. I wrote about this in detail last year in the post about Rocking your SQL Source Control world with Red Gateso I won’t delve into the detail again; suffice to say that database versioning is now easy!

Honestly, if you’re not versioning your databases by now you’re carrying a lot of risk in your development for no good reason. You have no single source of truth, no rollback position and no easy collaboration with the team when you make changes. Life is just better with the database in source control 🙂

8. Compilation output does not belong in source control

Here’s an easy way of thinking about it: nothing that is automatically generated as a result of building your project should be in source control. For the .NET folks, this means pretty much everything in the “bin” and “obj” folders which will usually be .dll and .pdb files.

Why? Because if you do this, your co-workers will hate you. It means that every time they pull down a change from VCS they’re overwriting their own compiled output with yours. This is both a merge nightmare (you simply can’t do it), plus it may break things until they next recompile. And then once they do recompile and recommit, the whole damn problem just gets repeated in the opposite direction and this time you’re on the receiving end. Kind of serves you right, but this is not where we want to be.

Of course the other problem is that it’s just wasteful. It’s wasted on the source control machine disk, it’s wasted in bandwidth and additional latency every time you need to send it across the network and it’s sure as hell a waste of your time every time you’ve got to deal with the inevitable conflicts that this practice produces.

So we’re back to the “ignore” patterns mentioned earlier on. Once paths such as “bin” and “obj” are set to ignore, everything gets really, really simple. Do it once, commit the rule and everybody is happy.

In fact I’ve even gone so far as to write pre-commit hooks that execute on the VCS server just so this sort of content never makes it into source control to begin with. Sure, it can be a little obtrusive getting your hand slapped by VCS but, well, it only happens when you deserve it! Besides, I’d far rather put the inconvenience back on the perpetrator rather than pass it on to the entire time by causing everyone to have conflicts when they next update.

9. Nobody else cares about your personal user settings

To be honest, I think that quite often people aren’t even aware they’re committing their own personal settings into source control. Here’s what the problem is: many tools will produce artefacts which manage your own personal, local configurations. They’re only intended to be for you and they’ll usually be different to everyone else’s. If you put them into VCS, suddenly you’re all overwriting each other’s personal settings. This is not good.

Here’s an example of a typical .NET app:

Typical .NET app showing user setting files

The giveaway should be the extensions and type descriptions but in case it’s not immediately clear, the .ReSharper.user file and the .suo (Solution User Options) file are both, well, yours. They’re nobody else’s.

Here’s why: Let’s take a look inside the ReSharper file:

<Configuration>
  <SettingsComponent>
    <string />
    <integer />
    <boolean>
      <setting name="SolutionAnalysisEnabled">True</setting>
    </boolean>
  </SettingsComponent>
  <RecentFiles>
    <RecentFiles>
      <File id="F985644D-6F99-43AB-93F5-C1569A66B0A7/f:Web.config" 
caret="1121" fromTop="26" />
      <File id="F985644D-6F99-43AB-93F5-C1569A66B0A7/f:Site.Master.cs" 
caret="0" fromTop="0" />

In this example, the fact that I enabled solution analysis is recorded in the user file. That’s fine by me, I like it, other people don’t. Normally because they’ve got an aging, bargain basement PC, but I digress. The point is that this is my setting and I shouldn’t be forcing it upon everyone else. It’s just the same with the recent files node; just because I recently opened these files doesn’t mean it should go into someone else’s ReSharper history.

Amusing sidenote: the general incompetence of VSS means ignoring .ReSharper.user files is a bit of a problem.

It’s a similar story with the .suo file. Whilst there’s not much point looking inside it (no pretty XML here, it’s all binary), the file records things like the state of the solution explorer, publishing settings and other things that you don’t want to go forcing on other people.

So we’re back to simply ignoring these patterns again. At least if you’re not running VSS, that is.

10. Dependencies need a home too

Works on my machine badgeThis might be the last of the Ten Commandments but it’s a really, really important one. When an app has external dependencies which are required for it to successfully build and run, get them into source control! The problem people tend to have is that they get everything behaving real nice in their own little environment with their own settings and their own local dependencies then they commit everything into source control, walk away and think things are cool. And they are, at least until someone else who doesn’t have some same local decencies available pulls it down and everything fails catastrophically.

I was reminded of this myself today when I pulled an old project out of source control and tried to build it:

Failing build due to missing NUnit dependencies

I’d worked on the assumption that NUnit would always be there on the machine but this time that wasn’t the case. Fortunately the very brilliant NuGet bailed me out quickly, but it’s not always that easy and it does always take some fiddling when you start discovering that dependencies are missing. In some cases, they’re not going to be publicly available and it can be downright painful trying to track them down.

I had this happen just recently where I pulled down a project from source control, went to run it and discovered there was a missing assembly located in a path that began with “c:\Program Files…”. I spent literally hours trying to track down the last guy who worked on this (who of course was on the other side of the world), get the assembly, put it in a “Libraries” folder in the project and actually get it into VCS so the next poor sod who comes across the project doesn’t go through the same pain.

Of course the other reason this is very important is that if you’re working in any sort of continuous integration environment, your build server isn’t going to have these libraries installed. Or at least you shouldn’t be dependent on it. Doug Rathbone made a good point about this recently when he wrote about Third party tools live in your source control. It’s not always possible (and we had some good banter to that effect), but it’s usually a pretty easy proposition.

So do everyone a favour and make sure that everything required for your app to actually build and run is in VCS from day 1.

Summary

None of these things are hard. Honestly, they’re really very basic: commit early and often, know what you’re committing and that it should actually be in VCS, explain your commits and make sure you do it yourself, don’t forget the databases and don’t forget the dependencies. But please do forget VSS 🙂

via: http://www.troyhunt.com/2011/05/10-commandments-of-good-source-control.html