As promised yesterday, I've put together a screen-capture movie with commentary (AKA screencast) of my Concern Map tool. If you've seen it all before, skip to the eight minute 40 second mark and check out the auto-generation of Class Diagrams based on query results. Download the screencast (.wmv, 4mb).

Today I hacked on the Class Diagram functionality in Whidbey to allow rule-based generation of diagrams. Rather than manually dragging and dropping classes, I can now write a statement to build a diagram containing classes matching certain conditions. This is useful because it cuts down the work involved in creating a diagram, and allows the contents of the Class Diagram to be recalculated so any new classes or changes are picked up dynamically. Ever wanted to see a top-level view of your system that selectively includes things like:

• Classes in a particular namespace
• Classes that implement/extend a particular interface or type
• Classes that call a certain method
• Pretty much any arbitrary condition based on code structure
• Or some combination of the above?

If so, then read on...

The background

Since late 2004 I have been working on a project to build diagrams within Visual Studio which are linked to development artefacts using queries. The project is called Concern Maps, and provides a simple box and arrow visual language for modelling arbitrary "stuff" (e.g. patterns, architectures, source and related documents, cross-cutting concerns, etc...). The example below is a Concern Map representing a data access concern, with three "concepts": Data access classes, Order entity CRUD methods and Customer entity CRUD methods.

The large orange boxes within this visual language can be annotated with query expressions, which are contained within the small paler boxes. These query expressions could be one of several languages, and at the moment I have implemented regular expressions (for selecting sections of text), file names (for selecting entire files), XPath and XQuery (for selecting program elements from an XML representation of source code), and several shortcut queries (to hide some common XPath or XQueries like "types named X" or "type implementing interface Y", see this post for a full list). The data access example above uses XPath and XQuery expressions to select individual methods and classes based on their properties.

Query expressions are executed when a user double-clicks the associated box on the Concern Map. The matching artefacts (e.g. classes, documents etc) are displayed to the user in a tree view, and can be opened at the correct position within their file by double clicking the relevant node in the tree view. The example below shows the tree view that is displayed when double clicking the Data access classes shape within the diagram.

What does this have to do with the Class Diagram?

Until now I had been content to allow users to skip straight from the Concern Map diagram to the source code by clicking on the tree view. But while jogging this morning, I had a eureka moment - why not select classes into a Class Diagram?

This improves the Concern Map experience by allowing users to hook into the Class Diagram editing capabilities. It is also a nice tweak on the Class Diagram, since the user can populate the diagram based on the properties of the code rather than manually enumerating types. This might not be a big deal when working with tens of classes, but once the number of classes gets into three of four figures it becomes a much more interesting proposition.

I've roughed up a solution containing three projects to try to illustrate the capability. The solution looks like this:

• Acme.ConcernMaps, which holds the Concern Maps associated with the solution
• Acme.WidgetManager, which holds a single Manager class
• Acme.Widgets, which holds three different widget classes

I've also added a Concern Map the Acme.ConcernMaps project to represent the Manager / Widget relationship:

The Widgets query uses the Class Diagram language I put together today, and selects classes into a Class Diagram when it is evaluated. The expression 'InNamespace Acme.Widgets' is used within the query to select all classes in the Acme.Widgets namespace. Double clicking the Widgets node causes the Concern Map system to run this query and present one result to the user - a new Class Diagram file called temp.cd.

Double clicking this new file in the treeview loads it in the main frame, as shown in the screenshot above. It has also been added to the Acme.ConcernMaps project behind the scenes, so it can be loaded from there if the user doesn't have the Concern Map open.

I'll put together a screencast tomorrow that shows this running within the IDE.

I am going to be sore tomorrow - I played my first game of tennis in a long time (two years? I remember playing in Vegas in 2003) today.

On the mental exertion front, I got v3 of my DSL tools project out the door today. The big change is the shift to a hybrid CodeModel / IL approach for building an XML representation of the solution in Visual Studio. This solves quite a few subtle problems and makes the tool a lot friendlier to use. I did have to counter quite a few mismatches between the CodeModel (e.g. source code) and IL names for things like constructors, class constructors, and explicit interface implementations, but it is working pretty well now.

I also spent a couple of hours revisiting a paper I wrote in October 2004 which now looks like it will be published: I was suprised that it all pretty much made sense, even though I don't think I had written a line of code at the stage. Anyway, hopefully that paper will see the light of day before I get the urge to mess with it again.

Tim O'Reilly's radar (slides).

I spent yesterday altering my DSL tools project to serialise the CodeModel into an XML representation of source code. It works well, and gets around a few problems with my old approach of disassembling: I don't need a project to be built, and I don't need debug information (pdbs). Unfortunately I don't get all the info I need from CodeModel, so I'm still using disassembly to read method bodies out of PE files and turn them into XML.

• Read Eric von Hippel's Democratizing Innovation, looks at the sort of ideas/products that come from users and how to encourage them. In software this is all about hacking and how to encourage it via frameworks. I'm becoming quietly obsessed with innovation, user re-purposing and hacks, to wit: my subscription to Make magazine; re-reading of Clayton Christensen; addiction to the conferences on IT Conversations. I've now started on Dan Gilmor's We The Media, which deals with some similar ideas and is fascinating.

• On Sunday I found an incredibly obvious but hitherto undiscovered problem with my research project. I had been executing XPath and XQuery expressions against XML representations of each loaded project within Visual Studio. Then I tried out this XQuery:

  for $c in //class for $m in $c/method[@visibility='Public']
  for $client in //method
  for $clientCall in $client/body/methodCall
  where $c/attributes/attribute[@type='Demo1_Common.TrustDomainBoundaryAttribute']
  and $clientCall/@name = $m/@name
  and $clientCall/@type = concat($c/@namespace, '.', $c/@name)
  return $client

  (Translation: if there is a class with the attribute 'TrustDomainBoundary', find any method that calls into a public method on that class)

  Because my schema is a fairly literal translation of IL, this just didn't work for method calls between assemblies, since it requires data from assembly A (the custom attribute and method visibilities) and from assembly B (the method body for any calling methods). God bless NUnit, I was able to extend pretty quickly to use multi-assembly XML.

• Had two great discussions on my Visual Studio 2005 / DSL tools project with the very helpful people from Microsoft on Monday. On the DSL tools front, I think I am going to totally re-build my UI based on the functionality in the next DSL tools release and then freeze it. Not sure exactly what it will look like, but I'll settle on the final UI this month. I also had a very interesting chat about different approaches to schematising source code. At the moment I'm disassembling IL and PDBs to produce XML, but I can escape some limitations and bugs by going to a hybrid approach bringing in data from CodeModel and IL. I'll try it this weekend.

• via Amazon, a Logic3 i-Station 2.1 Sub-woofer Speakers and Docking Station for iPod, currently playing Welcome to the Working Week.

I saw Interpol at the Brixton Academy last night. They were great. Last time I saw them it was Christmas day on some MTV music awards, when I was nearly dead from the flu in Sicily, so this was much better.

I just submitted a paper for the fifth joint meeting of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering. I'll post the whole thing on line at some stage over the next few days, but the abstract sums it up pretty well:

This paper discusses a tool for separating concerns within software systems. Separation of concerns helps programmers understand and alter particular aspects of a system effectively. We have developed a tool that allows programmers to represent a concern using a simple visual language, and then annotate this representation to relate parts of the concern to development artifacts like source code and documents. Programmers can then use the concern representation to interactively navigate to the relevant program artifacts. This tool takes a novel approach to the separation of concerns problem by providing several different languages for identifying relevant artifacts, as well as providing a framework for adding more languages. This allows the tool to flexibly deal with high level and low level concerns spread across many types of artifacts, and include concern definitions within other concerns to enable representations at various levels of abstraction.

Rob Caron has linked to my recent Software factories post in his TeamSystem rollup.