.NET

anything related to .NET

Semantic Development Environments

The semantic web is a GOOD THING by definition – anything that enables us to create smarter software without also having to create Byzantine application software must be a step in the right direction. The problem is – many people have trouble translating the generic term “smarter” into a concrete idea of what they would have to do to achieve that palladian dream. I think a few concrete ideas might help to firm up people’s understanding of how the semantic web can help to deliver smarter products.

Software Development as knowledge based activity

In this post I thought it might be nice to share a few ideas I had about how OWL and SWRL could help to produce smarter software development environments. If you want to use the ideas to make money, feel free to do so, just consider them as released under the creative commons attribution license. Software development is the quintessential knowledge based activity. In the process of producing a modern application a typical developer will burn through knowledge at a colossal rate. Frequently, we will not reserve headspace for a lot of the knowledge we acquire to solve a task. Frequently, we bring together the ideas, facts, standards, API skills and problem requirements needed to solve a problem then just as quickly forget it all. The unique combination is never likely to arise again.

I’m sure we could make a few comments about how it’s more important to know where the information is than to know what it is – a fact driven home to me by my Computer Science lecturer John English, who seemed to be able to remember the contents page of every copy of the Proceedings of the ACM back to the ’60s. You might also be forgiven for thinking this wasn’t true , given the current obsession with certifications. We could also comment about how some information is more lasting than others, but my point is that every project these days seems to combine a mixture of ephemera, timeless principles and those bits that lie somewhere between the two (called ‘Best Practice’ in current parlance ;).

Requires cognitive assistance
Software development, then, is a knowledge intensive activity that brings together a variety of structured and unstructured information to allow the developer to produce a system that they endeavor to show is equivalent to a set of requirements, guidelines, nuggets of wisdom and cultural mores that are defined or mandated at the beginning of the project. Doesn’t this sound to you like exactly the environment for which the semantic web technology stack was designed?

Incidentally, the following applications don’t have much to do with the web, so perhaps they demonstrate that the term ‘Web 3.0’ is limiting and misleading. It’s the synergy of the complementary standards in the semantic web stack that makes it possible to deliver smarter products and to boost your viability in an increasingly competitive market place.

Documentation

OK, so the extended disclaimer/apology is now out of the way and I can start to talk about how the semantic web could offer help to improve the lives of developers. The first place I’ll look is at documentation. There are many types of documentation that are used in software development. In fact, there is a different form of documentation defined for each specific stage of the software lifecycle from conception of an idea through to its realization in code (and beyond). Each of these forms of documentation is more or less formally structured with different kinds of information related to documents and other deliverables that came before and after. This kind of documentation is frequently ambiguous, verbose and often gets written for the sake of compliance and then gets filed away and never sees the light of day again. Documentation for software projects needs to be precise, terse, rich and most of all useful.

Suggestion 1.

Use ontologies (perhaps standardised by the OMG) for the production of requirements. Automated tools could be used to convert these ontologies into human-readable reports or tools could be used to answer questions about specific requirements. A reasoner might be able to deduce conflicts or contradictions from a set of requirements. It might also be able to offer suggestions about implementations that have been shown to fulfill similar requirements in other projects. Clearly, the sky’s the limit in how useful an ontology, reasoner and rules language could be. It should also help documentation to be much more precise and less verbose. There is also scope for documentation reuse, specialization and for there to be diagramming and code generation driven off of documentation.

Documentation is used heavily inside the source code used by developers to write software too. It serves to provide an explanation for the purpose of a software component, to explain how to use it, to provide change notes, to generate API documentation web-sites, and to even store to-do list items or apologies for later reference. In .NET and Java, and now many other programming languages, it is common to use formal languages (like XML markup) to provide commonly used information. An ontology might be helpful in providing a rich and extensible language for representing code documentation. The use of URIs to represent unique entities means that the documentation can be the subject or other documents and can reach out to the wider ecology of data about the system.

Suggestion 2.

Provide an extensible ontology to allow the linkage of code documentation with the rest of the documentation produced for a software system. Since all parts of the software documentation process (being documented in RDF) will have unique URIs, it should be easy to link the documentation for a component to the requirements, specifications, plans, elaborations, discussions, blog posts and other miscellanea generated. Providing semantic web URIs to individual code elements helps to integrate the code itself into other semantic systems like change management and issue tracking systems. Use of URIs and ontologies within source code helps to provide a firm, rich linkage between source code and the documentation that gave rise to it.

Suggestion 3.

Boosted with richer, extensible markups to represent the meaning and wider documentation environment means that traditional intellisense can be augmented with browsers that provide access to all other pertinent documentation related to a piece of code. Imagine hovering over an object reference and getting links not only to a web site generated from the code commentary but to all the requirements that the code fulfills, to automated proofs demonstrating that the code matches the requirements, to blog posts written by the dev team and to MP3s taken during the brainstorming and design sessions during which this component was conceived.

It doesn’t take much imagination to see that some simple enhancements like these can provide a ramp for the continued integration of the IDE, allowing smoother cooperation between teams and their stakeholders. Making documentation more useful to all involved would probably increase the chances that people would give up Agile in favour of something less like the emperor’s clothes.

Suggestion 4.

Here’s some other suggestions about how documentation in the IDE could be enriched.
○ Guidelines on where devs should focus their attention when learning a new API
○ SPARQL could be exposed by code publisher
§ Could provide a means to publish documentation online
○ Automatic publishing of DOAP documents to an enterprise or online registry, allowing software registries.

Dynamic Systems

Augmenting the source code of a system with URIs that can be referenced from anywhere opens the semantic artifacts inside an application to analysis and reference from outside. Companies like Microsoft have already described their visions for the production of documentation systems that allow architects to describe how a system hangs together. This information can be used by other systems to deploy, monitor, control and scale systems in production environments.

I think that their vision barely glimpses what could be achieved through the use of automated inference systems, rich structured machine readable design documentation, and systems that are for the first time white boxes. I think that DSI-style declarative architecture documents are a good example of what might be achieved through the use of smart documentation. There is more though.

Suggestion 5.

Reflection and other analysis tools can gather information about the structure, inter-relationships and external dependencies of a software system. Such data can be fed to an inference engine to allow it to make comparisons about the runtime behavior of a production system. Rules of inference can help it to determine what the consequences of violating a rule derived from the architect or developers documentation. Perhaps it could detect when the system is misconfigured or configured in a way that will force it to struggle under load. Perhaps it can find explanations for errors and failures. Rich documentation systems should allow developers to indicate deployment guidelines (i.e. this component is thread safe, or is location independent and scalable). Such documentation can be used to predict failure modes, to direct testing regimes and to predict optimal deployment patterns for specific load profiles.

Conclusions

I wrote this post because I know I’ll never have time to pursue these ideas, but I would dearly love to see them come to pass. Why don’t you get a copy of LinqToRdf, crack open a copy of Coco/R and see whether you can implement some of these suggestions. And if you find a way to get rich doing it, then please remember me in your will.

Wanted: Volunteers for .NET semantic web framework project

 LinqToRdf* is a full-featured LINQ** query provider for .NET written in C#. It provides developers with an intuitive way to make queries on semantic web databases. The project has been going for over a year and it’s starting to be noticed by semantic web early adopters and semantic web product vendors***. LINQ provides a standardised query language and a platform enabling any developer to understand systems using semantic web technologies via LinqToRdf. It will help those who don’t have the time to ascend the semantic web learning curve to become productive quickly.

The project’s progress and momentum needs to be sustained to help it become the standard API for semantic web development on the .NET platform. For that reason I’m appealing for volunteers to help with the development, testing, documentation and promotion of the project.

Please don’t be concerned that all the best parts of the project are done. Far from it! It’s more like the foundations are in place, and now the system can be used as a platform to add new features. There are many cool things that you could take on. Here are just a few:

Reverse engineering tool
This tool will use SPARQL to interrogate a remote store to get metadata to build an entity model.

Tutorials and Documentation
The documentation desperately needs the work of a skilled technical writer. I’ve worked hard to make LinqToRdf an easy tool to work with, but the semantic web is not a simple field. If it were, there’d be no need for LinqToRdf after all. This task will require an understanding of the LINQ, ASP.NET, C#, SPARQL, RDF, Turtle, and SemWeb.NET systems. It won’t be a walk in the park.

 

Supporting SQL Server
The SemWeb.NET API has recently added support to SQL Server, which has not been exploited inside LinqToRdf (although it may be easy to do).  This task would also involve thinking about robust scalable architectures for semantic web applications in the .NET space.

 

Porting LinqToRdf to Mono
LINQ and C# 3.0 support in Mono is now mature enough to make this a desirable prospect. Nobody’s had the courage yet to tackle it. Clearly, this would massively extend the reach of LinqToRdf, and it would be helped by the fact that some of the underlying components are developed for Mono by default.

 

SPARQL Update (SPARUL) Support
LinqToRdf provides round-tripping only for locally stored RDF. Support of SPARQL Update would allow data round-tripping on remote stores. This is not a fully ratified standard, but it’s only a matter of time.

 

Demonstrators using large scale web endpoints
There are now quite a few large scale systems on the web with SPARQL endpoints. It would be a good demonstration of LinqToRdf to be able to mine them for useful data.

 

These are just some of the things that need to be done on the project. I’ve been hoping to tackle them all for some time, but there’s just too much for one man to do alone. If you have some time free and you want to learn more about LINQ or the Semantic Web, there is not a better project on the web for you to join.  If you’re interested, reply to this letting me know how you could contribute, or what you want to tackle. Alternatively join the LinqToRdf discussion group and reply to this message there.

 

Thanks,

 

Andrew Matthews

 

* http://code.google.com/p/linqtordf

** http://msdn.microsoft.com/en-us/netframework/aa904594.aspx

*** http://virtuoso.openlinksw.com/Whitepapers/html/linqtordf/linqtordf1.htm

Announcing LinqToRdf v0.8

I’m very pleased to announce the release of version 0.8 of LinqToRdf. This release is significant for a couple of reasons. Firstly, because it provides a preview release of RdfMetal and secondly because it is the first release containing changes contributed by someone other than yours truly. The changes in this instance being provided by Carl Blakeley of OpenLink Software.

LinqToRdf v0.8 has received a few major chunks of work:

  • New installers for both the designer and the whole framework
    WIX was proving to be a pain, so I downgraded to the integrated installer generator in Visual Studio.
  • A preview release of RdfMetal. I brought this release forward a little, on Carl Blakeley’s request, to coincide with a post he’s preparing on using OpenLink Virtuoso with LinqToRdf, so RdfMetal is not as fully baked as I’d planned. But it’s still worth a look. Expect a minor release in the next few weeks with additional fixes/enhancements.

I’d like to extend a very big thank-you to Carl for the the work he’s done in recent weeks to help extend and improve the mechanisms LinqToRdf uses to represent and traverse relationships. His contributions also include improvements in representing default graphs, and referencing multiple ontologies within a single .NET class. He also provided fixes around the quoting of URIs and some other fixes in the ways LinqToRdf generates SPARQL for default graphs. Carl also provided an interesting example application using OpenLink Virtuoso’s hosted version of Musicbrainz that is significantly richer than the test ontology I created for the unit tests and manuals.

I hope that Carl’s contributions represent an acknowledgement by OpenLink that not only does LinqToRdf support Virtuoso, but that there is precious little else in the .NET space that stands a chance of attracting developers to the semantic web. .NET is a huge untapped market for semantic web product vendors. LinqToRdf is, right now, the best way to get into semantic web development on .NET.

Look out for blog posts from Carl in the next day or two, about using LinqToRdf with OpenLink Virtuoso.

State Machines in C# 3.0 using T4 Templates

UPDATE: The original code for this post, that used to be available via a link on this page, is no longer available. I’m afraid that if you want to try this one out, you’ll have to piece it together using the snippets contained in this post. Sorry for the inconvenience – blame it on ISP churn.


Some time back I wrote about techniques for implementing non-deterministic finite automata (NDFAs) using some of the new features of C# 3.0. Recently I’ve had a need to revisit that work to provide a client with a means to generate a bunch of really complex state machines in a lightweight, extensible and easily understood model. VS 2008 and C# 3.0 are pretty much the perfect platform for the job – they combine partial classes and methods, lambda functions and T4 templates making it a total walk in the park. This post will look at the prototype system I put together. This is a very code intensive post – sorry about that, but it’s late and apparently my eyes are very red, puffy and panda like.

State machines are the core of many applications – yet we often find people hand coding them with nested switch statements and grizzly mixtures of state control and business logic. It’s a nightmare scenario making code completely unmaintainable for anything but the most trivial applications.

The key objective for a dedicated application framework that manages a state machine is to provide a clean way to break out the code that manages the state machine from the code that implements the activities performed as part of the state machine. C# 3.0 has a nice solution for this – partial types and methods.

Partial types and methods

A partial type is a type whose definition is not confined to a single code module – it can have multiple modules. Some of those can be written by you, others can be written by a code generator. Here’s an example of a partial class definition:

public partial class MyPartialClass{}

By by declaring the class to be partial, you say that other files may contain parts of the class definition. the point of this kind of structure is that you might have piece of code that you want to write by hand, and others that you want to have driven from a code generator, stuff that gets overwritten every time the generator runs. If your code got erased every time you ran the generator, you’d get bored very quickly. You need a way to chop out the bits that don’t change. Typically, these will be framework or infrastructure stuff.

Partial classes can also have partial methods. Partial methods allow you to define a method signature in case someone wants to define it in another part of the partial class. This might seem pointless, but wait and see – it’s nice. Here’s how you declare a partial method:

// the code generated part public partial class MyPartialClass {
    partial void DoIt(int x);
}

You can then implement it in another file like so:

// the hand-written part partial class MyPartialClass {
    partial void DoIt(int x)
    {
        throw new NotImplementedException();
    }
}

This is all a little abstract, right now, so let’s see how we can use this to implement a state machine framework. First we need a way to define a state machine. I’m going to use a simple XML file for this:

<?xml version="1.0" encoding="utf-8" ?> <StateModels> <StateModel ID="My" start="defcon1"> <States> <State ID="defcon1" name="defcon1"/> <State ID="defcon2" name="defcon2"/> <State ID="defcon3" name="defcon3"/> </States> <Inputs> <Input ID="diplomaticIncident" name="diplomaticIncident"/> <Input ID="assassination" name="assassination"/> <Input ID="coup" name="coup"/> </Inputs> <Transitions> <Transition from="defcon1" to="defcon2" on="diplomaticIncident"/> <Transition from="defcon2" to="defcon3" on="assassination"/> <Transition from="defcon3" to="defcon1" on="coup"/> </Transitions> </StateModel> </StateModels>

Here we have a really simple state machine with three states (defcon1, defcon2 and defcon3) as well as three kinds of input (diplomaticIncident, assassination and coup). Please excuse the militarism – I just finished watching a season of 24, so I’m all hyped up. This simple model also defines three transitions. it creates a model like this:

Microsoft released the Text Template Transformation Toolkit (T4) system with Visual Studio 2008. This toolkit has been part of GAT and DSL tools in the past, but this is the first time that it has been available by default in VS. It allows an ASP.NET syntax for defining templates. Here’s a snippet from the T4 template that generates the state machine:

<#@ template language="C#" #>
<#@ assembly name="System.Xml.dll" #>
<#@ import namespace="System.Xml" #>

<#
    XmlDocument doc = new XmlDocument();
    doc.Load(@"TestHarness\model.xml");
    XmlElement xnModel = (XmlElement)doc.SelectSingleNode("/StateModels/StateModel");
    string ns = xnModel.GetAttribute("ID");
    XmlNodeList states = xnModel.SelectNodes("descendant::State");
    XmlNodeList inputs = xnModel.SelectNodes("descendant::Input");
    XmlNodeList trns = xnModel.SelectNodes("descendant::Transition");
    #>
using System;
using System.Collections.Generic;

namespace <#=ns#> {
public enum <#=ns#>States : int{
<#
string sep = "";
foreach(XmlElement s in states)
    {
    Write(sep + s.GetAttribute("ID"));
    WriteLine(@"// " + s.GetAttribute("name"));
    sep = ",";
    }

#>
} // end enum <#=ns#>States

public enum <#=ns#>Inputs : int{
<#
sep = "";
foreach(XmlElement s in inputs)
    {
    Write(sep + s.GetAttribute("ID"));
    WriteLine(@"// " + s.GetAttribute("name"));
    sep = ",";
    }

#>
} // end enum <#=ns#>States

public partial class <#=ns#>StateModel{

        public <#=ns#>StateModel()
        {
            SetupStates();
            SetupTransitions();
            SetStartState();
        }
...

Naturally, there’s a lot in the template, but we’ll get to that later. First we need a representation of a state. You’ll see from the template that an enum get’s generated called <#=ns#>States. Here’s what it looks like for the defcon model.

public enum MyStates : int {
defcon1// defcon1 ,defcon2// defcon2 ,defcon3// defcon3 } // end enum MyStates 

This is still a bit too bare for my liking. I can’t attach an event model to these states, so here’s a class that can carry around one of these values:

public class State {
    public int Identifier { get; set; }
public delegate void OnEntryEventHandler(object sender, OnEntryEventArgs e);
    // ...public event OnEntryEventHandler OnEntryEvent;
    // ...}

There’s a lot left out of this, but the point is that as well as storing an identifier for a state, it has events for both entry into and exit from the state. This can be used by the event framework of the state machine to provide hooks for your custom state transition and entry code. The same model is used for transitions:

public class StateTransition {
    public State FromState { get; set; }
    public State ToState { get; set; }
public event OnStateTransitioningEventHandler OnStateTransitioningEvent;
public event OnStateTransitionedEventHandler OnStateTransitionedEvent;
}

Here’s the list of inputs that can trigger a transition between states:

public enum MyInputs : int {
diplomaticIncident// diplomaticIncident ,assassination// assassination ,coup// coup } // end enum MyStates

The template helps to define storage for the states and transitions of the model:

public static Dictionary<<#= ns#>States, State> states                = new Dictionary<<#= ns#>States, State>();
public static Dictionary<string, StateTransition> arcs                = new Dictionary<string, StateTransition>();
public State CurrentState { get; set; }

which for the model earlier, will yield the following:

public static Dictionary<MyStates, State> states = new Dictionary<MyStates, State>();
public static Dictionary<string, StateTransition> arcs = new Dictionary<string, StateTransition>();
public State CurrentState { get; set; }

Now we can create entries in these tables for the transitions in the model:

private void SetStartState()
{
    CurrentState = states[<#= ns#>States.<#=xnModel.GetAttribute("start")#>];
}

private void SetupStates()
{
<#
foreach(XmlElement s in states)
    {
    WriteLine("states[" + ns + "States."+s.GetAttribute("ID")+"] =               new State { Identifier = (int)"+ns+"States."+s.GetAttribute("ID")+" };");
    WriteLine("states[" + ns + "States."+s.GetAttribute("ID")+"].OnEntryEvent               += (x, y) => OnEnter_"+s.GetAttribute("ID")+"();");
    WriteLine("states[" + ns + "States."+s.GetAttribute("ID")+"].OnExitEvent               += (x, y) => OnLeave_"+s.GetAttribute("ID")+"(); ;");
    }
#>
}
private void SetupTransitions()
{
<#
foreach(XmlElement s in trns)
    {
    #>
    arcs["<#=s.GetAttribute("from")#>_<#=s.GetAttribute("on")#>"] = new StateTransition
    {
        FromState = states[<#= ns#>States.<#=s.GetAttribute("from")#>],
        ToState = states[<#= ns#>States.<#=s.GetAttribute("to")#>]
    };
    arcs["<#=s.GetAttribute("from")#>_<#=s.GetAttribute("on")#>"].OnStateTransitioningEvent              += (x,y)=>MovingFrom_<#=s.GetAttribute("from")#>_To_<#=s.GetAttribute("to")#>;
    arcs["<#=s.GetAttribute("from")#>_<#=s.GetAttribute("on")#>"].OnStateTransitionedEvent              += (x,y)=>MovedFrom_<#=s.GetAttribute("from")#>_To_<#=s.GetAttribute("to")#>;
    <#
    }
    #>
}

which is where the fun starts. First notice that we create a new state for each state in the model and attach a lambda to the entry and exit events of each state. For our model that would look like this:

private void SetupStates()
{
    states[MyStates.defcon1] = new State {Identifier = (int) MyStates.defcon1};
    states[MyStates.defcon1].OnEntryEvent += (x, y) => OnEnter_defcon1();
    states[MyStates.defcon1].OnExitEvent += (x, y) => OnLeave_defcon1();

    states[MyStates.defcon2] = new State {Identifier = (int) MyStates.defcon2};
    states[MyStates.defcon2].OnEntryEvent += (x, y) => OnEnter_defcon2();
    states[MyStates.defcon2].OnExitEvent += (x, y) => OnLeave_defcon2();

    states[MyStates.defcon3] = new State {Identifier = (int) MyStates.defcon3};
    states[MyStates.defcon3].OnEntryEvent += (x, y) => OnEnter_defcon3();
    states[MyStates.defcon3].OnExitEvent += (x, y) => OnLeave_defcon3();
}

For the Transitions the same sort of code gets generated, except that we have some simple work to generate a string key for a specific <state, input> pair. Here’s what comes out:

private void SetupTransitions()
{
    arcs["defcon1_diplomaticIncident"] = new StateTransition {
                 FromState = states[MyStates.defcon1],
                 ToState = states[MyStates.defcon2]
             };
    arcs["defcon1_diplomaticIncident"].OnStateTransitioningEvent                  += (x, y) => MovingFrom_defcon1_To_defcon2;
    arcs["defcon1_diplomaticIncident"].OnStateTransitionedEvent                 += (x, y) => MovedFrom_defcon1_To_defcon2;
    arcs["defcon2_assassination"] = new StateTransition {
                 FromState = states[MyStates.defcon2],
                 ToState = states[MyStates.defcon3]
            };
    arcs["defcon2_assassination"].OnStateTransitioningEvent                += (x, y) => MovingFrom_defcon2_To_defcon3;
    arcs["defcon2_assassination"].OnStateTransitionedEvent                += (x, y) => MovedFrom_defcon2_To_defcon3;
    arcs["defcon3_coup"] = new StateTransition {
                 FromState = states[MyStates.defcon3],
                 ToState = states[MyStates.defcon1]
           };
    arcs["defcon3_coup"].OnStateTransitioningEvent                += (x, y) => MovingFrom_defcon3_To_defcon1;
    arcs["defcon3_coup"].OnStateTransitionedEvent                += (x, y) => MovedFrom_defcon3_To_defcon1;
}

You can see that for each state and transition event I’m adding lambdas that invoke methods that are also being code generated. these are the partial methods described earlier. Here’s the generator:

foreach(XmlElement s in states)
    {
    WriteLine("partial void OnLeave_"+s.GetAttribute("ID")+"();");
    WriteLine("partial void OnEnter_"+s.GetAttribute("ID")+"();");
    }
foreach(XmlElement s in trns)
    {
    WriteLine("partial void MovingFrom_"+s.GetAttribute("from")+"_To_"+s.GetAttribute("to")+"();");
    WriteLine("partial void MovedFrom_"+s.GetAttribute("from")+"_To_"+s.GetAttribute("to")+"();");
    }

Which gives us:

partial void OnLeave_defcon1();
partial void OnEnter_defcon1();
partial void OnLeave_defcon2();
partial void OnEnter_defcon2();
partial void OnLeave_defcon3();
partial void OnEnter_defcon3();
partial void MovingFrom_defcon1_To_defcon2();
partial void MovedFrom_defcon1_To_defcon2();
partial void MovingFrom_defcon2_To_defcon3();
partial void MovedFrom_defcon2_To_defcon3();
partial void MovingFrom_defcon3_To_defcon1();
partial void MovedFrom_defcon3_To_defcon1();

The C# 3.0 spec states that if you don’t choose to implement one of these partial methods then the effect is similar to attaching a ConditionalAttribute to it – it gets taken out and no trace is left of it ever having been declared. That’s nice, because for some state models you may not want to do anything other than make the transition.

We now have a working state machine with masses of extensibility points that we can use as we see fit. Say we decided to implement a few of these methods like so:

public partial class MyStateModel {
    partial void OnEnter_defcon1()
    {
        Debug.WriteLine("Going Into defcon1.");
    }
    partial void OnEnter_defcon2()
    {
        Debug.WriteLine("Going Into defcon2.");
    }
    partial void OnEnter_defcon3()
    {
        Debug.WriteLine("Going Into defcon3.");
    }
}

Here’s how you’d invoke it:

MyStateModel dfa = new MyStateModel();
dfa.ProcessInput((int) MyInputs.diplomaticIncident);
dfa.ProcessInput((int) MyInputs.assassination);
dfa.ProcessInput((int) MyInputs.coup);

And here’s what you’d get:

Going Into defcon2.
Going Into defcon3.
Going Into defcon1.

There’s a lot you can do to improve the model I’ve presented (like passing context info into the event handlers, and allowing some of the event handlers to veto state transitions). But I hope that it shows how the partials support in conjunction with T4 templates makes light work of this perennial problem. This could easily save you from writing thousands of lines of tedious and error prone boiler plate code. That for me is a complete no-brainer.

What I like about this model is the ease with which I was able to get code generation. I just added a file with extension ‘.tt’ to VS 2008 and it immediately started generating C# from it. All I needed to do at that point was load up my XML file and feed it into the template. I like the fact that the system is lightweight. There is not a mass of framework that takes over the state management, it’s infinitely extensible, and it allows a very quick turnaround time on state model changes.

What do you think? How would you tackle this problem?

LinqToRdf v0.7.1 and RdfMetal

I’ve just uploaded version 0.7.1 of LinqToRdf. This bug fix release corrects an issue I introduced in version 0.7. The issue only seemed to affect some machines and stems from the use of the GAC by the WIX installer (to the best of my knowledge). I’ve abandoned GAC installation and gone back to the original approach.

Early indications (Thanks, Hinnerk) indicate that the issue has been successfully resolved. Please let me know if you are still experiencing problems. Thanks to 13sides, Steve Dunlap, Hinnerk Bruegmann, Kevin Richards and Paul Stovell for bringing it to my attention and helping me to overcome the allure of the GAC.

Kevin also reported that he’s hoping to use LinqToRdf on a project involving the Biodiversity Information Standards (TDWG). It’s always great to hear how people are using the framework. Please drop me a line to let me know how you are using LinqToRdf.

Kevin might find feature #13 useful. It will be called RdfMetal in honour of SqlMetal. It will automate the process of working with remotely managed ontologies. RdfMetal will completely lower any barriers to entry in semantic web development. You will (in principle) no longer need to know the formats, protocols and standards of the semantic web in order to consume data in it.

clip_image001[5]

Here’s an example of the output it generated from DBpedia.org for the FOAF ontology:

 ./RdfMetal.exe -e:http://DBpedia.org/sparql -i -n http://xmlns.com/foaf/0.1/ -o foaf.cs

Which produced the following source:

namespace Some.Namespace
{
[assembly: Ontology(
    BaseUri = "http://xmlns.com/foaf/0.1/",
    Name = "MyOntology",
    Prefix = "MyOntology",
    UrlOfOntology = "http://xmlns.com/foaf/0.1/")]


    public partial class MyOntologyDataContext : RdfDataContext
    {
        public MyOntologyDataContext(TripleStore store) : base(store)
        {
        }
        public MyOntologyDataContext(string store) : base(new TripleStore(store))
        {
        }

                public IQueryable<Person> Persons
                {
                    get
                    {
                        return ForType<Person>();
                    }
                }
        
                public IQueryable<Document> Documents
                {
                    get
                    {
                        return ForType<Document>();
                    }
                }
        
                // ...        

    }

[OwlResource(OntologyName="MyOntology", RelativeUriReference="Person")]
public partial class Person
{
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "surname")]
  public string surname {get;set;}
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "family_name")]
  public string family_name {get;set;}
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "geekcode")]
  public string geekcode {get;set;}
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "firstName")]
  public string firstName {get;set;}
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "plan")]
  public string plan {get;set;}
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "knows")]
  public Person knows {get;set;}
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "img")]
  public Image img {get;set;}
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "myersBriggs")]
  // ...
}

[OwlResource(OntologyName="MyOntology", RelativeUriReference="Document")]
public partial class Document
{
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "primaryTopic")]
  public LinqToRdf.OwlInstanceSupertype primaryTopic {get;set;}
  [OwlResource(OntologyName = "MyOntology", RelativeUriReference = "topic")]
  public LinqToRdf.OwlInstanceSupertype topic {get;set;}
}
// ...

As you can see, it’s still pretty rough, but it allows me to write queries like this:

[TestMethod]
public void TestGetPetesFromDbPedia()
{
    var ctx = new MyOntologyDataContext("http://DBpedia.org/sparql");
    var q = from p in ctx.Persons 
            where p.firstName.StartsWith("Pete")
            select p;
    foreach (Person person in q)
    {
        Debug.WriteLine(person.firstName + " " + person.family_name);
    }
}

RdfMetal will be added to the v0.8 release of LinqToRdf in the not too distant future. If you have any feature requests, or want to help out, please reply to this or better-still join the LinqToRdf discussion group and post there.

Functional Programming – lessons from high-school arithmetic

I’ll start out with an apology – it was only by writing this post, that I worked out how to write a shorter post on the same topic. Sometime I’ll follow this up with something less full of digressions, explorations or justifications. The topic of the post started out as ‘Closure‘. It then became ‘Closure plus Rules of Composition‘ and finally ended up as ‘functional programming – lessons from high school arithmetic‘. The point of this post is to explore the API design principles we can learn from rudimentary high school arithmetic.  You already know all the mathematics I’m going to talk about, so don’t be put off by any terminology I introduce – it’s only in there to satisfy any passing mathematicians. ;^]

The topic of this post is much more wide-ranging than the previous ones. The post will eventually get around to talking about API design, but I really started out just wanting to dwell on some neat ideas from philosophy or mathematics that just appealed to me. The first idea is ‘Closure‘.

Closure has many meanings, but the two most relevant to this blog are:

  • A function that gets evaluated with a bound variable
  • A set that is closed under some operation.

It’s this second meaning that I want to explore today – it’s one of those wonderful philosophical rabbit holes that lead from the world of the mundane into a wonderland of deeply related concepts. As you’ll already know if you’ve read any of my earlier posts on functional programming, I am not a mathematician. This won’t be a deep exposition on category theory, but I do want to give you a flavour so that hopefully you get a sense of the depth of the concepts I’m talking about.

First let’s start with two little equations that seem to bear almost no information of interest:

(1)     1 + 1 = 2

and

(2)     2 – 1 = 1

(1) involves adding two natural numbers to get another natural number. (2) involves subtracting one natural number from another to get a third natural number. They seem to be very similar, except for the fact that if you keep repeating (2) you eventually get a negative number which is not a natural number. If you repeatedly perform addition, you can go on forever. That property is called ‘closure‘. It means that if you perform addition on any natural number you are guaranteed to get a valid result. That closure guarantee for some operations is one of the first things I want you to ponder – some operations give you guarantees of object validity, while others don’t. We need to learn how to spot those ideas.

Another interesting thing that some introspection reveals about equation (2) is that the set from which it takes it’s values is bounded in one direction, and that at the lower bound is a value that is idempotent for the operation. That term idempotent is daunting to us non-mathematicians but what it means is simply that when the operation is performed the result remains unchanged, no matter how many times it gets performed. Here’s another thing that is worth pondering – some operations are stable because they guarantee not to change your state.

Digression. Why on earth would anyone ever waste their time in writing code that was designed at the outset to do nothing? It seems like the ultimate exercise in futility. The answer is that idempotent operations are not doing nothing when in the presence of ‘rules of combination’. With rules of combination (of which more later), idempotent operations become a useful tool in composing functions.

SubDigression: A rule of combination is a feature of a system allowing you to combine distinct entities of a domain together to form a new entity. You can see how this relates to closure. It relates to closure on two levels. For example, when adding two integers:

  • The result of adding two integers is an integer. That’s closure on the set of integers.
  • The composition of two closed functions is itself closed. That’s closure at the level of functions on integers.

In other words, you can choose to provide closure at the level of domain object, or on the functions that manipulate them. LINQ queries of type IQueryable<T> are a good example. You can combine together two queries to get a sequence of T, thus providing domain-level closure. You can also combine together IQueryables to create new IQueryables that also yield sequences of T. That’s functional closure. LINQ is closed on both levels. It’s closed at the level of the entities that it is retrieving, but it’s also closed at the level of the functions it uses to represent queries.

It’s that level of composability that gives LINQ its power. And finding those design principles that we can apply to our own APIs is the purpose of this post. Ponder this: we don’t often provide rules of combination in our object models. If we did, our systems would probably be more flexible. End of SubDigression

Several years ago I produced a graphics library for producing montages in a telepathology app. The system used a scripted generator to produce a tree of graphics operations. Each node on the tree manipulated an image then passed it on to its children. Without an idempotent operation it would have been extremely difficult to add orthogonal operations  (like comms, or microscope operations) or to bind together trees, or to create a default root of an operation graph.

The point of this outer-digression is that there are plenty of cases where at first sight Idempotence seems like architectural overkill. When you have rules of combination you find idempotent operations complete the puzzle making everything just click together. While the idempotent operation does nothing, it creates a framework on which other operations can be composed. Ponder this: Sometimes targeting an architectural abstraction might seem overkill, but if done wisely it can yield great simplicity and flexibility. If you don’t believe this – play with LINQ a little. End of Digression.

If these were properties that only applied to natural numbers under addition or subtraction then they wouldn’t be worth a blog post. It’s the fact that this is a pattern that can be observed in other places that makes them worth my time writing about, and your time reading. Lets stay with integers a bit longer, though:

(3)     2 * 2 = 4

(4)     1 * 2 = 2

You probably noticed right away that the number 1 is idempotent in (4). We could keep multiplying by 1 till the cows come home and we’d always get 2. Now, I’m not setting out to explore the idea of idempotence. The reason I’m mentioning it is that it is an important property of an algebraic system. Closure is another. When you multiply two integers together you get another integer – that’s closure.

Just as addition has it’s inverse in the form of subtraction, so too does multiplication have an inverse in the form of division. Take a look at this:

(5)     4 / 2 = 2

(6)     1 / 2 = 0.5

In (6), the result is not an integer. As an interesting byline – the history of mathematics is littered with examples where new branches of mathematics were formed when non-closed operations were performed that led to awkward results. The process of creating a closed version of an operation’s inverse led mathematicians to create new mathematical structures with new capabilities, thus extending mathematics’ reach. The non-closure of subtraction (the inverse of addition) led to the introduction of the integers over the natural numbers. The non-closure of the division operation (the inverse of multiplication) led to the introduction of the rational numbers over the integers. And the non-closure of the square root operation (the inverse of the power operation) led to the introduction of the irrational numbers. On many occasions through history the inverse of an everyday closed operation has led to the expansion of the space of possible data types. Ponder that – attempting to produce data structures on which the inverses of closed operations are also closed can lead to greater expressivity and power. A bit of a mouthful, that, and probably not universally true, but its something to ponder.

Again, if that were all there were to the idea, I (as a programmer) probably wouldn’t bother to post about it – I’d leave it to a mathematician. But that is not the limit to closure. Closure has been recognized in many places other than mathematics – from physics to philosophy and from API to language design. Lets describe an algebraic system in the abstract to isolate what it means to be closed. The simplest mathematical structure that fits my description is called a Magma:

(7)     A Magma is any set M equipped with a binary function M \times M \rightarrow M

This kind of thing is known to mathematicians as an Algebraic Structure. There are LOTS of different kinds, but that’s one digression I’m not going to make. One thing to notice is that closure is built into this most basic of algebraic structures. What M \times M \rightarrow M means is that if you apply the operation ‘\times ‘ to the two values from M you get another value from M. By that definition, division doesn’t qualify as a Magma if the set M is integers, but it does if the set is the rational numbers.

(8)     2 + 3 + 5 = 10

(9)     (2 + 3) + 5 = 10

(10)     2 + (3 + 5) = 10

Each of these equations demonstrates what is known as associativity. If you add that to the definition of a Magma, you get what is called a semigroup. Integers with addition have that property of associativity, so it counts as a semigroup.

(11)     2 – 3 -5 = -6

(12)     (2 – 3) – 5 = -6

(13)     2 – (3 – 5) = 4

Clearly the subtraction operation on the integers is not associative, so it doesn’t qualify to be called a semigroup.  Try this on for size – associative operations are inherently flexible and composable. Abelson and Sussman even went so far as to say that designing systems with such properties was a better alternative to the traditional top-down techniques of software engineering.

We saw earlier that the property of idempotence means that there may be an element that yields the same value for that operation. If the Magma has an identity property, then it is called a ‘loop’. The point of this is to point out the other properties that operations can have (and how they contribute to membership of an algebraic structure). The key properties are:

  • Closure
  • Associativity
  • Identity
  • Inversibility

I’m going to throw a code snippet in at this point. If you’re a programmer with no particular interest in algebra, you might be wondering what on earth this has to do with programming

var q = from u in MyDataContext.Users
where u.Name.StartsWith("And")
select u;

var q2 = from v in q
where v.Name.EndsWith("rew")
select v;

Here’s an example taken from something like LINQ To SQL. Take a look at the ‘where’ keyword. It is clearly closed, since the application of where to a query yields another query (regardless of whether it gives you any useful results). The example is also associative, since you can reverse the order of the clauses and the resulting set will be the same. LINQ has an identity as well – “.Where(t => t)” which does nothing. LINQ lacks and inversion operation, so you can’t add a clause, then cancel it out with another – instead, if you tried to do that, you’d get no results or everything. Here’s something to ponder – would link be more or less powerful if it had the property of inversibility? It’s clearly possible (though probably extremely difficult to implement).

I started thinking about these ideas because I wanted to understand why LINQ is powerful. It’s flexible and easy to understand because of the  mathematical ‘structure’ of the standard query operations. Ponderable: is any API going to be more powerful and flexible (and less brittle) if it displays some of the properties of an algebraic structure?

What are the advantages of creating APIs that have group structure? Just because we could design an API that has a group structure does not mean that we must. There must be an advantage to doing so. So far I have only hinted at those advantages. I now want to state them directly. If “we can regard almost any program as the evaluator for some language“[r], then we can also regard some languages as a more effective representation of a domain than others.  For many years, I’ve felt that the object oriented paradigm was the most direct and transparent representation of a domain. At the same time, I also felt there was something lacking in the way operations on an OO class work (in a purely procedural approach).

To cleanly transition the state of a class instance to another state, you (should) frequently go to extreme lengths[r] to keep the object in a consistent state. This kind of practice is avoided in those cases where it is feasible to use immutable objects, or more importantly to design your API so that the objects passed around might as well be immutable. Consider a class in C++ that implements the + operator. You could implement the operator in two ways:

  1. add the value to the right to this, and then return this.
  2. create a temporary object, add the value on the right to it and return the temporary object.

The following pseudo-code illustrates the point by imaging a class that supports “operator +”:

MyClass a = new MyClass();
MyClass b = new MyClass();
MyClass c = new MyClass();
MyClass d = a + b + a + c;

If you implement ‘+’ using technique 1 the result in d is (3a + b + 3c) whereas if you implement it using technique 2, the result in d is correctly (2a + b + c). Can you work out where the 3c comes from? The state, being mutable, is modified in a way that is incorrect during the addition operator. The operands of an operation like ‘+’ should be unaffected by the fact that they took part in the assignment of a value to d. Something else to ponder: immutable objects or operations can make it easier to produce clean APIs that work with the language to create a correct answer.

You might complain that what I’m aiming for here is a programming style that uses mathematical operators to implement what would be otherwise done using normal methods. But you’d be missing the point. Whether your method is called ‘+’ or if it’s called ‘AddPreTaxBenefits’ is irrelevant. The structure of the operation, at the mathematical level, is the same. And the same principles can apply.

The method signature of a closed method is T 'times T \rightarrow T. There are plenty of methods that don’t fit this model. Lets pick one that pops into my mind quite readily – bank account transactions:

void Transfer(Account debit, Account credit, decimal sumToTransfer); 

There is an entity in here that does fit the bill for such group like transactions – Money. There are endless complexities in financial transactions between currencies, like currency conversion, exchange rates and catching rounding errors. But the point is that it makes sense to be able to implement group operators on currency values. That ability allows you to define a language of currencies that can be exploited on a higher level item of functionality – the Account. BTW: I’m not talking about the algebraic structure of addition on decimals. I’m talking about adding values of locale specific money values – a different thing.

void Transfer(Account debit, Account credit, Currency sumToTransfer)
{
debit.Balance = debit.Balance - sumToTransfer;
credit.Balance = credit.Balance + sumToTransfer;
}

Would it be better to define the operation on the Account class itself? The operator might actually be updating the internal balance property, but we don’t care about that.

void Transfer(Account debit, Account credit, Currency sumToTransfer)
{
debit = debit - sumToTransfer;
credit = credit + sumToTransfer;
}

Lets take a look and see whether operator ‘+’ fits the criteria we defined earlier for group-like structures:

Closed If you take an account and you add a value to it, you get another valid account, so yes, this is closed.

Associative Yes – though I’m not sure what that would mean in terms of bank accounts. Double entry bookkeeping is kinda binary…

Identity OK – The identity element of the ‘+’ operation on accounts is the zero currency value.

Inverse operation Easy – subtraction. Or the negative currency value. Do you allow negative currency values? that’s incompatible with double entry bookkeeping, so it might not be possible to provide an inverse operator for some systems. There’s an example where trying to get an inverse could lead you to change the design of your system.

This approach passes the criteria, but it also highlights a conceptual layer difference between money types and bank account types that makes for an awkward API if you try to treat them as equivalent. From a design perspective you can see that if there are non-obvious rules about how you can combine the elements of your class, you’re no better off than with a conventional API design. One thing that does occur to me, though, is that the inconclusive group structure here pointed to a mismatch of levels. The addition operator applies at the level of quantities of cash – account balances. Accounts are more than just balances, and attempting to make them behave like they were nothing more than a number highlights the pointlessness of doing so. Ponder this: the concept of ‘levels’ may be something that arises naturally out of the algebraic structure of the entities in a system? I’m not sure about this yet, but it’s an intriguing idea, don’t you think?

Obviously, we could have expected group structure at the level of balances, since we’re dealing with real numbers that are a known group under addition and subtraction. But what about higher level classes, like bank accounts? What are the operations we can perform on them that fits this structure?

I wasn’t sure whether I’d come away with any conclusions from this post, but I did come away with some very suggestive ideas to ponder:

  • Some operations give you guarantees of object validity. As a programmer, you need to learn how to spot them.
  • Some operations are preferable because they guarantee not to change your state.
  • Provide rules of combination in our object models would probably make them more flexible.
  • Sometimes abstraction might seem overkill, but if used wisely it can yield great simplicity and flexibility. If you don’t believe this – play with LINQ a little.
  • Produce data structures on which the inverses of closed operations are also closed can lead to greater expressivity and power.
  • Associative operations are inherently flexible and composable.
  • Maybe all APIs will be more expressive and flexible (and less brittle) if they displays some of the properties of an algebraic structure?
  • Immutable objects or operations can make it easier to produce clean APIs that work with the language to create a correct answer.
  • Trying to get an inverse for an operation could lead you to change the design of your system.
  • The concept of ‘levels’ may be something that arises naturally out of the algebraic structure of the entities in a system.

It’s funny that these ideas flow simply from looking at high-school algebra, especially since some of them read like a functional-programming manifesto. But, hopefully, you’ll agree that some of them have merit. They’re just thoughts that have occurred to me from trying to understand an offhand comment by Eric Mejer about the relationship between LINQ and Monads. Perhaps I’ll pursue that idea some more in future posts, but for now I’ll try to keep the posts coming more frequently.

Søren on DBC

Recently, Søren Skovsbøll wrote a excellent follow up to a little post I did a while back on using C# 3.0 expression trees for representing predicates in design by contract. The conclusion of that series was that C# was inadequate in lots of ways to the task of doing design by contract. Having said that, you can still achieve a lot using serialisation of object states and storage of predicates for running before and after a scope.

Søren was not happy with the format of errors being reported, nor the potential for massive serialisation blowout. Rather than comment on the blog, he went away and did something about it. And it’s pretty good! Go take a look, and then pick up the baton from him. Your challenge is to extract the parmeter objects from the expression trees of the predicates and take lightweight snapshots of the objects refered to. You also need a “platform independent” way to serialize objects for this scheme (i.e. one that doesn’t depend on XmlSerialisation or WCF data contracts.

Think you can do it? Apply here! :P