Code Generation

Polymorphism in C#/.NET Makes Code Incomprehensible

I’ve recently been working on a series of blog posts that require me to reverse engineer the contents of the LINQ assembly System.Query. My experience has largely been very positive, but I have one problem that has slowed my progress – navigating beyond polymorphic method calls. This isn’t a problem with LINQ (it uses polymorphism in a number of key places) it’s a problem with code reading generally.

I find that while abstraction frameworks are essential to allow clean factoring and decoupling of object models, they make interpretation of runtime behaviour very hard – the more fully encapsulated a framework is, the harder it is to read it’s code and deduce how it works.

I don’t have a problem with polymorphism as a rule, well I do, but it’s a price I am more than willing to pay. But I really have struggled to get through the LINQ code base – it is complex enough, without me being hindered at every step by the fact that I have to guess what is going on at runtime for a specific combination of parameters. The struggle for me was keeping it all in my head – the structure of the LINQ expression tree (specifically the Type of each node in the tree), the call stack, the type parameters passed to generic methods, and the progress in iterating collections of those types. It’s hard, and it’s not using everyday idioms so you can’t guess with very much confidence what it ought to be doing. I wonder whether there might be some tool support that I can exploit to ease the code reading process.

If you know of a tool (or some way to augment the debugger) that will allow me to profile the LINQ expression creation and compilation at runtime, please let me know. I’ve tried combinations of Reflector, Hawkeye and dotTrace. None of them were able to penetrate the fog individually, and were impossible to use in combination. Perhaps you know of a way to tackle these issues?

LINQ & Reflection in C# 3.0

I posted an article the other day showing you how to exploit the query
capabilities of LINQ to do reflection over the attributes on a C# class. I want
to show you how to exploit some of the extension methods in System.Query to make
the code even cleaner. I used a method called Members that got all of the
members in order of member type (i.e. fields first, properties next and so on).
The code looked like this:


public static IEnumerable<MemberInfo>Members(this Type t) 
{ 
    foreach (FieldInfo fi in t.GetFields()) 
        yield return fi; 
    foreach (PropertyInfo pi in t.GetProperties()) 
        yield return pi; 
    foreach (MethodInfo mi in t.GetMethods()) 
        yield return mi; 
    foreach (EventInfo ei in t.GetEvents()) 
        yield return ei; 
}


I needed to split the queries into each of the types we required in order to
get elements ordered by type. Well System.Query provides a neater way to do this
sort of thing. As luck would have it, I don't care what order I bring the
members back in so long as they are grouped by type and then by name. We can use
the ordering queries in LINQ to do this:

foreach (MemberInfo mi in from m in t.GetMembers() orderby m.GetType().Name, m.Name select m) 
{ 
    yield return mi; 
}

Much cleaner, and it also grabs the constructors which I forgot to add in the
previous post! ;-) The query syntax there is equivalent to the following C# 2.0
syntax:

foreach (MemberInfo mi in t.GetMembers().OrderBy(a => a.GetType().Name).ThenBy(b => b.Name)) 
{ 
    yield return mi; 
}

I can impose some specific order by using the union method:

foreach (MemberInfo mi in t.GetFields().Union<MemberInfo>(t.GetProperties().Union<MemberInfo>(t.GetMethods()))) 
{ 
    yield return mi; 
}

We can mix and match these operators since they are all extension
methods on the IEnumerable<T> or IEnumerable types. The methods are
defined in System.Query.Sequence, so you can use the object browser or reflector
to find out what's available.

In the code generation patterns posts I wrote last year, I demonstrated that
we often need to recursively enumerate structural elements in assemblies or
databases. Here is an untyped example of how we can apply these enumerators to
simply dig through the capabilities of the type:

public static IEnumerable<object> ParseType(Type t) 
{ 
    foreach (MemberInfo mi in t.Members()) 
    { 
        yield return mi; 
        foreach (object x in mi.Attributes<DbcPredicateAttribute>()) 
        { 
            yield return x; 
        } 
    } 
}

You can see that now the code required to enumerate the type is very simple.
But we haven't gained simplicity at the expense of power. We can explore the
object in any way we please.

Using C# 3.0 For Reflection

C# in Orcas will provide a bunch of really useful tools for those who want to perform tasks involving reflection. The reflection APIs in C# are already excellent but with the new query capabilities provided by LINQ, Reflection will be a real pleasure. There’ve been a few excellent posts on the topic.

In this post I’ll show you how to use the attribute metadata system to filter types using LINQ. In past posts I’ve talked about the Design By Contract (DBC) system I used to generate pre and post conditions for methods on a class. In the post I’ll use Queries, Extension Methods, and iterators to show you how to get types for inserting into the code generation system. Just to recap, here’s how I annotate a class:

[Dbc, Invariant("Prop1 >= 1")] 
public class MyTestClass 
{ 
    public MyTestClass() 
    { 
        Prop1 = 10; 
    }            

    int prop1 = 0; 
    [Requires("value != 1561")] 
    public int Prop1 
    { 
        get 
        { 
            return prop1; 
        } 
        set 
        { 
            prop1 = value; 
        } 
    }            

    [Requires("arg1 > 10")] 
    [Requires("arg2 < 100")] 
    [Ensures("$after(Prop1) == $before(Prop1) + 1")] 
    public void TestMethod(int arg1, int arg2, string arg3) 
    { 
        Prop1 = Prop1 + 1; 
        Debug.WriteLine("MyTestClass.TestMethod.Prop1 == {0}", prop1); 
    } 
}

The class is annotated by a DbcAttribute plus some optional InvariantAttribute attributes. The members are optionally annotated with InvariantAttribute, RequiresAttribute and EnsuresAttribute. The code generator needs to create proxies for any classes that contain any of these attributes, but by convention I wrap only those classes that are adorned with the DbcAttribute to make life simpler. With queries we can do away with the DbcAttribute, but at the cost of having to do more itieration which will affect performance. This will be an issue if you are using the queries for dynamic proxy generation. If you perform static code generation it’s less of an issue. We first need a query to check whether a class has the DbcAttribute.

public static bool HasAttribute(this Type t, Type attrType) 
{ 
    return t.GetCustomAttributes(attrType, true).Count() > 0; 
}

HasAttribute is an extension method that uses the Count() extension method from System.Query to get the number of elements in the collection returned. It just declares if there is more than one of them. Now we can perform a query to get all classes that have the attribute:

private void Example1() 
{ 
    IEnumerable<Type> annotated = 
	from t in GetType().Assembly.GetTypes() 
	where t.HasAttribute(typeof(DbcAttribute)) 
	select t; 
    Console.WriteLine("classes: {0}", annotated.Count()); 
    Console.ReadKey(); 
}

This is now every class that needs to have code generated for it. Now we can enumerate the collection of members of the classes.

private static void Example2() 
{ 
    Type T = typeof(MyTestClass); 
    int count = T.GetCustomAttributes(typeof(DbcAttribute), true).Count(); 
    IEnumerable<Type> annotated = from t in typeof(Program).Assembly.GetTypes() 
                                 where t.HasAttribute(typeof(DbcAttribute)) 
                                  select t; 
    Console.WriteLine("classes: {0}", annotated.Count()); 
    Console.ReadKey(); 
    foreach (MemberInfo mi in GetAllMembers(T)) 
    { 
        Console.WriteLine("member: {0}",mi.Name); 
        DisplayAttributes<InvariantAttribute>(mi); 
        DisplayAttributes<RequiresAttribute>(mi); 
        DisplayAttributes<EnsuresAttribute>(mi); 
    } 
    Console.ReadKey(); 
}            

private static void DisplayAttributes<AttrType>(MemberInfo mi) where AttrType : DbcPredicateAttribute 
{ 
    foreach (AttrType ra in GetAllAttributes<AttrType>(mi)) 
    { 
        Console.WriteLine("tAttribute: {0} ({1})", ra.Predicate, typeof(AttrType).Name); 
    } 
}            

private static IEnumerable<MemberInfo> GetAllMembers(Type t) 
{ 
    foreach (FieldInfo fi in t.GetFields()) 
        yield return fi; 
    foreach (PropertyInfo pi in t.GetProperties()) 
        yield return pi; 
    foreach (MethodInfo mi in t.GetMethods()) 
        yield return mi; 
    foreach (EventInfo ei in t.GetEvents()) 
        yield return ei; 
}            

private static IEnumerable<AttrType> GetAllAttributes<AttrType>(MemberInfo mi) where AttrType : Attribute 
{ 
    foreach (AttrType ra in mi.GetCustomAttributes(typeof(AttrType), true) 
                           .Where(t => t.GetType().Equals(typeof(AttrType)))) 
    { 
        yield return ra as AttrType; 
    } 
}

Now we have iterators for all classes that are adorned with a specific attribute, and likewise for all members of those classes. Each member can have specific attributes iterated as well. That’s nice, but we really need a clean interface to provide iterators over a class.

public static class DbcClassIterator 
{ 
    public static IEnumerable<Type> DbcTypes(this Assembly asm) 
    { 
        return from t in asm.GetTypes() 
               where t.HasAttribute(typeof(DbcAttribute)) 
               select t; 
    }            

    public static IEnumerable<MemberInfo> Members(this Type t) 
    { 
        foreach (FieldInfo fi in t.GetFields()) 
            yield return fi; 
        foreach (PropertyInfo pi in t.GetProperties()) 
            yield return pi; 
        foreach (MethodInfo mi in t.GetMethods()) 
            yield return mi; 
        foreach (EventInfo ei in t.GetEvents()) 
            yield return ei; 
    }            

    public static IEnumerable<AttrType> Attributes<AttrType>(this MemberInfo mi) 
        where AttrType : Attribute 
    { 
        foreach (AttrType ra in mi.GetCustomAttributes(typeof(AttrType), true) 
                      .Where(t => t.GetType().Equals(typeof(AttrType)))) 
        { 
            yield return ra as AttrType; 
        } 
    }            

    public static IEnumerable<AttrType> Attributes<AttrType>(this IEnumerable<MemberInfo> emi) 
        where AttrType : Attribute 
    { 
        foreach (MemberInfo mi in emi) 
        { 
            foreach (AttrType ra in mi.GetCustomAttributes(typeof(AttrType), true) 
                    .Where(t => t.GetType().Equals(typeof(AttrType)))) 
            { 
                yield return ra as AttrType; 
            } 
        } 
    } 
}

This class makes heavy use of Extension methods to allow the comfortable treatment of framework classes for our reflection tasks. Now we can iterate over the collections in various ways.

private static void Example3() 
{ 
    Type t = typeof(MyTestClass); 
    foreach (Type type in t.Assembly.DbcTypes()) 
    { 
        // just get every requires attribute in the type 
        foreach (DbcPredicateAttribute pa in type.Members().Attributes<RequiresAttribute>()) 
        { 
            Console.WriteLine("predicate: {0}", pa.Predicate); 
        } 
        Console.ReadKey(); 
        // get all members and then get all attributes in order. 
        foreach (MemberInfo mi in type.Members()) 
        { 
            Console.WriteLine("member: {0}", mi.Name); 
            mi.DisplayAttributes<InvariantAttribute>(); 
            mi.DisplayAttributes<RequiresAttribute>(); 
            mi.DisplayAttributes<EnsuresAttribute>(); 
        } 
    } 
    Console.ReadKey(); 
}

It’s really pretty neat. It doesn’t do anything we couldn’t do before but it does it in a way that makes it seem as though the framework classes provide all of the features we need from them out of the box. I like that, it makes code much more readable.

Policy and Configuration I

A couple of posts ago, I said I'd explore the issues of code generation, DBC and ORM. The first thing I wanted to do is address the issue of 'policy vs configuration'. Every project has to address this issue in some way, most systems are developed on a platform that varies from its deployment environment. Most software must be tailored to behave differently on different platforms. Typically, a developer environment is different from a production environment, with some resources replaced by mock objects or stubs. The differences have to be communicated to the software and that's where configuration comes in.

I've seen the following strategies employed in the past:

  • Configuration file stored in a well-known location that is loaded at system start-up.
  • Configurations stored in a database.
  • Compile-time settings used to choose between settings (using #if pragmas)
  • Compile-time settings used to do search and replace on regex tokens in the system.

I'm not interested, for the sake of this argument, in the format the settings data uses. Conceptually, it is irrelevant whether the config settings are provided as XML, name-value collections, or serialised objects. The key scenarios are listed above, and they are certainly the most common. More elaborate APIs can be built on top of them, which goes some way towards providing a typed encapsulation to the settings system. .NET provides in-built support for the first most common scenario. The Enterprise Library provides support for serialised objects. It is a great improvement, but has not penetrated far into the standard practice of the .NET community. One can only hope that it will, eventually.

Another common strategy is the use of the hierarchical or cascading config system. I have called it a cascading config system because of its similarity to the mechanism of cascading style sheets. In a cascading config system a default set of properties are initially set up and then overridden selectively with configurations that define how the settings for the current system differ from the default. Typical layouts are just two layers deep, with default settings being loaded first and then instance specific version overloading where appropriate, although I have worked on systems that provide up to five levels of overloading.

Ultimately the provenance of a run-time configuration setting is not as important to the developer as how it is presented to the system (except when it goes wrong). One important factor is how well encapsulated the configuration system is from the vagaries of config fashion. Often, with a long term project where staff turnover is high, you may find that numerous approaches are used leading to a confusion within the team about where to get a setting and how one can override it without bringing down the whole system. Without consistency you may find configs being ignored in favour of hard coded settings because only then can the programmer be sure that the setting is what they need it to be.

Unless the configuration system is clean, well ordered and consistent, code will eventually become brittle, unpredictable, opaque and non-deterministic.

There is another issue with how config systems are generally presented to the rest of the system. Normally, an object-oriented designer would frown at exposing raw data from a data source in an untyped way. It creates points in the system where explicit checks have to be performed to convert types, check formats, check business rules and so on. Configuration code is often not well isolated, it can be diffused throughout the whole system so it will be hard to convert a config system after the fact. The best way to overcome this is to provide a typed wrapper around the configuration settings. Which of these scenarios would you prefer?

string maxRecipients = ConfigManager.GetConfig(MAX_RECIPIENTS);          

if (maxRecipients != null && maxRecipients != ""){ 
  int mr = Convert.ToInt32(maxRecipients); 
  if (mr > 0 && mr < SOME_UPPER_LIMIT) 
    IList someData = GetRecipients(mr); 
  SendTo(someData);

Or something like this:

IList someData = GetRecipients(MailSettings.MaxRecipients); 
SendTo(someData);

Admittedly, I'm taking some artistic liberties here; I'm assuming the MailSettings config class enforces business rules and gets initialised appropriately. The point is that, in our business rules, we should not have to be interested about the format or origin of configuration data. In fact we should not even know that we have a config system! Configuration is a convenience that makes distributed systems and testing easier. Generally it makes development harder for the reasons described above. I've worked with config systems so byzantine that it became a lottery whether unit tests would run consistently between developer machines. That's where the brittleness of configuration arises from. Subtle dependencies arise between security settings, config files, availability and access to resources and run-time identities. Many of these implicit dependencies are in areas that get little or no attention by the development team, often being neglected till the project is in its final user acceptance testing phase. My first contention is therefore that configuration settings should always be typed, and should always be well documented and predictable. This sounds like good advice for all software, and that's true, but config plays a special part in most software systems and thus tends to get neglected in favour of some unencapsulated string based name value collection mechanism.

What does the injunction mean in practice? It means that whatever the format that you store the settings in, you should deserialise them into an object that exposes them as typed properties. In the case of things like connection strings, they require no conversions, but for other types the conversion should be done immediately. In some software systems the config system is so useful and all-pervasive that programmers come to rely heavily on them. They resort to config settings for things that have no place outside of source code. Excessive reliance on config settings can lead to serious performance deficits with continuous type conversions and checking finding their way into every method.

As mentioned before, if you have config settings that are overridden as the platform definition gets more specific, you get a configuration inheritance hierarchy. My next point is: If you are going to provide a typed interface to configuration, why not use explicit object-oriented inheritance hierarchies to encapsulate it? Consider the following model:

public interface IMailSettings{ 
 string MailServer{get;set;} 
 string DefaultFromAddress{get;set;} 
}               

public class MasterMailSettings : IMailSettings{ 
 public virtual string MailServer{get{...}set{...}} 
 public virtual string DefaultFromAddress{get{...}set{...}} 
}               

public class DevMailSettings : MasterMailSettings{ 
 public override string MailServer{get{...}set{...}} 
}

The effect is the same, but the implementation is supported by the language and the idiom is much more harmonious with that of the rest of the system. The same arguments that justify the use of typed data access layers or object relational mapping systems apply here. There are numerous design patterns that can be used to improve performance and consistency. In fact we could describe part of our problem as an 'object/configuration impedance mismatch' to paraphrase Scott Ambler. Normally, we're at liberty to change the format of our config files to make deserialising into typed containers easier, but as with relational systems, people often find it hard to justify expending the up-front effort to produce a system that does this automatically. That's why the config system of the enterprise library is such a welcome addition.

Since in a hierarchical configuration system, default settings are often left unchanged for long periods and get stored as embedded resources, if you want to change them you have to recompile the assembly and restart the application. Aside from leading to a potential violation of service level agreements, this practice undermines the whole point of config files – that of being run-time control of program behaviour. In this circumstance you might as well employ an object-oriented class hierarchy and dispense with default configurations altogether.  I don't think that in 20 years of programming I have ever seen this approach used. Maybe the reason is that people fail to see that polymorphism provides a nice little work-around for overriding default behaviours at run-time. The default behaviour corresponds to the base class (as in MasterMailSettings, above) and the settings in that code could be hard-coded. Yes, hard coded, just like they are in the config files! When relied on excessively, config files become code. And next time, I will show how with policy settings that fact is even more apparent. So, we might have MasterMailSettings look like this:

public class MasterMailSettings 
{ 
    private string mailServer = @"mail.bob.com"; 
    public virtual string MailServer 
    { 
        get { return mailServer; } 
        set 
        { 
            if(CanPingMailServer(value)) 
            mailServer = value; 
        } 
    } 
    private bool CanPingMailServer(string serverAddress) 
    { 
        // ... 
    } 
}

What have we got for this minor piece of heresy? We don't have to maintain a separate config file. We don't have to perform type conversions. It's faster. We can enforce business rules when overriding these values for specific machines. Our code is a lot more comprehensible.  And lastly, there is no framework for our developers to learn to understand what config settings are in play.

But! What about when you really do want to dynamically adjust runtime behaviour? That's when the second level of our config hierarchy comes in. That will deserialise the configurations into a descendant class of the default settings class (as in DevMailSettings, above). Here, the settings can be retrieved from a separate file that is monitor with a file system watcher for example. The client of the config system would ask for an IMailSettings and get a DevMailSettings if on a dev machine. The devmailsettings object would override MasterMailSettings on its MailServer property, but would just return whatever MasterMailSettings defined in all other regards. So, we have encapsulation, run-time control, hierarchical configurations and good performance in the vast majority of cases.

So what should be in the master settings and which should be left for an external file? I guess the answer is – it depends. Does it change between all machines? Is it common to certain classes of machines? Is it modified at runtime? Does it require costly type conversions (such as in DateTime conversions) or does it have complex validation rules? All these things contribute to your final decision, but one thing is for sure. Much less than is commonly configured, needs to be stored in an external file!

Next time I am going to follow this trail a bit further, and distinguish between straight configuration settings, and policy settings. Till then, please post to tell me what you think. I may be way off the mark, but it seems to me that configuration should be kept to an absolute minimum. That's not to say the settings shouldn't be there, just that they shouldn't be externalised.

May was a quiet month

It's been almost a month since the last time I posted, and I have made a few changes to the DBC framework since then. I'm making changes now in readiness for release to GotDotNet or Sourceforge. So I'm just making sure everything is as logical and consistent as I can make it.

Firstly, I have completely rewritten the NVelocity template assembly to simplify it even more (as if that were possible!). Now you can use it like this:

public void TestDefaultCreationOfTemplate()
{
  ITemplate tpl = TemplateManager.CreateTemplate(@"Hello" +
    " ${otherpart}");
  tpl["otherpart"] = "world";
  Assert.AreEqual(tpl.ProcessedOutput, "Hello world",
    "strings didn't match");
}

The observant amongst you will also notice that I'm now using unit test cases from visual studio 2005's unit testing framework – why MS couldn't just use NUnit I don't know (well I do, and it doesn't reflect well on them), especially since their framework seems excessively complicated when using autogenerated tests.
Anyway, the template class is now just a very thin veneer over the velocity framework. I am still able, should the need arise, to swap out NVelocity and replace it with something else, but the changeover was mostly involved in recompilation of NVelocity to .NET 2.0 and to add a strong name for deployment to the GAC.

I've placed the code generation templates into the Core assembly as resources to allow easier deployment and to make them tamper proof and to get rid of the needless configuration settings to allow the system to find the templates.

I've broken out the predicate attributes and the exception types to a separate assembly, so development can be done with DBC attrs even if the DBC system is not in use. It also means the attributes can be put to other uses like documentation or validation.

Right now, I'm mostly in performance tweeking and refactoring mode. Here's an example of one of the tests I'm using that shows how the system is to be used now:


[TestMethod]
public void TestTimeToCreateAThousandProxyObjectPairs()
{
  ArrayList al = new ArrayList();
  DbcProxyDynRTCF cf = new DbcProxyDynRTCF();
  int totalToCreate = 10000;
  cf.CreateProxyFor(typeof(TestObjectWithEvents));
  DateTime start = DateTime.Now;
  for (int i = 0; i < totalToCreate; i++)
  {
    al.Add(cf.CreateProxyFor(typeof(TestObjectWithEvents)));
  }
  TimeSpan durn = DateTime.Now - start;
  double d = durn.TotalMilliseconds / ((double)totalToCreate);
  Debug.WriteLine("Average time to create a proxy-object pair" +
    " (ms): " + d.ToString());
  Assert.IsTrue(durn.TotalMilliseconds < ((double)
    totalToCreate) * .17);
}

Currently the average time needed to instantiate a proxy with a target object is around .2 milliseconds. Interestingly it was around 0.16ms with .NET 1.1, but now I'm working with .NET 2.0 the performance is roughly 25% slower. It may be that some of the techniques I am using can be performed in a better way now, but that remains to be seen — the only deprecated feature that I have not updated is in my use of CreateCompiler to compile the generated code. This piece of code gets called only once, before the timer is turned on. So it couldn't affect the code performance unduly. So where is this much publicised performance boost that we are supposed to get from .NET 2.0?

public CompilerResults GenerateAssembly()
{
  Assert(SourceCollection != null && SourceCollection.
     Count > 0);
  AssertCodeGenParamsAreOK();
  CSharpCodeProvider cscp = new CSharpCodeProvider();
  ICodeCompiler compiler = cscp.CreateCompiler();
  CompilerParameters cp = ConstructCompilerParameters();
  string[] classes = new string[SourceCollection.Count];
  SourceCollection.CopyTo(classes, 0);
  CompilerResults cr = compiler.CompileAssemblyFromSourceBatch
    (cp, classes);

  if (cr.Errors.HasErrors)
  {
    foreach (CompilerError ce in cr.Errors)
    {
      Log(ce.ErrorText);
    }
  }
  return cr;
}

DBC & AOP don’t mix

This time I'm going to explore some of the issues I've found in converting the prototype over to work in a JIT environment. If you've seen any of my previous posts, you'll know that I have a working prototype that uses static code generation to produce proxies to wrap objects that identify themselves with the DbcAttribute. These proxies are produced solely for those objects, so the developer has to know which items have predicates on them in order to instantiate the proxies rather than the target objects. If the programmer forgets, or doesn't know, that an object has predicates, he may create the target object and not benefit from the DbcProxy capabilities.

I've therefore been looking at ways to guarantee that the proxy gets generated if the library designer has used DBC. There are a lot of ways to do this, and having not explored all of them, I want to find a way to construct the proxies that will allow any of these approaches to be employed at run time. That naturally makes me think of abstract class factories and abstraction frameworks. It is nice to allow the library designer to define predicates on interfaces, but that may not fit the design they are working on, so I don't want to mandate it.

The solution I've looking at uses the code injection system that comes with the System.Enterprise namespace's ContextBoundObject classes. This framework guarantees that the object gets a proxy if it has an attribute that derives from ProxyAttributes. I would derive my DbcAttribute from ProxyAttribute and get an automatic request to create a proxy. Sounds perfect? Yes! But as you will see, there is a gotcha that will prevent this approach. Read on to find out more…

Status of the prototype

The prototype uses a Windows console application that is invoked as part of the build process to create the proxies for an assembly. The program is invoked in association with an application config file to provide it with the details of the target assembly, where the resulting proxy assembly is to be stored, and what it's to be called. It has a few other config settings to define how it will name the proxy for a target object, and what namespace it shall live in. So far, I have not added any guidance to it on how to handle the assemblies in a not static way. I haven't implemented any dispenser to intercede in the object creation process. In this blog I'm going to take a look at what config settings I need to add to control the choice between dynamic and static proxy generation.

I'm using the Microsoft Enterprise Application Blocks to handle configuration, and I'm using log4net to take care of logging issues. NVelocity is being used to construct the source code for the proxy. None of that will change in the refactoring that is to follow. In fact, very little has to change in the event model to turn the system into a dual static/dynamic proxy generator. What changes is in the issue of how to handle the prodicates when you are using a dispatch interface. More on that to follow.

static code gen via CSharp compiler

The static code generation process has been described in a quite a bit of detail in previous posts, but in essence it works by scanning through all of the types in an assembly, looking for ones that are adorned wit the DbcAttribute. For each of those, it scans through each of the InvariantAttributes, and then through each of the members in the class to get their details and any predicates that are attached to them. It then creates a proxy class that imitates the target type, except that prior to each invocation it checks a set of pre-conditions, and afterwords checks post-conditions. This proxy type is kept around until all of the assemblies types have been scanned, and then a new proxy assembly is compiled and saved to disk. This Proxy assembly is then used instead of the target type.

The CSharpCompiler provides a set of options to allow you to keep the source code used to create the assembly, and to create debug information to tie the assembly to the source code at run-time. This is useful for debugging the DBC system, but should not be necessary when using the framework in the field. If errors occur they will be on either side of the proxy (assuming I debug the proxies properly) and the user shouldn't want to see the source for the proxy itself. We therefore need to be able to turn this off, along with debug information.

Another feature of the compiler is the ability to dispense an in-memory only assembly that is available for the lifetime of the process. This is just what we need for dynamic proxy generation.

New library for dynamic proxies

One refactoring that is inevitable is that we now need two front-ends to the DBC system. A command line tool for static proxy generation that can be invoked from VS.NET or NAnt. We also need something that can be used in-proc to dispense dynamic proxies on demand. This can't be another process unless you want to use remoting to access your objects. You may want to do that, but you need to have some control of that situation, and it would be a little intrusive to have that decision taken away from you.

Dynamic proxy code generation pattern

The process for the dynamic proxy is much the same as for the static proxy, it just tweaks the compiler settings a little, and provides some speed ups for handling the storage and retrieval of assemblies for types that have already been code generated. The procedure is as follows:

  • client requests an objects of type X
  • creation process is intercepted by the CLR, because the DbcAttributes is declared on the object or one of ins ancestors.
  • The DbcAttributes provides an object that can be used to intercept calls to the object, and it passes the creation of that object off to the dynamic proxy code generator.
  • The dynamic proxy CG checks in a HashTable whether the target type has been requested previously, if so it instantiates the generated proxy and returns.
  • if no type has been generated for this type it creates a scanner and a code generator as per the static proxy CG, and requests the scanner to scan the type.
  • After the type has been scanned, there will be exactly one element in the ProcessedTypes context variable in the CG.
  • The dynamic proxy CG then requests that the CG generate an assembly for the type, specifying that the assembly must be in-memory only.
  • The assembly is generated, and it is stored along with the target type in the HashTable for future reference.
  • The new proxy type is dispensed by the DbcAttribute, and the target object creation commences as normal.
  • Next time a call on the target object is performed the proxy object is allowed to intercept the call, and run its checks.
  • voila!

This seems almost exactly how I want the dynamic proxy to work. The client doesn't have an option but to use it if it is available. This convenience comes at a price. That price is detachment from the state of the target object. If you want to do any interesting kinds of test on the current state of the target object, you need a reference access to the various public members of the type.

Without access to the target instance you can't make pre or post or invariant checks, because you can't access the state to see how it has changed or take snapshots of the initial state. It is in effect unusable. So does that mean that the dynamic proxy is incompatible with DBC?

The problem here is not in the idea of a dynamically generated proxy, it is in the kind of proxy we are using, and in its relationship to the target object. It is not connected to the target object in any way – it could even be on a different machine from the target. Even if it was directly connected (within the enabling framework) it wouldn't be of much use since it is designed to deal in IMessage objects, not target object instances.

The IMessage instance is a property bag that contains name value pairs for each of the parameters that is passed into a single method called "Invoke" which is able to then process the parameters. This pattern will be familiar to those who ever wrote COM components that needed to be compatible with VB – it is a dispatch interface. This sort of thing is great if you want to log the values of parameter calls, or to perform rudimentary checks on the incoming parameters.

We want to go a lot further than that. You can't make many assertions about how an object behaves just by describing its input and output values. And that's the problem that I have with standard interfaces – apart from a name which may or may not mean anything they only have a signature. A signature is just a list of the parameters and return types. Nice to have obviously, but not a lot of value when it comes time to work out what happens inside the implementation(s) of the signature.

So, we need direct access to the target object to be able to go any further. Does that mean that the context bound object approach is dead in the water. I'm not sure and this is fairly sparsely documented area of the framework so I haven't been able to track down an answer to that question yet. Answers on a postcard if you know a definitive answer to this. I am going to assume so for the time being. This hasn't been a wasted time – we've learned a useful fact about the limitations of dynamic proxies – namely that they are not compatible with aspect oriented programming paradigms. If anything they can the end-point of an AOP proxy chain, but they cannot sit anywhere in the middle of a chain of responsibility. It's a depressing fact of life – but useful to know.

What else can we deduce from this? We know that any kind of DBC proxy is going to need access to the state that is somehow deterministic. The key concept is state here. Anything that transforms the state of the target object also changes the semantics. Imagine, for example that the DBC proxy sat at the far end of an AOP chain – it will have an instance to a target object but the target object will delegate all the way down to the real target object at the other end of the chain. All of the state inquiry methods are present, but are converted into IMessage objects and passed through the chain of proxies before arriving at the real target object. The AOP chain can modify the data going into the real target, and modify the results coming out of it. They can also change the timing characteristic making a synchronous call where previously the interface was not, or vice versa. They could transform XML documents or add and subtract values from parameters. The point is, when you are out of touch with the target object, you have no guarantees event hat the parameters you are making tests against are the same as are received by the real target object. You consequently face a situation where the semantics are similar but potentially subtly modified, and you have to track that in your assertions.

This may sound like a rather academic discussion of the potential to modify an interface to have diferent semantics, but that is the essence of the adapter design pattern, and is an occupational hazard with SOA architectures. Don't forget that when you invoke an API on a third party software system, you think you know what is happening, but you often don't. You generally have some documentation that acts as a contract saying what goes on inside the service when the service gets invoked. We all know that such documentation often gets out of date, and we also know that there is little you can do to stop a well-meaning-fool from breaking your system. That's Murphy's law at work again. If something can go wrong, it generally will. And normally it is introduced by someone who thought they were solving some other problem at the time.

So published interfaces aren't worth the paper they are written on, especially if you're playing a game of chinese wispers through layers of other people's code. We need a system that is direct and compulsory, with no middlemen to dilute the strength of the assertions that you make on a component.

Next time I will describe how I solve this problem – probably by paying homage to the Gang Of Four

An interesting proposition

As you will recall from the previous post, I have been wondering about how to implement the functionality for dynamic proxies. Well I saw this great article, by Viji Sarathy, on the web, and think that this might be the way to go.

It uses Context Bound Objects to ensure the interception of all calls by the CLR without any explicit framework setup by the user. We can guarantee that the contracts get followed, whether the user tries to evade them or not.

The only misgiving I have is over the performance characteristics of the thing. Any thoughts?

Code Generation Patterns – Part 3

This time I’m going to describe the source snippet that I posted last time. In previous posts I showed the implementation that I use of the Observer pattern to fire off events notifying the code generator about code structures that it has found. The code generator was shown last time, and you can see that there are a whole bunch of event handlers that take note of the code structures and perform code generation tasks as required.

A typical example is the handler NewType that generated code for classes, structures and interfaces. As you will recall from my last post, I am using depth-first recursion in my scanner, to allow my code generator to generate in one pass. That means that there will be a lot of generated code knocking about from all of the invariant predicates, methods, events, fields and properties that were detected before the type event was invoked.

The first thing the CG does is add the namespace of the type to a list of Namespaces that is being kept for addition at the top of the code for the proxy. Obviously the proxy needs to know about the target object. As a general rule you should use the fully qualified name of any type that you refere to in argument lists, return types etc. Readability is not an issue here (very much) but not having to keep track of all the namespace references that could be needed is worth this stricture.

Following the namespace manipulations the generated code is added to the context variables for the NVelocity template. The generated code is kept in the context vars between lines 10 and 21. For common code structures such as assemblies, types and methods the generated code is added to an ArrayList and when the next most senior CG template needs to include them, it just iterates through the list, inserting everything that it finds. The HashTables are an implementation convenience that helps the CG to maintain a unique list of variables that need to have snapshots taken.

After the CG has added all of the snippets of code that have been generated so far, it invokes the template for the type itself. The template looks like this:

namespace ${type.Namespace}.Proxies{
#region Class ${type.FullName}Proxy
public class ${type.Name}Proxy
{
internal ${type.FullName} TheTargetObject = null;
public ${type.Name}Proxy(${type.FullName} target)
{
TheTargetObject = target;
}

#foreach($method in $methods)
$method
#end
#foreach($property in $properties)
$property
#end
#foreach($field in $fields)
$field
#end
#foreach($event in $events)
$event
#end
} // end class ${type.FullName}Proxy
} // end namespace ${type.Namespace}.Proxies
#endregion

As you can see the template is minimal. All of the interesting work has been done in other templates or “Pragma processors“. I’ll be going into pragma processors in depth next time. Work ahead of me includes making the proxy implement the same interfaces as the target object. In the template above the proxy class is generated with exactly the same name as the target class, but with “Proxy” appended. It has the same namespace with “.Proxies” appended. The proxy stores a reference to the target object called TheTargetObject. TheTargetObject is delegated to when all the predicate checks have been performed. The template for a method is where the action is:

#if(!$utils.IsProperty($method))
#if($utils.IsVoidType($method.ReturnType))
#set($returntype = "void")
// found a void return type
#else
#set($returntype = ${method.ReturnType.FullName})
// didnt find a void return type
#end## if isVoidType

public new $returntype ${method.Name}(#set($comma = "")
#foreach($arg in ${method.GetParameters()})
$comma ${arg.ParameterType.FullName} ${arg.Name}#set($comma = ", ")#end){
// take the 'before' snapshots
#foreach($snapshot in $ssbefore)
$snapshot.TypeName ${snapshot.After} = ${snapshot.Before};
#end

//TODO: Invariant code here
#foreach($inv in $invariants)
$inv
#end
//TODO: Require code here
#foreach($req in $requires)
$req
#end

## now the call the the real object
#if(!$utils.IsVoidType($method.ReturnType))
$returntype result =
#end##if is void

#if($method.IsStatic)
${fullclsname}.${method.Name}(#set($comma = "")
#else
TheTargetObject.${method.Name}(#set($comma = "")
#end
#foreach($arg in ${method.GetParameters()})
$comma${arg.Name}#set($comma = ", ")#end);

// take the 'after' snapshots
#foreach($snapshot in $ssafter)
$snapshot.TypeName ${snapshot.After} = ${snapshot.Before};
#end

//TODO: Ensure code here
#foreach($ens in $ensures)
$ens
#end
//TODO: Invariant code here
#foreach($inv in $invariants)
$inv
#end

#if(!$utils.IsVoidType($method.ReturnType))
return result;
#end##if is void
}#end ## if is not property

The templates are pre-loaded with a “utils” object that is able to perform various jobs related to analyzing types that are beyond the capabilities of NVelocity. We use this to check whether the method is a property or not. Properties in C# are handled as methods in the CLR. If we ignore properties and generate them as methods we lose a lot of the syntactic flexibility that C# makes available.

We need to check then whether the method is void or not. If so, we provide the short form of the void type, since the code will not compile if we use “System.Void”. Not sure why that is the case. Answers on a postcode please!

Having determined what the return type is going to be, we can then create the method signature. We are only ever going to get notified about the public methods, since those are all that external users of the proxy or the target object would ever get to see. We don’t need to wrap protected, private or internal methods so we can hard-code the public access specifier. Next, the method parameters are inserted. They are a complete copy of the arguments of the target objects, but with the type names expanded.

Now we get to the whole purpose of this exercise – to insert predicate checks into the method body to check parameters, members and other features before and after invocation of the method on the target object. the pattern of the invariants is as follows:

  • invariant 1
  • invariant 2
  • invariant …
  • invariant n
  • require 1
  • require 2
  • require …
  • require n
  • TheTargetObject.Method(…)
  • ensure 1
  • ensure 2
  • ensure …
  • ensure n
  • invariant 1
  • invariant 2
  • invariant …
  • invariant n

Invariants are predicates that must be true at all times – they must be true before and after the method has been called. Therefore we bracket the call with the invariant tests, and have requires checks before and the ensure checks after. For more information on these predicates take a look at the language documentation for Eiffel. Since the invariants are invoked before and after every method call you should make sure that they are not too expensive.

At the end of the method template the script looks again to check whether the method returns a void, and if so skips the returning of a result. It also doesn’t bother to collect the result in the first place.

Each of the methods is built this way, converted into a string and then appended to the end of the ProcessedMethods properties in the CG. When the type event is finally emitted, the ProcessedMethods property contains a wrapper for each of the methods that the scanner found. That should be every public method that the target object implements or inherits. Obviously, the invariant properties of the object must be true for all properties, fields, events and methods, so it is not enough just to wrap the methods that have Require and Ensure attributes attached, since the Invariant attributes apply for the whole type at all times.

Next time I’ll show you how I convert the assertions made in the invariant, require and ensure attributes into working code that can be inserted into the proxy. I’ll leave you with this exercise – How would you convert this into a peice of working code. I’ll show you how I do it. Let me know if your approach is better, because I’d be glad to use it!

Code Generation Patterns – Part 2

Last time I described typed and un-typed events, and multicasting events to several listeners. One of those listeners would be a code generator. This time I'll go on to describe some of the necessary infrastructure that such a design requires. Generally this information is all about tieing the events together. The context in which an event takes place is as important as the event itself.

When you're generating code progressively, you need to keep track of what you have done, and what still needs to be done. In this case that means the code already generated, and the messages already received, but have not generated code for. It also indicates what code structure(s) an incoming predicate applies to.

There are two complementary modes of operation that a scanner/code-generator system can use for broadcasting events. They are variations of tree search algorithms. In the first case, the most senior element is broadcast first. that means that an event would be generated for a type before the events for its methods are fired. I shall call this (for want of a better name) the "depth-last recursive search". The second approach is a true "depth-first search" since elements that are lowest in the object tree are announced sooner. These two modes support different code-generation plan. The choice will have an effect on what sort of state one has to keep, and how long it has to hang around. More on that later.

With depth-first recursion a method event will be broadcast before the corresponding type event , and a predicate such as an Ensure attribute, will be received before the method on which it was attached. Therefore, when you are generating code, you can't just slot what you get into what you already have, because you don't have anywhere to put it.

An object tree. Depth means further from the assembly object.

With a depth first search context variables track the predicates detected till an event for the method is broadcast and we can generate the whole code for the method. We still have to keep the generated code around till we get the type event! Needless to say, we could hunt for such information as the method event arrives. We could use reflection to navigate up the type object graph as far as the assembly if we wanted to. But if we rely too much on that sort of navigation we break the whole broadcast idiom, and morph the algorithm into a depth-last recursive search (which has it's own unique requirements that I'll come onto next).

In the depth-last search the event for the more senior object is fired before that of its subordinates. That means we get the event for the type before that of the method, and the event for the method before that of its predicates. That's helpful because we now have something to hang the subsequent events off of. If you were producing a whole object graph in memory then this would be ideal, since the tree would always be properly connected rather than fractured. This approach is not without its drawbacks, not least of which is that you have to build up an object graph in memory before you can generate the code for it! With depth-first recursion you know that when you get the event for the method that there are no more predicates coming. You know when it's safe to generate code. With the depth last approach you have to send a "finish" message that says when the stream of sub-objects has stopped. On the whole I've found for this project that depth-first navigation of the assembly object graph works fine, and simplifies the event interface of the listeners that I want to use. In other projects I've used this on I've done the opposite, and everything has gone OK, it really depends on the sizes of the data stream, and the performance characteristics required. There are drawbacks with either approach and you should probably decide on the basis of the following criteria:

  • Is the stream of events large (or never-ending)
  • Do you need to keep the resultant data structures around in memory
  • Would single-pass processing be an advantage to you.

The snippet below shows some of the code that I use to process events in the code generator. Much has been chopped out of course, but from this you should see how all the talk about depth-first searches and events translates into code generation.

public class DbcProxyCodeGenerator : DbcSupertype
{
    public DbcProxyCodeGenerator()
    {
        InitialiseTemplates();
    }

    #region Context Variables

    private ArrayList processedAssembly = null;
    private ArrayList processedTypes = null;
    private ArrayList processedMethods = null;
    private ArrayList processedProperties = null;
    private ArrayList processedFields = null;
    private ArrayList processedEvents = null;
    private Hashtable processedSnapshotsBefore = null;
    private Hashtable processedSnapshotsAfter = null;
    private ArrayList processedInvariants = null;
    private ArrayList processedEnsures = null;
    private ArrayList processedRequires = null;
    private Hashtable processedNamespaces;

    public void NewAssembly(object sender,
  NewAssemblyEventArgs e)
    {
        vtAssembly.SetAttr("assembly", e.TheAssembly);
        string[] namespaces = new string[
  ProcessedNamespaces.Keys.Count];
        ProcessedNamespaces.Keys.CopyTo(namespaces, 0);
        vtAssembly.SetAttr("namespaces", namespaces);
        vtAssembly.SetAttr("types", ProcessedTypes);
        ProcessedAssembly.Add(vtAssembly.Merge());
    }

    public void NewType(object sender, NewTypeEventArgs e)
    {
        ProcessedNamespaces.Add(e.TheType.Namespace, null);
        vtType.SetAttr("type", e.TheType);
        vtType.SetAttr("methods", ProcessedMethods);
        vtType.SetAttr("fields", ProcessedFields);
        vtType.SetAttr("properties", ProcessedProperties);
        vtType.SetAttr("events", ProcessedEvents);
        ProcessedTypes.Add(vtType.Merge());
        ProcessedMethods = null;
        ProcessedFields = null;
        ProcessedProperties = null;
        ProcessedEvents = null;
        ProcessedInvariants = null;
    }

    public void NewMethod(object sender, NewMethodEventArgs e)
    {
        vtMethod.SetAttr("method", e.Method);
        vtMethod.SetAttr("invariants", ProcessedInvariants);
        vtMethod.SetAttr("requires", ProcessedRequires);
        vtMethod.SetAttr("ensures", ProcessedEnsures);
        ArrayList beforeSnapshots =
  SnapshotProcessor.GetBeforeSnapshots
           (e.Method as MemberInfo, ProcessedSnapshotsBefore.Keys);
        ArrayList afterSnapshots =
  SnapshotProcessor.GetAfterSnapshots
           (e.Method as MemberInfo, ProcessedSnapshotsAfter.Keys);
        vtMethod.SetAttr("ssbefore", beforeSnapshots);
        vtMethod.SetAttr("ssafter", afterSnapshots);
        ProcessedMethods.Add(vtMethod.Merge());
        ProcessedEnsures = null;
        ProcessedRequires = null;
    }

    public void NewInvariantAttribute(object sender,
  NewInvariantAttributeEventArgs e)
    {
        EnsureSpecification es = DbcPragmaProcessor.
  ProcessEnsure(e.Invariant.Predicate);
        SnapshotProcessor.RegisterSnapshots(es,
  ref this.processedSnapshotsBefore,
  ref this.processedSnapshotsAfter);
        vtInvariant.SetAttr("invariant", es);
        ProcessedInvariants.Add(vtInvariant.Merge());
    }

    public void NewEnsuresAttribute(object sender,
  NewEnsuresAttributeEventArgs e)
    {
        EnsureSpecification es = DbcPragmaProcessor.
  ProcessEnsure(e.Ensure.Predicate);
        SnapshotProcessor.RegisterSnapshots(es,
  ref this.processedSnapshotsBefore,
  ref this.processedSnapshotsAfter);
        vtEnsure.SetAttr("ensure", es);
        ProcessedEnsures.Add(vtEnsure.Merge());
    }
}

Iterative vs final expansion of templates

A seemlingly simple decision like the order in which you notify interested parties leads to significant performance issues that must be known about in advance. To illustrate this, consider the following scenario:
I want to use this framework in a number of different scenarios. I want to be able to statically generate the code for a proxy object that I then use as though it were the real object. That proxy object will enforce all of my rules at runtime. Projects that use my proxies must include the proxy assembly as well as the assembly of the target object. This is the simple case, I also need to be able to dispense with that and use dynamic proxies. Thus situation is one where I use dynamic code generation at runtime. To do that I need to request access to the object through a class factory that will perform code generation of the proxy on-the-fly. In other situations I want the framework to dovetail with aspect oriented programming frameworks like ACA.NET. In that situation the generated code could be either static or dynamic, but the specifics need to abstracted into a configuration file.

As you can see from these requirements our needs can swing between static and dynamic code generation, and we may even want to use both in the same program. Performance must take precedence if we want to use dynamic code generation. Static code generation won't suffer over much if we choose a code generation strategy that is biased towards dynamic code generation since its costs are at compile time, and won't affect runtime performance.

Code Generation Patterns – Part 1

This time I'm going to show you the design pattern that I use for code generation. Generally I use this for analysis of hierarchical structures, such as assemblies or database metadata. You could use it in other scenarios, where data is less structured. If you are familiar with the operation of a SAX parser and .NET events then you'll have no trouble understanding the design.

SAX-like Parser

A SAX parser works by iterating through its material (in this case XML, but that's immaterial) firing off events for everything that it finds. It doesn't do anything with what it finds, it just acts as an event generator that embodies some knowledge of the internal structure of the incoming material. It is up to whoever is listening to the events to do something with them. In the case of a SAX parser an XML DOM tree is probably built up, but not necessary, and that is the point. You have the option to tailor your processing of the sequence of events according to your requirements.

event scanner & generator

In my design pattern the objective is to decouple event generation from event handling. As far as the event handler is concerned, it is the order of events and their content that matters, not where they originated from. Likewise for the scanner it is not important how the data is handled, only that it navigates through the data properly. So we decouple and encapsulate the two aspects of the process and enable the enhancement and extension of the system easily.

multicast events

The use of events means that you can multicast events frm the scanner to as many handlers as you need.

typed events vs overloaded events

A natural question that occurs to me when I liken the pattern I use for code generation with that of a SAX parser is why I use a plethora of events with typed parameters rather than an overloaded interface with a single event and event parameter type. With SAX events come in many shapes and sizes and are sent out through a single event interface. In my design I have chosen an interface and a set of events that represent each of the possible entities that we are interested in. The interface for the assembly scanner looks like this:

In a previous project I used the delegate/event interface as a common language between a set of components of an ORM system. The data initially made its way into the system through a reverse engineering database metadata schema reader. Apart from being a mouthful to say, it read the structure of a SQL Server database, and generated a set of events that were multicast in the ways I've described above. One of the listeners built an in-memory model of the schema. Later on we needed to persist this model to disk as a configuration file that would be used at runtime as the class map of the ORM (if this intrigues you, goto Scott Ambler's web site for some very influential papers). So when the ORM service started up it loaded the configuration file and read through that firing off exactly the same events. The same listener was stationed on the other end of the event interface, so I was able to guarantee that the class map model was brought into memory in the same way as was used for the code generation cycle. My point is that this typed event interface acts as a common language between components participating in the data's lifecycle. If provided a natural depth first search model for passing data about in a way that allowed on the fly processing. It also allowed a complete decoupling of the data traversal mechanism from the data handling, the sequence of events was all that mattered to my code generation system, not the format on disk or even in memory – for all it cared the data could be stored as metadata in an RDBMS or nodes in an XML file or DOM. The decoupling is total, but not at the expense of vagueness as can be the case with catch all interfaces.

public class AssemblyScannerEventNotifier : MarshalByRefObject { public delegate void NewAssemblyHandler(object sender,   NewAssemblyEventArgs e); public event NewAssemblyHandler newAssembly; public void NotifyNewAssembly(Assembly assembly) { NewAssemblyEventArgs e = new NewAssemblyEventArgs(assembly); if (newAssembly != null) newAssembly(this, e); } }

The EventArg object simply carries the assembly that has been found.

public class NewAssemblyEventArgs : EventArgs { private Assembly theAssembly = null; public Assembly TheAssembly { get{return theAssembly;} } public NewAssemblyEventArgs(Assembly assembly) { theAssembly = assembly; } }

What this allows is for us to multicast the event to a number of different interested parties. Generally that will be a small set including the code generator or some intermediate party, plus the test harness to provide a progress bar.