Further down the fluent IBatis road

My design is only loosely following the original iBatis specification, but at this point, I have built a syntax that you can use for end-to-end data mapping.
As usual, the unit test shows the example:

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Data.SqlClient;
using System.Linq.Expressions;
using FluentIbatis.Core;
using NUnit.Framework;

namespace FluentIbatis.UnitTests
{
    [TestFixture]
    public class DBActionTests
    {
        /// <summary>
        /// dummy class
        /// </summary>
        public class Employee
        {
            public int ID { get; set; }
            public string Name { get; set; }
            public DateTime LuckyDay { get; set; }
            public object UnmappedObject { get; set; }
        }
        [Test]
        public void TestBasicFunctionality()
        {
            var connnectionString = ConfigurationManager.ConnectionStrings["default"].ConnectionString;
            using (var conn = new SqlConnection(connnectionString))
            {
                IList<Employee> employees = DBAction<Employee>
                    .CreateCall("spGetEmployees")
                    .WithMapping(
                        new List<Expression<Func<Employee, object>>>
                            {
                                x => x.ID,
                                x => x.Name,
                                x => x.LuckyDay
                            }
                    ).GetAll(conn);
            }
        }
    }
}

Fluent IBatis

Fluent NHibernate is a killer framework when you are working with a simple straightforward database schema. Its syntax makes ORM configuration trivial work in comparison to writing out the XML files.

I've come across plenty of scenarios where I need a clean ORM, but the database is far from straightforward:

• Objects do not correspond to a single table
• Objects are saved using a stored proc, loaded using an amalgam of views/procs/functions
• The same object is loaded and saved in the database in multiple ways depending on the use case.

Unfortunately, fluent NHibernate isnt geared for these kind of scenarios. In these kind of cases, we want a clean ORM that gives us tight control over database interaction, possibly at the sacrifice of simple session management.

IBatis has been around possibly longer than Hibernate, it is a simple ORM that shines in these kind of scenarios. Its an old framework, it hasnt been changed in years, but thats because it already works perfect for its design. The main drawback is the XML configuration files.

I'm working on updating a subset of the IBatis framework related to mapping.

The following code below demonstrates using fluent syntax to map a datatable to an object collection. I've done the same for the System.Data.IDataReader interface. My next step is to build out a fluent syntax that covers most of the iBatis XML configuration files.

If you want to lend a hand, I've posted my code on github:

git://github.com/pweissbrod/FluentIBatis.git

using System;
using System.Collections.Generic;
using System.Data;
using FluentIbatis.Core;
using NUnit.Framework;

namespace FluentIbatis.UnitTests
{
    [TestFixture]
    public class PresentationTests
    {
        /// <summary>
        /// dummy class
        /// </summary>
        public class Employee
        {
            public int ID { get; set; }
            public string Name { get; set; }
            public DateTime LuckyDay { get; set; }
            public object UnmappedObject { get; set; }
        }

        public class DataSources
        {
            public static DataTable GetBasicValidDataTable()
            {
                var result = new DataTable();
                result.Columns.Add("ID", typeof(int));
                result.Columns.Add("Name", typeof(string));
                result.Columns.Add("LuckyDay", typeof(DateTime));
                result.Rows.Add(1, "Pete", DateTime.Now);
                result.Rows.Add(2, "Dave", DateTime.Now);
                result.Rows.Add(3, "Jeff", DateTime.Now);
                return result;
            }
        }

        /// <summary>
        /// mapping for the dummy class
        /// </summary>
        public class BasicValidEmployeeClassMap : ClassMap<Employee>
        {
            public BasicValidEmployeeClassMap()
            {
                Map(x => x.ID, "ID");
                Map(x => x.Name);
                Map(x => x.LuckyDay);
            }
        }

        [Test]
        public void TestBasicFunctionality()
        {
            DataTable dataTable = DataSources.GetBasicValidDataTable();
            IList<Employee> employees = ObjectBuilder.Map(new BasicValidEmployeeClassMap(), dataTable);
            Assert.AreEqual(employees.Count, 3);
            Assert.AreEqual(employees[0].Name, "Pete");
            Assert.AreEqual(employees[0].ID, 1);
            Assert.AreEqual(employees[0].LuckyDay.Date, DateTime.Now.Date);
            Assert.AreEqual(employees[0].UnmappedObject, null);
        }
    }
}

Rapid keyboard-friendly grid with jQuery

I try to avoid using a mouse when possible. The single pointer feels ham-handed once you practice keyboard shortcuts. I have yet to get my grubby fingers on a solid multi-touch OS, but in the mean time keyboard shortcuts make a big difference in a program.

Keyboard shortcuts on a web page are a relatively new concept at the time of this writing. Not sure why, because the jQuery js-hotkeys plugin makes it an easy feature to fulfill.

People often need to sift through grids of information to drill down to the information they need. There are many jQuery grid plugins, but after extensive research I've found DataTables (not jqGrid) to be the pick of the litter.

Add keyboard shortcuts to the datatables plugin and I've got the fastest grid in the west**, with the following features:

• Mouse-free searching, paging, filtering, and row selection

• Server-side searches, client side sorting, filtering, and paging

• Plugs in to jQuery UI "theme rolling" for programmers like me that are css-challenged.

**this is a read-only grid. I assume that you will select a row and navigate to a "details" page to perform edits.

Demo

Example code

Some notes:

• To perform the on-demand paging, I added an extra js file "jquery.dataTables.ext.js" which adds a new function to the dataTables plugin to make paging forward/backward possible.
• Use caution with IE and keyboard shortcuts because sometimes conflicting shortcuts (such as CTRL+S) will trigger the IE action and short circuit your custom action

An example of forcing unwanted features: my searching is broken

I have actually run into this problem so many times now that I feel compelled enough to ask for help on how to fix it.

I dont want indexing service. It slows down my machine and locks files against my will. Its file locking behavior causes elusive time-consuming build errors. It slows my dual-core machine down at times I do not want it to run. Installing office 2007 means you invite a newer, more annoying breed of indexing service into your machine.

Here is how you search for a file in linux:

>find . -name '*.xsd'

or

>grep *.xsd \.*

Here is how you find a file in windows with MS office 2007:

Click on the folder, choose search. You will see a new window pop up similar to the screen below, which basically says this window is of no use to you, click on the happy dog in the lower left corner to continue, and then you finally get to a search screen that allows rudimentary regular expressions.

And here is the full source code for the ideal unix-style command line grep written in Boo:

namespace Grep
import System
import System.IO
import System.Text.RegularExpressions

def GetFiles(dir as string, pattern as string) as string*:
folders = Directory.GetDirectories(dir)
for folder in folders:
for file in GetFiles(folder, pattern):
yield file
for file in Directory.GetFiles(dir, pattern):
yield file

if argv.Length == 2:
print "Usage: grep [file pattern] [file content regex]"
filePattern = argv[0]
textPattern = Regex(argv[1])

for file in GetFiles(Environment.CurrentDirectory, filePattern):
using sr = StreamReader(file):
while line = sr.ReadLine():
if textPattern.IsMatch(line):
print file, ":", line

Is it too much to ask to just search my disk without having to index my entire hard drive? Somewhere I envision the talking microsoft paperclip laughing ominously.

(currently using my own grep function until I find a better way)

-pete the curmudgeon

**Edit** I found a DOS command which is more to my liking (and a ton faster).

Dos to the rescue! It turns out the find functionality I want was right under my nose this whole time. To search for a specific file, use the dir command:

>dir *.xsd /s

The "/s" switch will recursively scan subdirectories.

To search the file contents, just use the FIND command:

>find "filesubstring" *.txt

The only problem now is that the FIND command is not recursive. So apparently the only thing similar to a recursive grep in DOS is a command like the following:

C:\>for /F "usebackq" %i in (`dir /b /s *.txt`) do find /n "filesubstring" %i 

Thats a mouthful! after some more tinkering around I made a batch file that does precisely the kind of finding function I was looking for. Here is the batch file contents:

for /F "tokens=*" %%i in ('dir /b /s %1') do find /n /i %2 ^"%%i^"   

Here is how I call it:

C:\>rgrep.bat *.txt "substring"

Now, to make a GUI with a happy dog on it, and submit a patch to microsoft :)

Use Excel to generate configuration code

In the past few projects I found myself depending on Microsoft Excel as a mechanism to turn requirements/specs into code, and it has paid off on some level every time I use it.

Excel is a great specification format

• It has better clarity than a Microsoft Word for outlining hard spec details
• Excel is pretty universal, which makes it a convenient medium of communication between you and your users
• Defining configurations and specs in excel format is a great way to implicitly set forth conventions of how the program behaves, which makes inner-workings a little more transparent to anyone who understand the excel sheet
  

What can an Excel sheet buy you in terms of code generation?

• It is a transparent, direct connection between the business meaning of the configuration and the code that represents it. Code is GUARANTEED to represent the specs without any chance of a typo or miscommunication
• Excel is powerful. If you dont see the functions you need for your formula, make your own in the VB editor
• Clients like to have supplementary tools that streamline maintenance and avoid anything that looks like code. This is a value-add in their eyes

How do you get started?

1. Think of the target code/syntax you wish to generate. It could be part of a web.config file, or some mundane C# implementation, or IOC configuration such as windsor XML or a StructureMap class.  
2. Design the layout of the Excel sheet so it asks the right questions. How can you represent the meaning behind the code in a transparent fashion in excel?
3. Connect the dots by writing excel formulas!

Here are some scenarios where excel generation paid off for me:

The problem:

The application has a handful of functional resources, and multiple user groups with different access to each resource. The access seldom changes, and they want to define access provisioning inside of an XML file as opposed to creating an admin console.

The target generated code:

An XML document containing a node for each resource containing a node for each user group for the given resource.

The excel sheet design:

Create a matrix consisting of the functional regions on one axis, and user groups on the other axis. By marking an X in the matrix,  the configuration should allow the group/resource. With a simple excel macro, I wrote a formula that reads the matrix for each row/column combination and yields all of the code for the XML file. Now all we need to do is paste into the XML config file and we are done.

The problem:

A complex ETL process depends on multiple data sources of various formats ranging from a text file to an ODBC connection, to a SharePoint list. Each data source must be properly configured with all of its location, credentials, and data access syntax. Order of precedence dictates the sequential order of the import. Data sources and their connection information are numerous, and in a state of flux. Connection configuration cannot contain errors.

The target generated code:

The goal was to generate the "appsettings" section of an app.config file with all of the necessary connection information to let the code run.

The excel sheet design:

I asked the users to give me an excel sheet with a row for each data source in the order of dependency. The columns should contain information such as data source name, type, UNC, URL, username, password, comments for each data source. Certain fields were optional/mandatory depending on the data source type (eg URL is necessary for a sharepoint list, but not for an ODBC connection). The formula yields a clean app.config every time the data sources changed or were updated. The value is one less step in troubleshooting faulty configurations.

Useful HtmlHelper Extensions for MVC.NET Beta

In the past day I have been digging into the ASP.NET MVC source code trying to find the best ways to spot-weld on some of the features I found missing. It was frustrating to find numerous blog examples that were deprecated, and vital parts of the MVC framework that were INTERNAL. 

Fortunately, that pain is over and now the fun is just getting started.

For a warm-up, I made and Html.ActionButton. This is the same thing as Html.ActionLink in the form of a button.

Calling this:

<%=Html.ActionButton("Foo","Index","Admin",new{Bar=1}) %>

Yields this:

<button onClick="location.href='/Admin?Bar=1'">Foo</button>

Next, I made the Html.ActionImage, for those times you want an ActionLink to appear in the form of an image.

Calling this:

<%=Html.ActionImage("~/Content/images/package.png","Index","Admin",new{Bar=1}) %>

Yields this:

<a href="/Admin?Bar=1"><img src="/Content/images/package.png"/></a>

Finally, here is a security-aware ActionLink. This link will only be enabled/visible to users with authorization to the specified controller/action. This is a simple solution for security trimming in MVC.NET. The code comes primarily from this blog post. It wasnt forking for me, but I modified it slightly by adding a typpe parameter to the method.

Here is the syntax:

<%=Html.SecurityTrimmedActionLink<AdminController>("Foo","Index","Admin") %>

To pull this off, I needed to re-write two utility functions that were hidden/internal in the MVC framework. This method converts a virtual path into the correct physical path (like Url.Resolve method):

public static string ResolveUrl(this HtmlHelper helper, string virtualUrl)
     {
         if (!virtualUrl.StartsWith("~/"))
             return virtualUrl;

         virtualUrl = virtualUrl.Remove(0, 2);

         string applicationPath = helper.ViewContext.HttpContext.Request.ApplicationPath;
         if (string.IsNullOrEmpty(applicationPath) || !applicationPath.EndsWith("/"))
         {
             applicationPath = applicationPath + "/";
         }

         return applicationPath + virtualUrl;
     }

And here is a way to get the proper URL for a controller/action and value dictionary without access to the internal UrlHelper methods by using reflection:

public static string GenerateUrl(HtmlHelper htmlHelper, string actionName, string controllerName, object values)
     {
         var urlHelper = new UrlHelper(htmlHelper.ViewContext);
         Type t = typeof (UrlHelper);
         Object[] paramArray = {null, actionName, controllerName, new RouteValueDictionary(values)};
         MethodInfo m = t.GetMethod("GenerateUrl", BindingFlags.NonPublic | BindingFlags.Instance);
         var s = (string) m.Invoke(urlHelper, paramArray);
         return HttpUtility.HtmlAttributeEncode(s);
     }

Given these two methods makes the rest of the code relatively easy. Here is the code for all of the helper methods above.

public static class HtmlHelperExtensions
  {
      /// <summary>
      /// Generates an actionlink in the form of a button
      /// </summary>
      /// <param name="htmlHelper">The HTML helper.</param>
      /// <param name="buttonText">The button text.</param>
      /// <param name="actionName">Name of the action.</param>
      /// <param name="controllerName">Name of the controller.</param>
      /// <param name="values">The values.</param>
      /// <returns></returns>
      public static string ActionButton(this HtmlHelper htmlHelper, string buttonText, string actionName, string controllerName, object values)
      {
          string link = SystemWebMVCUtils.GenerateUrl(htmlHelper, actionName, controllerName, values);
          link = string.Format("<button onClick=\"location.href='{0}'\">{1}</button>", link, buttonText);
          return link;
      }

      /// <summary>
      /// Generates an actionLink in the form of an image
      /// </summary>
      /// <param name="htmlHelper">The HTML helper.</param>
      /// <param name="imageUrl">The image URL.</param>
      /// <param name="actionName">Name of the action.</param>
      /// <param name="controllerName">Name of the controller.</param>
      /// <param name="values">The values.</param>
      /// <returns></returns>
      public static string ActionImage(this HtmlHelper htmlHelper, string imageUrl, string actionName, string controllerName, object values)
      {
          string link = SystemWebMVCUtils.GenerateUrl(htmlHelper, actionName, controllerName, values);
          link = string.Format("<a href=\"{0}\"><img src=\"{1}\"/></a>", link, htmlHelper.ResolveUrl(imageUrl));
          return link;
      }
    
      /// <summary>
      /// Generates an actionLink based on role/permissions of the currently logged in user.
      /// </summary>
      /// <param name="htmlHelper">The HTML helper.</param>
      /// <param name="linkText">The link text.</param>
      /// <param name="action">The action.</param>
      /// <param name="controller">The controller.</param>
      /// <returns></returns>
      public static string SecurityTrimmedActionLink<Controller>(this HtmlHelper htmlHelper, string linkText, string action, string controller)
      {
          return SecurityTrimmedActionLink<Controller>(htmlHelper, linkText, action, controller, false);
      }

      /// <summary>
      /// Generates an actionLink based on role/permissions of the currently logged in user.
      /// </summary>
      /// <typeparam name="Controller">The type of the Controller.</typeparam>
      /// <param name="htmlHelper">The HTML helper.</param>
      /// <param name="linkText">The link text.</param>
      /// <param name="action">The action.</param>
      /// <param name="controller">The controller.</param>
      /// <param name="showDisabled">if set to <c>true</c> [show disabled].</param>
      /// <returns></returns>
      public static string SecurityTrimmedActionLink<Controller>(this HtmlHelper htmlHelper, string linkText, string action, string controller, bool showDisabled)
      {
          if (IsAccessibleToUser(action, controller, typeof(Controller)))
          {
              return htmlHelper.ActionLink(linkText, action, controller);
          }
          else
          {
              return showDisabled ? String.Format("<span>{0}</span>", linkText) : "";
          }
      }

      /// <summary>
      /// Determines whether [is accessible to user] [the specified action authorize].
      /// </summary>
      /// <param name="actionAuthorize">The action authorize.</param>
      /// <param name="controllerAuthorize">The controller authorize.</param>
      /// <param name="controllerType">Type of the controller.</param>
      /// <returns>
      ///  <c>true</c> if [is accessible to user] [the specified action authorize]; otherwise, <c>false</c>.
      /// </returns>
      public static bool IsAccessibleToUser(string actionAuthorize, string controllerAuthorize, Type controllerType)
      {
          var controller = (IController) Activator.CreateInstance(controllerType);
          var controllerAttributes = new ArrayList(controller.GetType().GetCustomAttributes(typeof (AuthorizeAttribute), true));
          var actionAttributes = new ArrayList();
          MethodInfo[] methods = controller.GetType().GetMethods();
          foreach (MethodInfo method in methods)
          {
              object[] attributes = method.GetCustomAttributes(typeof (ActionNameAttribute), true);
              if ((attributes.Length == 0 && method.Name == actionAuthorize) || (attributes.Length > 0 && ((ActionNameAttribute) attributes[0]).Name == actionAuthorize))
              {
                  actionAttributes.AddRange(method.GetCustomAttributes(typeof (AuthorizeAttribute), true));
              }
          }
          if (controllerAttributes.Count == 0 && actionAttributes.Count == 0)
              return true;

          IPrincipal principal = HttpContext.Current.User;
          string roles = "";
          string users = "";
          if (controllerAttributes.Count > 0)
          {
              var attribute = controllerAttributes[0] as AuthorizeAttribute;
              roles += attribute.Roles;
              users += attribute.Users;
          }
          if (actionAttributes.Count > 0)
          {
              var attribute = actionAttributes[0] as AuthorizeAttribute;
              roles += attribute.Roles;
              users += attribute.Users;
          }

          if (string.IsNullOrEmpty(roles) && string.IsNullOrEmpty(users) && principal.Identity.IsAuthenticated)
              return true;

          string[] roleArray = roles.Split(',');
          string[] usersArray = users.Split(',');
          foreach (string role in roleArray)
          {
              if (role == "*" || principal.IsInRole(role))
                  return true;
          }
          foreach (string user in usersArray)
          {
              if (user == "*" && (principal.Identity.Name == user))
                  return true;
          }
          return false;
      }
  }

Asp.NET MVC - Attempts to extend HtmlHelper making ActionButton instead of ActionLink

The goal:

I want to make an Html.ActionButton class as an alternative to Html.ActionLink.

I want a button that routes to a controller instead of a hyperlink. You know, its like a form button with a GET action instead of a POST.

Currently, if you write:

<%=Html.ActionLink("Delete","Delete", new {ID=package.ID})%=>

Then the templating engine will yield:

<a href="/Delete?ID=40">Delete</a>

And I really want:

<button onClick="location.href='/Delete?ID=40'>Delete</button>

MVC is open source code, so it should be a piece of cake, right?

Looking at the ActionLink source code (inside of System.Web.Mvc.Html) I see:

public static string ActionLink(this HtmlHelper htmlHelper, string linkText, string actionName, RouteValueDictionary values, RouteValueDictionary htmlAttributes) {
    ...
       return htmlHelper.GenerateLink(linkText, null /* routeName */, actionName, null /* controllerName */, new RouteValueDictionary(values), htmlAttributes);
   }

So following down the path I learn that the ActionLink method gets its URLs using the UrlHelper class such as the following manner:

string url = UrlHelper.GenerateUrl(null, actionName, null, values, htmlHelper.RouteCollection, htmlHelper.ViewContext);

So I try and use UrlHelper in my code so I can get the same nicely formatted URL. You can imagine my frustration when I find out that the code I need, URLHelper is INTERNAL and cannot be used by my extension.

Grrrrrrrrrrrrrrrrrr. 

Apparently the "static" UrlHelper methods were deemed too volatile to release for public consumption, because they depend on an omnicscient RequestContext object.
So this is what I end up with, I take the result of the ActionLink method, and use a regular expression to extract the URL that I desired and re-tool it into a button:

       public static string ButtonLink(this HtmlHelper htmlHelper, string buttonText, string actionName, object values)
    {
        string link = htmlHelper.ActionLink("-", actionName, values);
        link = new Regex(@"<a.*?href=[""']([^""']*)").Match(link).Groups[1].Value;
        link = string.Format("<button onClick=\"location.href='{0}'\">{1}</button>", link, buttonText);
        return link;
    }

This solution does not make me happy, to say the least. Please someone tell me that I am doing it wrong, and that it isnt always this ugly to extend HtmlHelper!!

**UPDATE** I have a cleaner solution to this problem, explained in the linked post

Question to the readers: How do you enforce a deadline that dies at 95%?

I'm faced with an interesting situation in my practice that I'm having trouble solving. Maybe this has to do with the holiday season, maybe it has something to do with the economy, or perhaps it is entirely my fault for being too lenient when I drive a project.

We get 95% through a project. The last 5% is usually the time where to seek user acceptance testing (UAT), assimilate and resolve up any lingering loose end details, and work closely with the client to make sure that they are fully satified with the project. The last 5% can be some of the hardest work in the project, both for myself and the client!

I now have multiple projects that have been hung in this state for a range of 3 weeks to six months now! Each one seems to be suspended for different reasons, sometimes it is due under-staffing, perhaps someone important is absent for awhile, and sometimes, I simply wont get a response until I send out multiple emails.

Is this because as a time window passes by, the priorities shift and the project is less important?

Maybe it is because 95% of the bill has been invoiced, and the client doesnt want to lose the on-call assistance of the developer?

Why do some clients become "gun-shy" right before it is time to close out a project?

What stept do you take to ensure against a client's lack of responsiveness near the tail-end of a contract? 

Genetic Art in Silverlight

The plan:
-Build a tool for creating evolutionary art.
-Most evolutionary art is done with genetic algorithms. This one will be done through hacking and crowdsourcing!

General approach:
*Create an AST capable of representing 3d plots. 

An expession is any math function that evaluates in terms of two variables s and t eg:

x=cos(S)*2T

A 3D expressions is composed of an expression for each of its three axes.

*Build a UI for visualizing the equations. Silverlight has no 3D native support, fortunately, the 3D code is not too hard to emulate using Kit3D, and some creativity.

UI features include:

• Refresh (re-generate the equation, plot the new derivative)
• Adjust speed/rotation angle
• Adjust zoom/pan
• Adjust S and T ranges
• Displays the current equation on the left pane
• Adjust resolution (because this program can be very CPU intensive on your machine)

*Build some equations to represent shapes

• Sphere
• Torus
• Spiral

*Build "splicers" that combine equations in random ways to create new generations of hybrid figures.

Progress:

This is still in a very rudimentary at the moment, but I am pumped about the potential!

Currently, it is hard-wired to try different splices of a torus and a spiral, hit the refresh to see different generations.

Try it out at:http://www.reallifedata.com/s3d/testpage.html

Database schema management using automated "Diff Tools"

Awhile back, I wrote an article on the codeproject about this subject, proposing a solution using hand-coded sql migration files under version management. (I recently found out this is the same design that ruby uses for rake migration files! Cool!)

I was talking with the tech lead from another development team about how to manage database schema versions (as opposed to juggling unwieldy backup files), he gave me some new perspectives.

This post covers an alternative approach to the same problem using robust 3rd-party database diff tools, as opposed to using hand-written SQL migration scripts.

The core of the solution revolves around a tool which can compare two different schemas of the same logcal database, and generate SQL script files that synchronize the schemas. For this kind of job on SQL server I've used:

• RedGate Sql compare
• Apex sqldiff
• The sql comparison tools with Team Server 2008

The Versioning Database

The Versioning Database exists to serve as a model schema. At all times, it matches up with the code inside SVN. 

It is lightweight, it contains no data, except for the enumerations/lookup tables. It contains tables / sprocs / views / triggers / functions / defaults / constraints / and any other intrinsic part of the database. 

(1)When updates are comitted to SVN, the versioning database needs to be synchronized with the committing developers database to match the latest code within SVN.

(2)When deploying to staging/production, the staging production databases are synchronized using the versioning database.

This approach has one big advantage:

• It eliminates the time and human error involved with maintaining long-running SQL migration scripts.

There are some drawback I havent been able to figure out solutions for yet:

• What about when you dont need to change the schema but you need to update the data inside of an enumeration/lookup table? 
• Sometimes that changes are too delicate to simply trust generated SQL scripts when migrating a production database.
• At the current time, I dont know a way to automate the procedure into cruise control. Some sort of a command-line utility would be a great solution.

Overall, I think this is a viable approach for small development shops, or crews that may not have the skill set to maintain a solid SQL migration script. 

An alternative would be to use the generated SQL migration code as a starting point for your manually maintained SQL migration scripts.

Advanced NANT Techniques

This post covers a few "real-world" build-automation tricks that have helped me tame the complexity of a sophisticated build system including:

• Modular builds (build "slices" of a source code tree)
• Intelligent deployments using BeyondCompare
• Code reuse design pattern with NANT
• Different build configurations for different branches

Back when I learned how to write makefiles, I always assumed that a build system would be something transparent and easy to understand. Boy how times have changed! 

 

I've been working on a cruise-control system that will be passed on to an external team for ownership, so I wanted something that was simple, transparent, and scaled out well as codebase grows. Here are a few of the requirements:

• Play nicely with cruise control systems such as TeamCity, Cruise Control, or Draco
  
• Minimize the amount of NANT code
  
• Automated unit testing with Nunit/MBUnit
  
• Modular build/deploy granularity (build/deploy target folders, solutions, or projects)
  
• Different code branches get deployed to different locations ("trunk" builds are deployed to "SERVER_A", feature branch builds are deployed to "SERVER_B")
  
• Regularly scheduled artifact backups
  
• Build in debug/release manifests for different branches ("trunk" gets release manifest, while "branches" are built with debug manifest)
  
• Folder sync deployments (deploy only the codebase "deltas" between releases, update necessary files, remove orphan files on the destination target)
  

Prerequisites:

To meet these kind of requirements, I procured some specialized tools:

• JetBrains TeamCity build server (I discussed this in an earlier post)
  
• NANT runtime .85 or higher - One of the problems with NANT and .net 3.5 is that you will need special configuration for .net 3.5 target framework version (http://codebetter.com/blogs/jeffrey.palermo/archive/2007/11/28/upgrade-nant-for-use-with-vs2008-solutions-and-net-3-5.aspx) 
  
• NANT contrib - specialized extension tasks for NANT
  
• .NET Framework SDK (SDK package is different than the .NET runtime. You can find out if you have the SDK installed by looking at the registry keys (HKLM\Software\Microsoft\.NETFramework\sdkInstall Rootv2.0))
  
• Beyond Compare 3 - Beyond Compare is the swiss-army knife of diff/merge utilities on windows. This is a small program with big capabilities, and you could easily spend days learning everything this program does. The cost of the program is $50, which is incredible considering its power. Their program should be a part of the core windows OS, because it _is_ that good. In these examples, we will be using the scripting features within NANT.
  

File structure:

The key to modular builds comes almost entirely from the folder structure. This example involves project/solution level build targets, but the approach should scale well for multiple solutions.

Project-level build:

Building/deploying a specific project is the finest level of granularity. I made a default.build file at the root level of each project. Each project-level build file contains a reference to the common build routines, and custom project-specific configurations

In this example, I reference and call the common build routines at the root of the source code tree:

<include buildfile="../../common.xml"/>


<target name="build">
<call target="common.compile"/>
</target>

<target name="test">
<call target="common.unittest" />
</target>

<target name="zip">
<call target="common.zip"/>
</target>

<target name="deploy">
<call target="common.folderSync"/>
</target>

<target name="configure">

</target>

Solution-level build:

For my concerns, all an "SLN" file really contains is a collection of projects, in dependency order. The default.build file on the project-level essentially means the same thing: it simply references the project-level build files in its sub-directories (dependency order matters).

The solution-level build file references all of the project-level build files in a dependency order (buildfiles.all):

 <fileset id="buildfiles.all">
  <!-- Files have to be in dependency order -->
  <include name="consoleapplication1/default.build" />
  <include name="webapplication1/default.build" />
 </fileset>

<target name="build" >
<nant target="build">
  <buildfiles refid="buildfiles.all" />
</nant>
</target>

<target name="test" >
<nant target="test">
  <buildfiles refid="buildfiles.all" />
</nant>
</target>

<target name="zip" >
<nant target="zip">
  <buildfiles refid="buildfiles.all" />
</nant>
</target>

<target name="deploy">
<nant target="deploy">
  <buildfiles refid="buildfiles.all" />
</nant>
</target>

Root-level build:

At the root of the VCS tree, we have the centralized build logic and commands. The following is a screenshot of the root VCS directory, where the central commands and beyond compare script is located:

Common.xml contains global build settings and reusable routines that specific projects will call within their targets.

<target name="common.compile" depends="initDependentVariables">
<exec
program="${framework::get-framework-directory(framework::get-target-framework())}/MSBuild.exe"
workingdir="${src.dir}"
verbose="${global.VerboseMode}"
failonerror="${global.FailOnErrorMode}"
>
<arg file="${src.dir}/${project.vsproj}" />
<arg value="/p:Configuration=${project.config}" />
</exec>
</target>

Using and configuring BeyondCompare

One of the "common" routines I hooked up leverages BeyondCompare via a command line call. I found this makes deployments more efficient and intelligent.

• Only the differences between SVN and the deployment destination are copied (this is more efficient than copying the entire codebase)
• I can easily strip out ".svn" folders, web.config, ".cs" files, or anything else I dont want my deployment destination to contain.

To do this, I call the program and specify a configuration file, a source dir, and a destination dir. An optional "--silent" mode is handy for quiet automated deployments.

Another attractive feature of the BeyondCompare deployment is that abiity to push changes over FTP, SFTP (ssh file transfer), or to a compressed zip folder.

<!--
 common.folderSync
 ==================================
 -src.dir: source directory
-destinationDir: determined within the "initDependentVariables" method
-project.rootLocation: used to locate the generic beyond compare script
-->
<target name="common.folderSync" depends="initDependentVariables">
<echo message="Folder sync from [${src.dir}] -> [${destinationDir}]"/>
<exec program="${global.BeyondComparePath}">
<arg value="@${project.rootLocation}BeyondCompareScript.txt"/>
<arg value="${src.dir}"/>
<arg value="${destinationDir}"/>
</exec>
</target>
</project>

BeyondCompareScript.txt is a file referenced in the command-line call to BeyondCompare. In this case, I use it to log deployment routines, strip out class files, web config, and .svn folders from being copied to the deployment destination, and only copy over files that are newer/different size to the deployment destination (as opposed to copying the entire codebase with each deployment)

#log the script
#log verbose append:"c:\website updates.txt"

#set the comparison criteria
criteria timestamp size

#load source and target
load %1 %2

#filter to only include source files, ignore CVS subdirectories
filter "-*.cs;-.svn\"

#Sync the local files to the web site, creating empty folders
sync create-empty mirror:lt->rt

To take a closer look, download the example build configuration with all of the source code here.

Objects and DataSets - Who says you can't have both?

This used to be a frequent debate amongst .NET developers not too long ago: should I use POCOs (plain old CLR objects) or DataSets in my application architecture?

Objects are the best choice for any application that is going to scale out beyond a trivial amount of complexity

DataSets give you some great shortcuts and advantages in the .NET realm:

• They're great for simple binding to standard grids in the .NET framework
  
• They play nicely with ADO
  
• They are great for converting to excel
  
• Their weak typing is advantageous for amorphic schemas
  
• They are great for converting to/from XML
  

I needed POCOs, but I wanted the convenience of DataSets. So I wrote some code that converts a collection of objects to/from a DataSet/DataTable. As a bonus, I added some functionality that makes it easy to take a collection of POCOs, select a few of their properties (in an order-specific fashion) and convert it to an excel worksheet.

Unit test- convert from an enumerable collection to a DataSet (notice how I specify the properties I want to include in the output):

public IEnumerable<Employee> Employees
      {
          get
          {
              return new List<Employee>
                              {
                                  new Employee {Name = "fred", ID = 1},
                                  new Employee {Name = "mary", ID = 2},
                                  new Employee {Name = "joe", ID = 3}
                              };
          }
      }
      [Test]
      public void TestBasicOperation()
      {
          var es = new CollectionToDataSet<Employee>(Employees,new List<string>{"ID","Name"},"dataSetName","dataTableName" );
          DataSet ds = es.CreateDataSet();
          Assert.AreEqual(ds.Tables.Count, 1);
          Assert.AreEqual(ds.Tables[0].Rows.Count, 3);
          Assert.AreEqual(ds.Tables[0].Columns.Count, 2);
      }

Convert from an enumerable collection to Excel re-using the code above, and stream it back as an excel workbook in a response:

    string attachment = "attachment; filename=test.xls";
      Response.AddHeader("content-disposition", attachment);
      Response.ContentType = "application/vnd.ms-excel";
      Response.Write(new CollectionToExcel<Employee>(Employees, new List<string> {"ID","Name"}).Output);
      Response.End();

Here is a unit test that converts code from a data table into POCOs. A caveat: this is potentially strict code in the send that the data table column names MUST match the target type's property names. Also, the target type must have a default constructor.

      [Test]
      public void TestBasicFunctionality()
      {
          var ds = new DataSet();
          ds.Tables.Add(new DataTable());
          ds.Tables[0].Columns.Add(new DataColumn("Name"));
          ds.Tables[0].Columns.Add(new DataColumn("ID", typeof (int)));
          ds.Tables[0].Rows.Add(new object[] {"joe", 1});
          ds.Tables[0].Rows.Add(new object[] {"mary", 2});
          ds.Tables[0].Rows.Add(new object[] {"fred", 3});
          var d = new DataTableToCollection<Entity>(ds.Tables[0]);
          IEnumerable<Entity> entities = d.ConvertToObjects();
          Assert.AreEqual(entities.Count(), 3);
      }

If you see any potential here, and you would like to use it/take it to the next level, here is a link to the CLASSES only (sorry, this is part of a bigger project, in which I can only share a slice)

Cross-browser compatible javascript sleep function

There are times when you need to add a delay to some form interaction.
In my case, I had a race condition with an ajax call.
At any rate, I quickly found out that there is no sleep function in javascript, jquery, or prototype. Maybe since javascript is single-threaded, this is a rare need.
Here is a quick snippet I used

function sleep(timeout) {
       var loop = true;
       var current = new Date();
       var now;
       var cTimestamp = current.getTime();
       while(loop) {
               now = new Date();
               nTimestamp = now.getTime();
               if(nTimestamp - cTimestamp > timeout) {
                       loop = false;
               }
       }
}

Umbrella- collections++

As soon as the .NET framework allowed room for extension methods, this kind of thing was bound to happen. Umbrella is a 3rd party open source lib that adds extension methods and intuitive helper classes that make you wonder why they weren't there in the first place. The official codeplex site is here and the author's site is here. There are quite a few useful bits in this library for multiple purposes, but the real beauty comes through in its collections enhancements.

 

The downside of umbrella is its immaturity. To this date of september '08 I couldnt find any "official" documentation of the library per se (outside of blogs such as mine or Ayende's). Then again, maybe the unit tests speak for themselves. I sense that this project is a moving target, meaning that it is one of those open source projects in state of flux, which can make it a little hard to document.

Foreach on an IEnumerable

IEnumerable<string> items = new List<string> {"1", "2", "3", "4", "5"};
IList<string> actionItems = new List<string>();
//the old way
foreach (string s in items)
{
 actionItems.Add(s);
}
//the umbrella way
items.ForEach(actionItems.Add);

Indexed foreach actions:

IEnumerable<string> items = new List<string> {"1", "2", "3", "4", "5"};
string[] actionItems = new string[5];
//the old way
int i = 0;
foreach (string item in items)
{
 actionItems[i++] = item;
}
//the umbrella equivalent
items.ForEach((i, item) => actionItems[i] = item);

Pairing two lists of equal length together

IEnumerable<string> items = new List<string> {"1", "2", "3", "4", "5"};
IEnumerable<string> otherItems = new List<string> { "6", "7", "8", "9", "10" };
IEnumerable<Pair<string>> pairedItems = items.Pair(otherItems);
Assert.Equal(5, pairedItems.Count());
Assert.Equal("1", pairedItems.First().X);
Assert.Equal("6", pairedItems.First().Y);

Lazy-loading lists:

//lists that are declared but not consturcted until access-time
var list = new LazyList<int>(() => new List<int> { 1, 2, 3 });

Read-only lists:

//lists that cannot be added/removed from (without resorting to DataViews)
//ReadOnlyCollection is a part of the core .NET library, it takes an IList<T> as a constructor //argument
ReadOnlyCollection<string> readonly =  list.AsReadOnly();

Lists that convert between types
(This one is intricate and I am still studying it)

//Funcs is actually a generic collection of strategies for converting between types.
//the real magic behind conversion comes from
//System.ComponentModel.TypeDescriptor.GetConverter(type).ConvertFrom(value)
IList<string> adapter = new ListAdapter<int, string>(
               new List<int> { 1, 3, 4 },
               Funcs<string,int>.Convert,
               Funcs<int,string>.Convert);

           Assert.Equal(1, adapter.IndexOf("3"));

These are the pieces that I am using within my projects, but I am only scratching the surface. I suppose when it somes to collections, you can't have too much extensibility.

Download it, use it, or simply wait until these features are folded into the .NET lib in the next release ;)

Linq brings back lisp memories

Often when I am searching for the most succinct way to write logic, I find myself using a declarative syntax:

public  ExecutionOutcome GetValidationMessages()
       {
           IEnumerable<ExecutionOutcome> failedExecutionOutcomes = Context.ItemGroups
               .Where(x=>x.IsValid==false)
               .Select(x=>ExecutionOutcome.CreateFailureOutcome(
                       string.Format(
                                                        "...",
                                                        x.ItemGroupName
                       )
                   )  
               );
           if(failedExecutionOutcomes.Count()>0)
           {
               return ExecutionOutcome.CreateFailureOutcome(failedExecutionOutcomes.Aggregate(string.Empty, (x,y) => x += y.Message + System.Environment.NewLine));
           }
           return ExecutionOutcome.CreateSuccessfulOutcome();
       }

Thats relatively "cutting edge" the C# crew, but its language concept is about as old as the hills. Here is the ballpark equivalent in LISP, a language concieved in the late 50's.

(apply
 (GetMessage '#),
 (mapcar
  (lambda (x)
   (if (IsValid x)
    nil,
    (CreateExecutionFailureOutcome("..." (ItemGroupName x))),
   )
  ),
  (ItemGroups Context)
 )
)

Lisp was a little on the cryptic side, but its true power was its declarative syntax. Lisp is a "functional language".

C# is a sequential language, with declarative extensions. Linq gives me the ability to jump in and out of "functional language" mode in C#. I find myself gradually turning back into a LISP programmer :)

One thing I wish I could do is take this kind of declarative power and apply it to SQL query plans, now that would impress me.

Sharepoint C# HTML Trick: Is this fluent programming or sleazy hack?

The challenge: writing custom sharepoint web-parts can be painful because there is no inline code markup. (and I'm NOT using the "smart part" )

How to write HTML using nothing but C#?
HTML is a tree just like any other AST.

This is the code I wanted to write:

<asp:Table Width="100%">
       <asp:TableRow>
           <asp:TableCell Width="100%" CssClass="headerRow">Alphabetical Search</asp:TableCell>
       </asp:TableRow>
       <asp:TableRow>
           <asp:TableCell Width="100%">
               <asp:GridView runat="server" Width="100%" />
           </asp:TableCell>
       </asp:TableRow>
   </asp:Table>

This is the code equivalent (plus databinding) using fluent constructors using C#:

protected override void CreateChildControls()

       {

           var dataFeed = new List

                              {

                                  "one",

                                  "two",

                                  "three"

                              };

           var gridView = new GridView

                              {

                                  Width = new Unit("100%"),

                                  DataSource = dataFeed,

                                  Visible = true

                              };

           gridView.DataBind();

           var table = new Table

                           {

                               Width = new Unit("100%"),

                               Rows =

                                   {

                                       new TableRow

                                           {

                                               Cells =

                                                   {

                                                       new TableCell

                                                           {

                                                               Text = "Alphabetical Search",

                                                               CssClass = "headerRow"

                                                           }

                                                   }

                                           },

                                       new TableRow

                                           {

                                               Cells =

                                                   {

                                                       new TableCell

                                                           {

                                                               Controls =

                                                                   {

                                                                       gridView

                                                                   }

                                                           }

                                                   }

                                           }

                                   }

                           };

           Controls.Add(table);

           base.CreateChildControls();

       }

Nothing earth shattering here, but I actually dont mind using this syntax. This is the result:

**EDIT** this style of programming runs out of gas rather quickly when you are faced with the need for better interaction. A better solution is to use the "smart template" project templates at codeplex:

http://www.codeplex.com/smarttemplates


Do you like it?

Temporarily disable a button in ASP.NET

Sometimes you have a button that can trigger an asynchronous long-running process in which the http response is not a new page, but a file (excel/pdf etc).

You may want to disable the button such that impatient users don't unwittingly click the server resources into submission, but you want to re-enable it after the response is complete.

The bad news is, there is no way on the client-side to recognize when a new response has been sent (without ajax), but we can really easily set a timeout to re-enable the button after a fixed threshold of time.

Consider the following javascript function:

   function TemporarilyDisableButton(id, delay)
{
    document.getElementById(id).disabled = true;
    setTimeout("document.getElementById('"+id+"').disabled = false",delay);
}

The id is the ClientSideID of the button. The delay is the time in milliseconds to disable the button.
Wiring this together in your code behind is as simple as:

btnSumbit.Attributes.Add("onclick", "TemporarilyDisableButton('" + btnSumbit.ClientID + "',15000);" + GetPostBackEventReference(btnSumbit).ToString());

And the above code will disable the button for 15 seconds on the client side after the click event has fired.

Stored procedures are not always the best tool for selecting

Working as a consultant in the past few years has given me an opportunity to view a diverse set of code bases, and see come of the common trends and habits of developers all over the map.

Literally every single codebase including (some frameworks) I have seen in the past few years bear a common design that bugs me:

READ-ONLY SQL STATEMENTS DONT NECESSARILY BELONG IN STORED PROCEDURES, THATS WHAT FUNCTIONS ARE FOR.

If you dont know what a database function is, just think of it as a parametric view. Now let me back this up some facts, and a code example that I feel justifies this argument once and for all.

1. There is NO performance difference between a user-defined function and a stored procedure. None at all. Since the introduction of UDFs in SQL Server 2000, functions and stored procedures leverage the same compiled/cached query optimizations. In sql 2005, both stored procedures and functions can be written in the CLR environment. Since a function is guaranteed to be read-only and deterministic, in some cases a function will have a more optimal query plan than the equivalent stored procedure.
   
2. A function GUARANTEES there are no side effects to the data. Since UDFs are parametric views, they are by definition deterministic and read-only opterations, so when you call them you never have to worry about any surprise side effects to your data (stored procedures cannot guarantee this).
3. Stored procedures are not as modular and flexible as functions. In many cases, you may need to continue ETL operations on the result set from a compiled query. Every wished you could select or join from a stored procedure? Consider this scenario: I need a northwind query to retrieve employees with a HiredDate between a given date range;
   Stored procedure:
   

   CREATE PROCEDURE sp_GetEmployeesHiredWithindateRange
   @startDate datetime,
   @endDate datetime
   AS
   BEGIN
   select * from employees where hireDate between @startDate and @endDate
   END
   

   Function equivalent:
   

   CREATE FUNCTION fn_GetEmployeesHiredWithindateRange
   (
   @startDate datetime,
   @endDate datetime
   )
   RETURNS TABLE
   AS
   RETURN
   (
   select * from employees where hireDate between @startDate and @endDate
   )
   

   I call them like this:
   

   select * from fn_GetEmployeesHiredWithindateRange('5/25/1990','5/25/2005')
   exec sp_GetEmployeesHiredWithindateRange '5/25/1990','5/25/2005'
   

   and they will return the same result set. Now, imagine I have a special case where I want to combine this query with whom each employee in the original results reports to.
   With a function, I just modify the query such as:
   

   select * from fn_GetEmployeesHiredWithindateRange('5/25/1990','5/25/2005') e1
   inner join employees e2 on e2.employeeid = e1.reportsto
   

   and I am done. If I chose the stored procedure route, there is no flexible way to do this, I would need to write another stored procedure just for this case. Because you can select from the result set of functions, you can do some GREAT code reuse design on the database side that would have been a nightmare in stored procedures.
   
4. Stored procedures ARE NOT the only way to protect against sql injection. I see quite a few people defending their choice of stored procedures because the rigid syntax of a sproc call protects the system against query injections. There are easier ways! Let me show you an example of how to write a UDF call that has the same level of protection:
   

          string sqlQuery = "select * from fn_GetEmployeesHiredWithindateRange(@startDate,@enddate)";
       DataTable dataTable = new DataTable();
       SqlCommand sqlCommand = new SqlCommand(query, sqlConnection);
       sqlCommand.CommandType = CommandType.Text;
       sqlCommand.CommandTimeout = 300;
       sqlCommand.Parameters.AddWithValue("@startDate",new DateTime(1990,5,25));
       sqlCommand.Parameters.AddWithValue("@endDate",new DateTime(2005,5,25));
       SqlDataAdapter sqlDataAdapter = new SqlDataAdapter(sqlCommand);
       sqlDataAdapter.Fill(dataTable);
       return dataTable;
   

   I am writing ad-hoc sql inside of my C#, and parameterizing it out like a stored procedure. You know what that ends up looking like on the sql profiler?
   

   exec sp_executesql N'select * from fn_GetEmployeesHiredWithindateRange(@startDate,@endDate)',N'@startDate datetime, @endDate datetime',@startDate='5/25/1990',@endDate='5/25/2005'
   

   sp_executesql is a call from ADO.NET for running generated queries in a controlled stored procedure fashion. Subsequent calls of a similar structure enjoy the speed boost of a compiled query. This is the magic behind NHibernate's query-generation logic. In terms of SQL injection, this is every bit as safe as a stored procedure call, but with more flexibility.

To sum this all up:

• Stored procedures are the right choice for compiled database functions with side effects (eg: inserts and updates)
• Functions are the right choice for compiled parametric queries (eg selects)
• When you think of functions, think of them as views with arguments.
  

Some other notes:

• You cannot "order by" inside of a function, but you can in a stored procedure. You cannot order the result set from a stored procedure, but you can order the result set from a function.
• Stored procedures have output parameters, functions don't.
• Functions can return tables or single values, for example:
  Here is the calling syntax for a single-valued output function:
  

  select fn_getStartDateForEmployee(employeeID) as startDate from employees
  

  and here is the calling syntax for a table-valued output function:

  select * from fn_GetEmployeesHiredWithindateRange('1/1/2000','1/1/2005')

VisualSVN + TeamCity + Nant + Sql Server = A Great CI solution

When you work full time as a programmer, the chances are that your codebase is a big portion of your business value.

You shouldnt just write code write code; protect it, keep it clean, version it out, and ensure its quality!
You should also be able to easily deploy and deliver that codebase in a clean, reliable automated fashion, limiting human-related error risks and giving yourself the ability to deploy quality release artifacts/MSIs/ website updates at the drop of a hat.

Its hard to argue against these ideals, but sometimes it just seems you never have the time to "do it right".

After a bit of experimentation, I've put together a solid CI system on the cheap. Its flexible, scales out well, and you could go from zero to solid build server in two day's time**.
**It may take 1 hour mechanically, but the two days factors in learning curve and any unanticipated setbacks.

The Tools:

SCM: VisualSVN Server
Cost: Free
Subversion may not be the perfect SCM, but short of its merging hassles, its a widely popular well-documented rock-solid standard, and handles true branching strategies perfectly. VisualSVN Server is a configuration of subversion straight-jacketed for windows environments. It has windows integrated authentication, and you can provision subsets of the code base to windows users and groups on a read-only/read-write basis. Https server access comes automatically as the program starts up. Their Visual Studio plugin works with Vs 2005 and 2008 flawlessly. Even though there are free alternatives for VS integration, dont be a cheapskate here, if you pony up the $50 for their VS plugin, you wont regret it.

CI: JetBrains TeamCity
Cost: Free for up to 20 unique build actions, 20 users, 3 separate build agent processes. (A generous number)
While its true that CruiseControl.NET or CIFactory gives you everything you need for CI, those options look like a scripting nightmare reminiscent of recompiling a linux kernel in comparison to TeamCity. Besides, with built-in features such as distributed build farm agents, reports, and alerting it rivals the feature set of Microsoft Team Foundation server. TeamCity integrates tightly with subversion and gives you the ability to perform "delayed commits", which is like running a hypothetical build on the server without committing any code. (This is the equivalent to "shelfing" a build in TFS). TeamCity is a platform-agnostic build server, which is nice if you have Java/rails/AIR, or other non-.NET codebases to manage.

Build Action Scripting: NANT and NantContrib
Cost: Free, small learning curve
Dont be scared, just learn it. Ant and Nant are powerful xml-based scripting languages designed specifically for building and deploying code. If you dont know nant, dont write a script from scratch, invest a focused 2-3 hours learning how it works by tinkering around with my pre-written example and you'll be ready to write your own. A well-built nant script can be executed locally or on the build server with the same results. This FAQ shows you how to add Nant-scripting intellisense into your visual studio editor to help you get started.

Build Server Database: Sql Server
Cost: If youre reading this, chances are you already have one.
TeamCity relies on a database. Out of the box, it uses a file-based database, which is not recommended for long-term stability. JetBrains recommends you migrate the server to a MySql database, but for those of us adventurous and adamant about consolidating our resources, there is experimental support** for Sql Server 2005. I'll show you how to do that.
**Its been working flawlessly for me with everything Ive thrown at it

The goal:


Basically, developers of diverse roles and platforms (web designers, QAs, architects), can update/commit to and from a common VisualSVN server over an https line, administered and secured by a "build master" role. The "build master" provisions read/write/none access to windows/AD identities and groups.

When changes have been committed to the SVN repository, a build action can be triggered on the TeamCity server. The particular action that is triggered can depend on factors such as:

• The location of the changed files within the source tree
  
• What person committed changes
  

During a build action, a build agent process will create a new clean temp directory, check out the latest version of the code base into the directory, and carry out a set of desired build actions. TeamCity offers some basic pre-fab build actions such as:

• SLN 2005/2007: Build an entire solution, run NUnit tests, and report build/unit test results
  
• Duplicate Finder: Report any duplicate code within a codebase

They are good starting examples, but more than likely your best option is to trigger your own custom NANT script. With custom scripting:

• You have versions of a build routine managed source control
• You can run a custom build script locally instead of the standard ctrl+shift-B in visual studio.
• Its easy to configure a build agent to run a script within the source code tree.
  

So, if this design sounds good to you, lets go through the steps to making it happen.

0. First things first: determine what machines are going to host each application listed above and make sure they have access to one another (maybe all of the resources can be on the same server?). make a list of the build routines you desire, and when they will need to run. Make a list of your target deployment servers, their target directories, how and when deployments are going to be delivered. Create a windows identity that will host the build agent service, and make sure that this identity has write access to your intended deployment targets. The more time you invest in step 0, the easier the rest of this system will be to implement.

1. Install and configure the VisualSVN server. This is mostly a no-brainer here, it needs to be readily available to anyone who works with the code, so install this on a server that you can reach via https, or at least a secret port number. A good repository design is to use one solution per repository.

2. Import your codebase into the repository. "Check out" a local copy of your new and empty repository, paste your codebase into the trunk folder, and commit what you just checked out. Avoid adding auto-generated (dlls, resharper temp files, bin/obj folders) to the code repository. There are many great articles about getting started with subversion, here is one about getting started with VisualSVN. Once your code base has been committed to the repository, you can set up user access to the source tree.

3. Install and configure TeamCity. Run the setup on the target server. Log in to ther web server -based console. Create a VCS root, and point it to the URL of your newly created repository. The URL of your repository can be found on the VisualSVN administration console.

4. Install and configure a build agent. Use the TeamCity console to add/install a build agent, install the build agent service on the desired host server. In the MMC services console (start->control panel->administration tools->services), locate the build agent service, and bind the service's identity to that identity you made in step 0.

5. Create the TeamCity Database in your SQL Server. Just login to your target sql server and create a new database and call it 'TeamCity'. Unfortunately, TeamCity uses JDBC for connectivity and that means you can't use windows integrated authentication via this connection, you will need to create a login for this database, and give the new login "dbowner" privileges for the TeamCity database.

5. Make TeamCity talk to Sql server 2005. (One more time, sql server has "experimental support", but it has worked well for me) TeamCity has migration scripts inside of the bin folder of your installation. When the migration scripts run, they install tables and stored procedures onto the target database. For a successful migration, you will need to configure the dbmigration.properties file, and point it to the traget database, as well as the correct jdbc driver. General instructions here. To migrate into sql server, you will need the sql server 2005 jdbc driver located here at microsoft, and you should copy it to the same driver folder specified in the general instructions (\webapps\ROOT\WEB-INF\lib folder off your root TeamCity installation directory). The connection string you will be using inside of the dbMigration file should look something like:

driverName=com.microsoft.sqlserver.jdbc.SQLServerDriver
connectionUrl=jdbc:sqlserver://localhost:1433;database=TeamCity;
connectionProperties.user=user
connectionProperties.password=password

maxConnections=50
poolPreparedStatements=false

and you should change the server/user/password/port connections to your needs. You can test the connection to your server using [telnet [server] [port]] command on the command line. After youve run the migration batch file and the migration is complete, restart the TeamCity web server service (under the services console). When you re-visit the server with a browser, dont worry about the prompt to create a new user, you have not lost your build configurations.

6. Install Nant/Nant-Contrib. You will need to install these both on your developer machine and on the build server machine. When defining a new Nant build action in TeamCity, you will need to specify the path to the Nant binaries. Nant does not support building sln files in VS2005 or older, nor does it build individual csproj files, but MSBuild does. TeamCity doesnt support MSBuild triggering, but it supports Nant triggering. I bridged this annoying little gap using a custom MSBuild Nant task, a Nant task that calls MsBuild. Thats one of the many great custom tasks found in the NantContrib library. Here is an example of a simple build, test, deploy script using Nant:

<?xml encoding="utf-8">
<project name="MyProj" default="run" basedir="..\" xmlns="http://nant.sf.net/release/0.85/nant.xsd">
<property name="SchemaFile" value="C:\Program Files\Microsoft Visual Studio 8\Xml\Schemas\NAnt.xsd">
<description>Build Script</description>

<!-- builds only the csproj, not the entire solution-->
<target name="build"
description="Compile the project using Debug configuration for more verbose error descriptions">
<echo message="Building...">
<msbuild verbose="true" verbosity="Detailed" failonerror="true" project="MyProject.csproj">
<property name="debug" value="true">
</msbuild>
</target>

<!-- deploys the project -->
<target name="deploy" depends="build, test" description="Copy production files">
<!-- Copy build results to Production folder -->
<echo message="Deploying...">
<copy
todir="\\mydestinationserver"
failonerror="true"
overwrite="true"
includeemptydirs="true"
verbose="true"
>
<fileset basedir=".">
<includes name="**/*">
<exclude name="**.resx">
<exclude name="**.cs">
<exclude name="**.csproj*">
<exclude name="**/obj/**">
<exclude name="**.pdb">
<!--<includes name="**.aspx">
 <includes name="**.ascx">
 <includes name="**.config">
 <includes name="**.gif">
 <includes name="**.jpg">
<includes name="**.mdb">-->
</fileset>
</copy>

<!-- Update connection string settings -->
<!--<xmlpoke file="Web.config"
xpath="/configuration/location/appSettings/add
[@key='ConnStringAdmin']/@value"
value="Data Source=whatever;User ID=...;" />-->
</target>

<!-- runs unit tests, if the unit tests fail, the process stops -->
<target name="test" depends="build" description="run unit tests">
<echo message="testing">
<nunit2 failonerror="false" verbose="true">
<test>
<assemblies basedir="UnitTest\bin\Debug\"> 
 <includes name="*Test.dll"> 
</assemblies>
</test>
</nunit2>
</target>

<!-- default target -->
<target name="run" depends="build, test, deploy" description="build, test and deploy">
</target>

</project>

This code will build a specified csproj file, run unit tests on a specified assembly, and if everything goes well enough, it will update a connection string in the configuration file as it copies the new code base to a specified destination (maybe change over to a staging server database). For web projects, if you have any trouble running this, you may need to copy your Microsoft.WebApplication.Targets file to a local directory, so you wont have to install visual studio on your build server. More details about that little trick here.

7. Define your build actions and how they are triggered. Maybe you will run an NDepend analysis or duplication search on a weekly basis. Maybe you have a different deployment destination for a branch of a project versus the trunk of a project, or maybe you need one build action to run in succession to another, forming a chain of build actions. Here is an example of location-based build triggering syntax in TeamCity:

+:/trunk/**
-:/branches/**

This build action would be triggered by committed files to the trunk, and not by changes to the branches directory of a repository.

8. Test it out, fine tune it. You should be able to run that nant script locally, or on the build server. Once you have the kinks worked out, release it to your development team and try it out in practice.

In summary, this plan is an approach to a relatively easy-to-implement build system that can pay off big time in both the immediate and long run.

I'll be interested in hearing any variations on this plan that work for you.

Extension Method? Visitor?

I was reading this really cool article from Scott Hanselman about a newer, smarter way to format objects into a string.

The gist of it is this: you have an extension method called "ToString(..)", with a signature like this:

public static class FormattableObject
{
...
public static string ToString(this object anObject, string aFormat, IFormatProvider formatProvider)
...
}

So what this offer us is a really cool way via reflection to format objects based on their property/field names. For example, consider the basic Person class:

public class Person
{
...
public string FirstName{ get; set;}
public decimal Money{get; set;}
}

and then you can make a call like:

Person p = new Person(3.43M, "Joe");
Console.WriteLine(p.ToString("{Money:C}, {LastName}"));

and the output would look like

3.43, Joe

^^Now thats cool! This is a really simple and powerful piece of formatting code!
But then I paused for a minute. I like the reflection and formatting, but what benefit does the extension method bring? Its a parlor trick with a slight syntactic advantage, but it also seems to unnecessarily obfuscate things. To prove my point, I extracted the meat of his code into a simple (read: "less-cool") static method.

In essense, consider the following two lines:

1. Console.WriteLine(p.ToString("{Money:C}, {LastName}"));
2. Console.WriteLine(StringFormattingVisitor.ConvertToString(p, "{Money:C}, {LastName}"));

Now the following two lines of code are functionally equivalent. Call me an old fart, but I like line #2 better, because if theres ever an issue related to the cool string formatting functionality, I can read it and see exactly where the method is defined, the first one abstracts this information away from me.
Basically, what I am seeing is that extension methods are a way to implement object visitors, but the actual identity of the visitor is hidden, such that it looks like a native method. I'm still struggling to find out why this is better than writing a plain old explicit visitor. Any opinions?

How to Export from Sql Server 2005 to a well-formed XML File

Its a shame I have to write this one. It started out innocent enough. I need some data dumped out of a query and into a well-formatted xml file for further xslt transformations.

Piece of cake, just run a "FOR XML" query on a SQL server, output the results to file, and you will have an easy-to-translate XML file, right? Not exactly...

Say this is Northwind and I want the categories joined with the products:

DECLARE @output xml;
SET     @output =
(
SELECT
categories.categoryid AS '@categoryID',
 (
SELECT  
products.productid AS '@productID'
FROM       
Products
WHERE   
products.categoryid = categories.categoryid
FOR xml path('product'), type, root('products')
)
FROM       
Categories
FOR xml path('category'), TYPE, root('categories')
);

SELECT     @output;

Notice that I need something more powerful than RAW or AUTO mode, I want to define my own custom xml schema using the PATH syntax.
Then, I can see in the sql output window something like this:

<categories>
<category categoryID="1">
 <products>
   <product productID="1" />
   <product productID="2" />
   <product productID="24" />
   <product productID="34" />
   <product productID="35" />
   <product productID="38" />
   <product productID="39" />
   <product productID="43" />
   <product productID="67" />
   <product productID="70" />
   <product productID="75" />
   <product productID="76" />
 </products>
</category>
<category categoryID="2">
 <products>
   <product productID="3" />
   <product productID="4" />
   <product productID="5" />
   <product productID="6" />
   <product productID="8" />
   <product productID="15" />
   <product productID="44" />
   <product productID="61" />
   <product productID="63" />
   <product productID="65" />
   <product productID="66" />
   <product productID="77" />
 </products>
</category>
<category categoryID="3">
 <products>
   <product productID="16" />
   <product productID="19" />

..you get the idea. Now I have the format and results I want, but this needs to programmatically be exported to a file. it time to switch my output from grid/text to a file... only to get this garbage in my file:

------------------------------------------------------------------------
<categories><category categoryID="1"><products><product productID="1" /><product productID="2" /><product productID="24" /><product productID="34" /><product productID="35" /><product productID="38" /><product productID="39" /><product productID="43" /><product productID="67" /><product productID="70" /><product productID="75" /><product productID="76" /></products></category><category categoryID="2"><products><product productID="3" /><product productID="4" /><product productID="5" /><product productID="6" /><product productID="8" /><product productID="15" /><product productID="44" /><product productID="61" /><product productID="63" /><product productID="65" /><prod
(1 row(s) affected)

And that is the result of a sql query for xml exported to a file. Obviously, we can't pass this to an xml parser and expect anything to work. What to do? I spent an extra 1/2 hour and ended up making a little ADO.NET program that would:

1. Open a sql query file
2. Connect to a sql server database
3. Execute the query, read the full result set as an xml stream, and save a well-formatted xml document at the specified destination.
   

Here is the usage:

XmlExtract.exe [sql query file name] [connection string] [output file destination]
For example:
XmlExtract.exe query.sql "data source=pweissbrod0308\\sql2k5;integrated security=true;initial catalog=northwind;" "C:\output.xml"


And finally, here is the source code for my program:

using System;
using System.Data.SqlClient;
using System.IO;
using System.Text;
using System.Xml;

namespace XmlExtract
{
  public class Program
  {
      private static string connectionString;

      private static string outputFileName;
      private static string sqlQueryFilename;

      private static void Main(string[] args)
      {
          sqlQueryFilename = args[0];
          connectionString = args[1];
          outputFileName = args[2];

          TextReader textReader = new StreamReader(sqlQueryFilename);
          string query = textReader.ReadToEnd();
          textReader.Close();
          Console.WriteLine(query);
          var stringBuilder = new StringBuilder();

          SqlDataReader sqlDataReader;
          using (var sqlConnection = new SqlConnection(connectionString))
          {
              var sqlCommand = new SqlCommand(query, sqlConnection);

              sqlConnection.Open();
              sqlDataReader = sqlCommand.ExecuteReader();

              while (sqlDataReader.Read())
              {
                  stringBuilder.Append(sqlDataReader.GetString(0));
              }
              sqlDataReader.Close();

              var xmlDocument = new XmlDocument();
              xmlDocument.LoadXml(stringBuilder.ToString());

              if (File.Exists(outputFileName))
                  File.Delete(outputFileName);

              var xmlTextWriter = new XmlTextWriter(outputFileName, Encoding.UTF8);
              xmlTextWriter.Formatting = Formatting.Indented;
              xmlTextWriter.WriteStartDocument();
              xmlDocument.WriteTo(xmlTextWriter);
              xmlTextWriter.Flush();
              xmlTextWriter.Close();
          }
      }
  }
}

I hope this helps anyone with similar endeavors, feel free to paste this into snippet compiler to get a quick app up and running, or patch it into your current project for something more custom.