Archive for the ‘Uncategorized’ category

Silverlight vs Flash

August 16, 2007

Metadata Programming and LINQ

August 6, 2007


One of my favorite uses of LINQ is being able to query over metadata and use it to create flexible programs. Conversely, I also like to use LINQ to quickly create metadata about my objects. Metadata simply put is “data about data” or, for example, information that describes content. I want to walk through a simple example that shows how to use information stored in a database that describes some content for some controls on a form. And then I’ll show you how to create data (XML) that describes that form and its controls. NOTE: The example I’m creating works with a pre-release Beta 2 version of Visual Studio 2008.

The idea for this example is that I want to read the text of the controls from a database and depending on the language I select, the text on the controls will change. I’ve created a very simple table in my database called ControlData that has three string fields, Name, Description and Language. The data looks like this:

Name Description Language

———- ————- ———

Button1 Hello English

Button2 Goodbye English

Button3 I love VB English

Button4 LINQ Rocks! English

TextBox1 Hello English

TextBox2 Goodbye English

Button1 Buon Giorno Italian

Button2 Ciao Italian

Button3 Amo VB Italian

Button4 LINQ è potente Italian

TextBox1 Buon Giorno Italian

So next what I’ve done is create a form with some controls on it. Some control names match the names here in the database and others do not. We’re going to write queries that can work over all the controls on the form and join in the matching information here to change the Text they display, depending on the language we select (which I’ve placed in a Combobox). So my form looks like this:

The first thing we’re going to do is create a LINQ to SQL class using the new O\R designer. It’s very simple to use this, just right-click on your project and select “Add New Item” and select the LINQ to SQL Class template. This creates a file with a .dbml file extension and opens the O/R designer (I just went with the default name of DataClasses1). You can simply drag tables from your Server Explorer onto the surface and it will generate all the code for you similar to the DataSet designer experience.

In my case I just dragged the ControlData table onto the surface and it created what’s called a DataContext with a single class called ControlData. We can now query over the ControlData and LINQ to SQL will translate our LINQ queries into SQL queries automatically. All we need to do is create an instance of the DataContext and then we can start querying over the ControlData.

So the first LINQ query I want to write will populate the contents of the Combobox that displays the Language. This is done by selecting the distinct rows and then setting the result as the DataSource of the Combobox:

Public Class Form1

Private db As New DataClasses1DataContext

Private Sub Form1_Load() Handles MyBase.Load

Dim choices = From info In db.ControlDatas _

Select info.Language Distinct

Me.ComboBox1.DataSource = choices

End Sub

What the variable ‘choices’ ends up being from this query is a simple list of strings (IQueryable(Of String)) and that works fine for our Combobox’s DataSource — we’ll end up with two strings, “English” and “Italian”.

Next I want to handle the Combobox’s SelectedIndexChanged event in order to change the Text property of our controls. I’m going to add a twist to this example and only set the text of Button controls. We can select all the top-level buttons on a Form my using the Me.Controls collection. This collection has been on WinForms since the beginning of time so it returns a ControlsCollection which means we’ll have to cast our Buttons out of it. I chose to do it like this:

Dim buttons = From c In Me.Controls _

Where TypeOf c Is Button _

Select CType(c, Button)

We could iterate this list of buttons (IEnumerable(Of Button)) now and set the Text property manually if we wanted:

For Each button In buttons

button.Text = “LINQ is cool”


But what we want to do is join this list of buttons with the ControlData objects matching on the Name and filtering by the Language so when all is said and done the SelectedIndexChanged event handler code will look like so (I split the query up into two parts for readability):

Private Sub ComboBox1_SelectedIndexChanged() Handles ComboBox1.SelectedIndexChanged

Dim buttons = From c In Me.Controls _

Where TypeOf c Is Button _

Select CType(c, Button)

Dim buttonInfo = From Button In buttons _

From info In db.ControlDatas _

Where info.Name = Button.Name _

AndAlso info.Language = Me.ComboBox1.SelectedValue.ToString _

Select Button, info.Description

For Each item In buttonInfo

item.Button.Text = item.Description


End Sub

What we’ve selected (projected) for buttonInfo is a list of anonymous types. If you notice we’re selecting the Button objects as well as the string Description field — this creates a list of anonymous types for us that we are iterating and using to set the Text property. That’s all there is too it. When we select the language in the combobox, the Text of the Buttons change.

The last thing I want to do is to create some metadata about this form by creating a simple XML document that describes the controls as well as the possible content in the database. We can use LINQ to XML and XML literals to do this easily. The only trick is that we’ll need to perform a group join because some of the controls on our form are not contained in the database and some of the controls have multiple rows of information. So we have zero or more matching ControlData objects for each of the Form’s Controls. So I perform a group join of the information in the database and then use that group as a sub-query to get the zero, one, or more Descriptions which I place into a <Text> element. This ends up under the <Control> elements that are selected which are under the root <Form> element:

Private Sub cmdMakeXML_Click() Handles cmdMakeXML.Click

Dim formData = <?xml version=”1.0″?>


name=<%= Me.Name %>

height=<%= Me.Height %>

width=<%= Me.Width %>

top=<%= Me.Top %>

left=<%= Me.Left %>

font=<%= Me.Font %>>

<%= _

From item In Me.Controls _

Let c = CType(item, Control) _

Group Join info In db.ControlDatas On info.Name Equals c.Name Into Group _

Order By c.Name _

Select <Control

type=<%= c.GetType %>

name=<%= c.Name %>

height=<%= c.Height %>

width=<%= c.Width %>

top=<%= c.Top %>

left=<%= c.Left %>

font=<%= c.Font %>>

<%= From info In Group _

Order By info.Language _

Select <Text language=<%= info.Language %>>

<%= info.Description %>

</Text> %>

</Control> _


My.Computer.FileSystem.WriteAllText(“Form1.xml”, formData.ToString, False)

Process.Start(“notepad.exe”, “Form1.xml”)

End Sub

Here’s a rip of the XML that is created:

LINQ is really powerful and simplifies a lot of common scenarios where we need to query over data in many forms, in this case a database, a collection of objects and XML. I’ve attached this project but please note that some features may not work as expected until the release of Visual Studio 2008 Beta 2. For more information on LINQ please check out the LINQ Project page on MSDN.


Metadata Programming and LINQ


August 6, 2007


Pat Helland has been talking about immutable data for a while. He last post on ‘Normalization is for sissies’ is quite fun. A not-very-accurate post from Dare made me remember about it and pushed me to post this.

Pat is playing with two ideas.

One is that immutable data should not be normalized as normalization is designed to help you dealing with updates.

Another is that you actually don’t need to delete/update the database. ‘Deleting’ a row means setting a InvalidationTimestamp = now(), and updating a row means setting InvalidationTimestamp = now() and inserting a new row with SinceTimestamp = now() and InvalidationTimestamp = null (you actually need two sets of dates, but that’s for another post).

Now, if you put the two ideas together, all the data is immutable, so you don’t need to normalize anything. This means you will have a record that have all the ‘extended table’: the ‘base table’ and all the fields from related tables in your normalized model. If you have Orders, Customers, Countries, your tables will look like

Order: OrderId, OrderDate, CustomerId, CustomerName, CountryId, CountryName, SinceTimeStamp, InvalidationTimeStamp

Customer: CustomerId, CustomerName, CountryId, CountryName, SinceTimeStamp, InvalidationTimeStamp

Country: CountryId, CountryName, SinceTimeStamp, InvalidationTimeStamp

You will be wasting a lot of disk space, but that’s not something to worry about. The advantages of this approach are very important. You don’t need to join, and you can cache/replicate most of your data.

The main physical issue I find today with this approach is that database engines have a limit in the number of columns they can store, and an approach like this one will require a large number of columns per table.

I wonder how a model like this will impact O/R mapping tools.

They can probably hide this kind of schema automatically by changing the semantics of delete/update, writing Order.Customer.Name should return the name in the Order row.

How would they handle object identity? Now if I have ‘Customer #1’ in memory, every reference to Customer #1 points to the same instance. This is because the object model is normalized. Now they should point to different read/only instances.


Levels of automated testing within a single application

July 4, 2007


We need a common language for the different types of automated testing.  We’re partially there, but the term “unit test” is still very confusing.  Here, I’ll lay out the different types of automated tests I find helpful with a single application:

  • Unit testing – testing a single class or possible a small group of collaborating classes (absolutely does not call out of process and is the fastest-running of all automated tests).  Running 1000 unit tests in 3 or 4 seconds is common.
  • Full system tests – through the UI integrated with the full application including the database.  May or may not use real system dependencies such as external web services.  (These are the slowest of all tests)
  • Integration testing.  Here, there are some categories.
    • Data access tests.  Used to test repositories, data access classes, etc.  These tests validate the translation from entities to data.  These tests run all SQL and test the structure of the database schema as well.  A real database must be involved.
    • General scenario testing.  Any time it’s appropriate to pull a section of the application in and run a lot of classes together, this is an integration test.  It involves several parts of the system, not just one.  It can run fast if completely in process, or it can be slow if it requires an out-of-process call such as leveraging the file system.

This is not an exhaustive list, but it includes most of the automated testing on a typical enterprise application.  Feel free to comment with any type I may have left out.

Scott Bellware reasoned that the database needs to be left out for unit testing.  I completely agree.  Unit testing, by common definition, excludes external dependencies.  It’s not a unit test if we reach out and touch things.  When you have the right number of unit tests (for example, I’ve worked on a smart client system with 80,000 lines of code and 1300 unit tests and another 700 integration tests), you can’t afford to take more than a few milliseconds to run each one.  You need your unit tests to run very quickly.  Otherwise, you won’t run them very often.

Conversely, this doesn’t mean that the database should be ignored when testing a system.  There are plenty of reasons why a database, SQL, or stored procedures, triggers (shudder), views, etc can cause a bug in the system.  I insist writing an automated integration test for every database operation.  How else can we verify that the database operation works correctly?  We can’t.  It is important, however, for communication’s sake, to understand that these database-inclusive tests are integration tests, as are any tests that exercise an external dependency.

Automated testing with the database REQUIRES the following:

  1. Every developer has a dedicated instance of the database that can be dropped and created at will.
  2. Tests must be responsible for their own data setup.  An empty database should be all that is required to run the test.  The test must be responsible for adding data for the appropriate scenario before testing the scenario.
  3. You will want to generalize test data setup because it isn’t feasible to expect EVERY test to set up all the data.  A general data set that sets a base line of data is very useful and can be invoked with a data helper class.  Then each test can just add specific data necessary for it’s test case.
  4. Data setup, database creation, etc should be automated.  If it’s manual, it cost more, and you won’t run the tests as often.
  5. Database schema must be in source control with the code.  Without that, you never know what the correct version of the schema is.

Another of Scott’s points: “As a side effect of doing the necessary dependency injection, you often get a cleaner and more explicit separation of concerns – which makes software easier to change and maintain.”

He’s right.  If you can’t unit test your domain classes because everything you do with them requires a real database to be online, you have an indication that you aren’t separating concerns.  Data access should be independent of domain object behavior in most cases.  I should be able to verify that a Customer object can Sort() itself without invoking a database query, but if constructing a Customer initiates a database call, my domain model is then materially coupled to the database and needs to be separated.

Jeremy Miller is of the same mind in his comment: “Referential integrity, non null checks, and sundry other data constraints.  All good things.  All a pain in the ass when you’re unit test only needs a single property set on the InvoiceItem class.”

To help clear up some confusion with the term “unit test”, I propose a simple constraint in our dialog:  If the test calls out-of-process, it is then disqualified from “unit test” status and falls into “integration test”.  Feel free to argue in the comments. 🙂

Levels of automated testing within a single application

Conducting a Scrum Meeting

July 4, 2007

LINQ to SQL: Objects all the way down

June 29, 2007


There are a lot of different opinions on just what LINQ to SQL is or is not. Some of them are actually based on investigation by people who have kicked the tires and taken a test drive. Being its architect, I realize that I am the foremost expert on what’s under the hood, yet that alone does not give me the omniscience to know how the product measures up to everyone’s expectations. What it does give me is the ability to share with you the impetus behind the design and hope that gives you enough insight to make up your own mind….(read more)

LINQ to SQL: Objects all the way down

SQL Server Performance

June 13, 2007


The Crib Sheet