Development of Frooly began in June 2009. Knowing a few basic and very broad objectives, three of us began planning and designing the system on scraps of paper in a cramped messy office above a hairdressers in Rotherham. Debates raged, decisions were made, code was bashed out and progress was swift; Frooly was taking shape. In these beginning stages there was no project management beyond scrawlings on a whiteboard and no development methodology in place but, like any software project at this embryonic stage, there was little need for it. Objectives were simple and obvious (“let’s get it showing something on screen”, “we need to be able to add text, upload images”), communication was simple and free-flowing, testing was a synch –commit a bug on a small system like that and it takes mere seconds for the room to erupt in remonstration – and the management team (our “project owner”) were wandering in and out, in constant discussion with the developers.

At this early stage of its life, a software development project is the very essence of Agile:
  • Individuals and interactions over processes and tools - Check
  • Working software over comprehensive documentation - Check
  • Customer collaboration over contract negotiation - Check
  • Responding to change over following a plan – Good god Check
Not that we were giving even a second’s thought to agile or any other methodology at that point, we were just getting on with the task at hand and it was working wonderfully, great software getting written incredibly quickly.

However, as the team grew to four, the software grew in complexity and the business grew in maturity; it wasn’t long before our approach began falling short.

Around three months into the project, a working prototype was in place and the management team were getting a clearer idea of the direction they wanted to go. Objectives and requirements were changing seemingly by the day, our adhoc discussions would form the basis of developments and the results of those developments were increasingly met with responses like:
“Oh, I thought it was going to work like this...”
“So, you can’t do this with it?”
and “I didn’t think it was going to be able to do that.”
This situation persisted and grew for the next few months and was increasingly frustrating for all involved. Frustrating for the management team because it felt like they weren’t being listened to and frustrating for the developers because it felt like their hard work was going unappreciated.

Our effectiveness as a development team was gradually deteriorating and with hindsight it’s easy to see why. There was no sudden moment of clarity; various attempts to improve things were put in place with varying degrees of effectiveness: project management systems, more regular meetings, and more paperwork; but in the end it was a change in the ethos, approach and processes of the whole company that was needed. The deteriorating effectiveness of the development team was just a small part of that wider change which needed to happen: Frooly needed to go from being a small project in a couple of rooms above a hairdresser’s to being a software company with a clearly-defined core product.

Agile to the Rescue

By this point, Frooly had moved to a larger managed office in Sheffield city centre, taken on two more developers and begun its first closed beta. At the same time, I was reading Mike Holcombe’s brilliant book ‘Running an Agile Software Development Project’ and the situations and problems he talks about in that book spoke directly to what was happening at Frooly. Certainly I was not new to Agile, I had been a follower before, but it was my first experience planning and implementing an Agile methodology myself.

Usually when books and articles are extolling the virtues of Agile, it is invariably from the perspective of its use in a large existing organization, unseating an older, long-established methodology such as the waterfall model and the many improvements and efficiency savings it offers over the old way of doing things. At Frooly, my challenge was not to justify an Agile process over the waterfall model or any other approach, it was to justify implementing any kind of a process at all, since we had none.

Initially, management was reticent to accept the changes I was pushing for, this was a management team that had not come from a software development background and was not used to the particular problems we were experiencing, and as a result the solutions they came up with to the problems we faced invariably lead to further problems. For example, when developments were continually overrunning their deadlines, they responded, as one would do in a non-software dev context, by applying more pressure and incentivising deadlines. This however lead to sloppy, buggy code, since the issue wasn’t with the speed at which the developers were working, it was with requirements not being clearly defined and time estimates being effectively plucked from thin air. (Ah, hindsight...)

Frooly’s management team was concerned that the processes I wanted to put in place would unnecessarily slow down development; this was not helped by the damage done to their faith in the development team by the frustrations they were experiencing in its declining effectiveness.
Looking back now, I could’ve pushed harder and affected my changes sooner but who was to say that my approach would be more effective than any other? I was far from confident that what I was proposing would in fact be the answer to our problems any more than any other solution. However, after some (if not quite enough) pushing on my part, the continuing problems and some changes in the management team itself, it was agreed to give it a go.

Our Solution

The problem Agile sets out to solve is how to hold on to the effectiveness, speed and dexterity we see in the initial stages of the project when the product becomes more complex, testing must be more thorough, when there are complex dependencies, more comprehensive requirements and more people involved in meeting them.

The pace of change was not quick, it’s still happening now and I certainly hope it will continue to happen, but once in place the improvements were vast and noticeable.
Every morning the development team gets together for a scrum (well, most mornings!) to discuss our progress and objectives, share ideas and agree the direction of the software and our methodology.

Each week or so, the management team meets to discuss the upcoming developments, review progress and decide priorities. For some time, these meetings would be long, unstructured rambling beasts that would eat up hours. To address this, clear goals for the product and the business were established and proposed developments were simplified and formalised into a Pitch document, which could be submitted by anyone, and contains a few standard questions about the feature.
Meetings were now a simple process of going through the pitches, and evaluating each one against the product and business goals. As a result, the meetings proceeded much more swiftly, more was achieved and the meetings were required less frequently.

The particular features management wants to move forward with are turned into requirement specifications, which sound long and rambling, but in fact are just a short series bullet points in the form for user stories which describe the completed development. They look like this:
  • Users can login using their Facebook account when they have never logged in before and a Frooly Account will be created for them
  • Users can return and login again using their Facebook login and gain access to their existing Frooly account
These are signed off by the development manager (me), the project manager and the person who originally pitched the feature. This solves the problem which was probably Frooly’s biggest, of each person having a different idea about what was being developed.

A developer is then tasked with taking each user story on the requirements spec and summarizing the series of tasks which must be completed to meet that requirement (e.g. create a stored procedure, alter a class, etc...) along with a time estimate for that task. All of this is done in our project management system - TargetProcess, which will add up the total number of developer hours required for each feature, which helps in the management team’s planning and addresses the issue we had with unrealistic time estimates being attributed to developments.

New developments are grouped into a release which is deployed to a staging area, subjected to a rigorous test plan before being signed off and deployed.

Conclusion

At the moment we’re seeing new releases going out every three weeks or so (unless there’s a particularly nasty bug to fix), and they are more timely, more bug free, they meet requirements and the needs of the business - and for all our concerns about the process slowing down development, we’re finding they’re developed, tested and released far more quickly -which is what we were aiming for, getting back to great software getting written incredibly quickly.

We’re not done yet either. For a start we’re looking at ways of making our development process more test-driven, improving how our design team’s work is organized and a bunch of other improvements to boot – I’ll post more of our experiences on this blog.

Learnings from this experience:
  • As developers, we must all be ready and able to properly spec and organize our own developments - it is as much a part of our job as tapping out the code
  • We must be aware that as a development project grows, its organizational needs will change and we must be ready to respond appropriately to that change
  • We must be constantly looking for and discussing ways to improve our development process
  • We must be staunch vocal advocates for that organisation and continuous improvement; it is unreasonable to expect non-technical managers to intuit and organize the unique processes required for good development to work - it’s all on us.
-Any ideas and suggestions as to how we can further improve are absolutely welcome!

The Problem


Today one of my most talented developers spent the morning putting together yet another address form and it set me off wondering why, as developers, we seem to spend so much time writing out the same methods, concepts and programs; faff around with the same config files getting them to do the same thing and bang our heads against the wall trying to achieve the same system & server setup to run our code over and over and over again.


Assumptions

If a developer were to start a new development project today, what is it likely to look like?
  • It's probably a Web Project
  • It's probably hosted on a server with url pointed at it
  • It's probably going to need source control and backup
  • It's probably going to need a live version and one or two staging versions
  • It's going to need documenting
  • Generally speaking, it's probably going to consist of a series of common objects which can be both stored in a data store and represented on a page with get, create, edit, update and delete methods
  • More specifically, it's probably going to need a system of user authentication, with logins, registration, passwords, forgot password functions, etc.
  • it's going to need the ability to create groups for users and to grant and deny those groups of users access certain pages and resources
  • It's probably going to need outbound (and possibly inbound) email and if so it'll need a message queueing system
  • It's probably going to need controls for collecting commonly required data sets (and the ability to store them in the usual way)
    • Names
    • Dates
      • date of birth
      • start date and end date
    • Addresses (one or multiple)
    • Card Payment information (incorporating addresses)
    • Captchas
    • Blog-style posts
    • Comments
    • Events
    • Geo-locations
  • Input fields are going to need client side and server side validation
  • Should be able to collect and report on traffic statistics
  • It's probably going to need the ability to interface with commonly used frameworks
    • OpenID
    • OAuth
  • It's probably going to need the ability to interface with at least one or two of the more commonly used APIs
    • Facebook
    • Twitter
    • Windows Live (maybe)
    • Google Adwords
    • Google Adsense
    • Google Base
    • Google Maps
    • Youtube
    • Amazon ecommerce
    • Google Analytics
    • Flickr
    • Sharethis
  • Probably going to need to integrate with a payment service too
    • Paypal
    • Worldpay
    • Google Checkout
    • a few banks...
I'm just scratching the surface there, but you get the idea. These are all components that we shouldn't have to write; tweak maybe, extend certainly, but we shouldn't have to be writing these from scratch over and over.

The Solution?

So here's my vision: A programming environment which is itself web-based, the software is installed on the server (or even better, is pre-installed on the server) and all administration, setup and programming can be done via a web interface. It handles source control, backup, documentation, data storage, publishing (live versions and test versions - this should be really simple since the software is already online), access control and server setup through this interface; it does away with the idea of a separate database and programming environment, handling all data storage through the same interface to minimize time-consuming configuration and complications.

It should be able to handle all the basic data types and also provide objects representing the common collections of those data types (Names, dates of birth, Addresses, Events, etc.) and be easy to extend them, create new ones and share them.
In fact, the idea of sharing and integrating objects and code with others should be central to the design, so as well as providing a range of reusable objects like those listed in the assumptions above, it should be trivially easy to post a component online and for others to include it in their project (whilst maintaining proper attribution, of course).

Don't get me wrong, I can see that there are movements in this direction, ASP.Net provides ready made controls, simplifies the process of creating objects from your database, etc.; Ruby on Rails follows a philosophy similar to what I'm proposing, helping with the setting up of your database and corresponding business layer objects and utilizing syntax which minimizes the amount of code that needs to be written, and (although I haven't had a chance to play with it yet) I understand Python also offers an approach which is also very high level; however I can't say I've found a programming language which completely solves the problem of writing the same code over and over.


Simplifying and streamlining software development also helps address the problem of software creation being alien to most people, even though they use software all the time. But most importantly, by making software development less repetitive and laborious we can achieve greater productivity and get on with the more interesting jobs, you know, the ones where you actually have to think.
There aren't many things that make me feel warm and fuzzy inside, but Recursion and SQL are two of them. It was back in 2004 when, young and naive, I first had need of a technique for recursively querying a tree of data in SQL. It was what is probably the most common scenario requiring this technique, given a Product Category ID, return it and the IDs of all child categories so that I could look up the corresponding Products. After much research the technique I hit upon was to get a Stored Procedure with the help of a Cursor to execute itself for each level of recursion, then store the results in a temporary table.

While this worked fine at the time, when I looked at it again recently to address a similar problem, I recoiled in shock at how horribly slow and inefficient it was - unsurprising really. Temporary tables are horribly slow to create and query, and as a rule of thumb, whenever you're using a Cursor in a SQL query, alarm bells should ring, although occasionally necessary it tends to be a sign that you're approaching your query with a function-based programming mindset rather than query language.

So after a bit of research I found Common Table Expressions (CTE) - a feature added to Microsoft SQL Server 2005 (pretty sure I've seen it in Oracle SQL too) which offers an efficient way to create a named result set which can be referenced.
Since CTEs can be referenced, they can reference themselves, which means they can be recursive, and here's how it's done:

A recursive CTE is basically structured in this form:

WITH MyResultSetName (my, result, fields) AS
(
/* a select statement which gets our base query data */
UNION ALL
/* a select statement which gets the next level of recursion */
)

then we select from our result set

SELECT
*
FROM
MyResultSetName

Looks simple enough, how about an example...

Create our example table:

CREATE TABLE [dbo].[Category](
[id] [int] IDENTITY(1,1) NOT NULL,
[name] [nvarchar](50) NULL,
[parentId] [int] NULL,
CONSTRAINT [PK_Category] PRIMARY KEY CLUSTERED
(
[id] ASC
) ON [PRIMARY]
) ON [PRIMARY]

GO

Add our example data (a category tree):

INSERT INTO [Category] ([name],[parentId]) VALUES('All Products',null);
INSERT INTO [Category] ([name],[parentId]) VALUES('Televisions',1);
INSERT INTO [Category] ([name],[parentId]) VALUES('Laundry',1);
INSERT INTO [Category] ([name],[parentId]) VALUES('Small Appliances',1);
INSERT INTO [Category] ([name],[parentId]) VALUES('Plasma TVs',2);
INSERT INTO [Category] ([name],[parentId]) VALUES('LCD TVs',2);
INSERT INTO [Category] ([name],[parentId]) VALUES('Projectors',2);
INSERT INTO [Category] ([name],[parentId]) VALUES('Washing Machines',3);
INSERT INTO [Category] ([name],[parentId]) VALUES('Tumble Dryers',3);
INSERT INTO [Category] ([name],[parentId]) VALUES('Washer Dryers',3);
INSERT INTO [Category] ([name],[parentId]) VALUES('Toasters',4);
INSERT INTO [Category] ([name],[parentId]) VALUES('Breadmakers',4);
INSERT INTO [Category] ([name],[parentId]) VALUES('Blenders',4);
INSERT INTO [Category] ([name],[parentId]) VALUES('40"-50"',5);
INSERT INTO [Category] ([name],[parentId]) VALUES('51"-65"',5);
INSERT INTO [Category] ([name],[parentId]) VALUES('66"+',5);
INSERT INTO [Category] ([name],[parentId]) VALUES('14"-22"',6);
INSERT INTO [Category] ([name],[parentId]) VALUES('23"-40"',6);
INSERT INTO [Category] ([name],[parentId]) VALUES('42"+',6);
INSERT INTO [Category] ([name],[parentId]) VALUES('1000-2000 Lumens',7);
INSERT INTO [Category] ([name],[parentId]) VALUES('2000-3000 Lumens',7);
INSERT INTO [Category] ([name],[parentId]) VALUES('3000+ Lumens',7);
INSERT INTO [Category] ([name],[parentId]) VALUES('1000 Spin',8);
INSERT INTO [Category] ([name],[parentId]) VALUES('1200 Spin',8);
INSERT INTO [Category] ([name],[parentId]) VALUES('1400 Spin',8);

and now our example CTE:

/* SET UP THE TARGET CATEGORY ID FOR THIS TEST (SET THIS TO WHICHEVER CATEGORY ID YOU WANT)... */
DECLARE @categoryId int;
SET @categoryId = 2;


--AND HERE'S THE CTE...
WITH CategoryTree (id, name, parentId) AS
(
--THE BASE QUERY...
SELECT
id,
name,
parentId
FROM
Category
WHERE
id = @categoryId
UNION ALL
--THE RESURSIVE QUERY...
SELECT
c.parentId
FROM
Category c
INNER JOIN CategoryTree CT -- JOIN TO THE RESULTS IN THE CTE
ON c.parentId = CT.id
)

--FINALLY, SELECT ALL FROM THE RESULTS IN THE CTE
SELECT
id,
name,
parentId
FROM
CategoryTree

And here are the results of the query:

And there we have it, the joyous union of those two most wonderful things... Recursion and SQL, and best of all, it's FAST!

More on Recursive Queries Using Common Table Expressions here:

How not to import and export your blog posts

This was going to be a post about how to export and gather together the posts from the various blogs you've got knocking around on the internet and import them into Blogger (which I chose because of its reportedly excellent import/export facility) however I made quite a hash of it, garbled a few posts, lost a load of formatting and hyperlinks, had a little cry...

That'll teach me to be so promiscuous, running off with every trendy new publishing platform that comes around. It's time to settle down with a blog I can depend on, that's all I'll say since I'm not the best person to be advising anyone on how exactly to do that.
The Windows Communication Foundation (WCF) api has been a part of the .Net framework for a good few years now, and if you're a .net developer you've probably dabbled in it or at least heard the name batted around. It offers a handy and flexible set of tools for letting external systems and programs (regardless of language or operating system) run your code and receive data as a response, in a similar way to how you interact with Amazon Web Services or Google Adwords API.

Very useful if you're writing software for distributed systems, or perhaps if you're trendy enough to be writing iphone & android apps or even if you just want to make sure that your system is scalable longer term; but there's an incidental feature in WCF which is so useful, I'm now using WCF in every web project I write.

It's this little sucka:















The WCF Test Client - isn't it beautiful?

The WCF Test Client allows you to individually execute your WCF services in a similar way to how you would execute a stored procedure, supplying the data and receiving the response. You can see why it was necessary, you need to be able to test your services.
As a developer, it's an incredibly easy way to quickly test a piece of code I've just written, but as a development manager who has had many a night's sleep ruined by nightmares of messy code, it's game changing.

Why WCF is Inadvertently Brilliant

What I've taken to doing is setting up a WCF Service project as the gateway to the business layer (a class library project), in the same way we use Linq as a gateway to the data store, and setting up our presentation layer project (usually a website project, but could be a windows forms app, or an iphone app, or all of the above) to interact only with the WCF Service project.
This way, when planning a development, I can schedule the development and testing of the business layer code completely separately from the development and testing of the presentation layer, even to the extent that I can assign them to entirely separate developers.

For example, we need a webpage to show a list of customer orders, Geoff the efficiency expert can write the stored procedure and the code to organize the customer order data, it can be fully tested and Geoff can move on to something else; then Alan who knows his drop-downs from his radio buttons and better understands user behaviour can come in a put together an effective interface without getting bogged down in SQL quirks.
(Actually in this situation I'm more a fan of exposing developers to the areas of development they are less familiar with to give them more rounded experience - but you see my point).

This gives more flexibility in scheduling but more crucially, enforces a healthy barrier between business layer and presentation layer code - even if it was late at night and in a pizza & caffeine fueled daze Alan attempted to write some processing method in his presentation layer code, it wouldn't work (or make sense) since our processing code is already written and tested and the format of the input and output data is fixed - so the code stays clean, readable and testable, and the development manager sleeps soundly.
Every once in a while a programming task will turn up that pushes all your nerd buttons at once, recently I was treated to one such occasion. In planning a development it became clear that one element that was needed was a SQL function to calculate the distance between two geolocations. My gasp of joy and excitement at the thought of combining geometry and SQL seemed not to be shared by the team so I thought I'd reward myself with a bit of programming (a rare treat these days).

What a joy it was!
Here's how you calculate the distance between two geolocations in a SQL function:


To calculate the distance between two points on the globe we need to use the Haversine formula: http://en.wikipedia.org/wiki/Haversine_formula
This fellow has an excellent page dedicated to implementing the formula in Javascript: http://www.movable-type.co.uk/scripts/latlong.html - it is on this explanation I based my code

Here is the Haversine formula from the moveable-type.co.uk site:

R = earth’s radius (mean radius = 6,371km)
Δlat = lat2− lat1
Δlong = long2− long1
a = sin²(Δlat/2) + cos(lat1).cos(lat2).sin²(Δlong/2)
c = 2.atan2(√a, √(1−a))
d = R.c


It basically calculates the distance by looking at the angles between the points in relation to the Earth's core and then in relation to each other and then uses a bit of simple Pythagoras to figure out the resulting distance.

In my version, I wanted to use miles rather than kilometres (simple task of switching the constant used for the Earth's radius) and I wanted to use a more accurate figure for the Earth's radius.

Here's what I came up with:

CREATE FUNCTION [dbo].[GetDistanceBetweenGeoLocations]
(
@lat1 float,
@long1 float,
@lat2 float,
@long2 float
)
RETURNS float
AS
BEGIN
-- Declare the return variable here
DECLARE @distance float
SET @distance = -1
DECLARE @earthRadius float
SET @earthRadius = 3958.761 --miles
--6371.009 --km
DECLARE @axis float

if ((@lat1 IS NOT NULL) AND (@long1 IS NOT NULL) AND (@lat2 IS NOT NULL) AND (@long2 IS NOT NULL)) -- bit of protection against bad data
BEGIN
SELECT @axis = (SIN(RADIANS(@lat2-@lat1)/2) * SIN(RADIANS(@lat2-@lat1)/2) +
COS(RADIANS(@lat1)) * COS(RADIANS(@lat2)) *
SIN(RADIANS(@long2-@long1)/2) * SIN(RADIANS(@long2-@long1)/2));
-- Multiply root of the axis by the earth's diameter
SELECT @distance = @earthRadius * (2 * ATN2(SQRT(@axis), SQRT(1-@axis)))
END


-- Return the result of the function, rounded down
RETURN ROUND(@distance, 3)

END


As a test, we will work out the distance between Big Ben and the Eiffel Tower - it should be 211.47 miles according to this site: http://www.trueknowledge.com/q/distance_from_big_ben_to_eiffel_tower

Big Ben geolocation: 51.5005, -0.1245 http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=51.5005,+-0.1245+&sll=53.21768,-103.452893&sspn=93.469044,270.527344&ie=UTF8&z=16
Eiffel Tower geolocation: 48.8582, 2.2945 http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=48.8582,+2.2945&sll=51.500874,-0.12447&sspn=0.011661,0.033023&ie=UTF8&z=16

Our test code:

SELECT
[dbo].[GetDistanceBetweenGeoLocations] (51.5005, -0.1245, 48.8582, 2.2945)


Our result:






Close enough, lets blame them for the difference.
Lately I've been experimenting with a technique for automatically converting ASP.Net postback events (the onclick event of an ASP.Net button initially, but there's no reason it would work with the postback event of any ASP.Net control) to Ajax calls using JQuery, much lighter than the calls made using Microsoft Ajax too. I think it should work however it's not fully implemented yet.

Here's a brief overview of the technique...

The way we do this on Frooly is a little more complex but for the sake of example, lets say we wrap all our controls in div of id "wrapper" followed by a unique ID, and include a hidden field containing our control address, Like this:
<div id="wrapper">
<asp:HiddenField ID="HiddenFieldControlAddress" Value="Controls/CreateProduct.ascx" runat="server" />
</div>

use a JQuery event to intercept the postback (in this case an ASP.Net button click), and then grab the id of the submitting button, the id of the wrapper div, then grab the id, the (URL encoded) value and the control address of each of the input controls within the wrapper , before sending them via the usual JQuery ajax call to our receiving method on server side
$("div[id^='wrapper]").find(":submit").click(function() {
//Grab the ID of the button
var submittingControlId = $(this).attr("name");
//grab the wrapper ID
var wrapperId = $(this).closest("div[id^='wrapper]").attr("id");
//child inputs
var valueString = "";
$(this).closest("div[id^='wrapper]").find("input").each(function() {
valueString = valueString + "&i=" + $(this).attr("id");
valueString = valueString + "&v=" + URLEncode($(this).val());
valueString = valueString + "&p=" + $(this).closest("div[id^='cntItem_']").attr("id");
valueString = valueString + "&t=" + $(this).closest("div[id^='cntItem_']").children("#cntrlAddr").val();
valueString = valueString + ",";
});

//Send via Ajax...
TryAjaxCall(wrapperId, submittingControlId, valueString);
});


Then, after receiving our data, we go through it, re-instantiating our controls using
Control.LoadControl(controlAddress);
I'm skipping over a lot here, but basically for each control address we collected, we repopulate the data using the corresponding IDs and Values, then call our the event using reflection - the reflection bit is a bit complicated but it looks something like this:
System.Web.UI.WebControls.Button button = (System.Web.UI.WebControls.Button)control;
Type t = typeof(Button);
object[] p = new object[1];
p[0] = EventArgs.Empty;
MethodInfo m = t.GetMethod("OnClick", BindingFlags.NonPublic | BindingFlags.Instance);
m.Invoke(button, p);

Having triggered the event, we render our parent control markup to a string, return that string via Ajax to our page and replace the markup of the parent control on the page.

I've been deliberately vague on the detail while I figure out all the niggles, it's at a very experimental stage at the mo, but I thought I'd talk about it since I've found no examples of anyone doing anything similar.

I'm concious that it's not always an appropriate option but if your web application builds its pages from individual WebControls (as Frooly does) it offers a number of advantages:
  • it allows you to perform a call which only posts back the contents of that control and in response only updates that control (keeping the calls nice and light)
  • it requires no extra work on the part of the developers when creating new controls to enable Ajax
  • it still works in postback mode if Javascript is turned off
  • we can automate pretty loading animations
  • best of all since we're controlling the entire transaction end to end, we can automate our client and server side validation
Has anyone else tried anything like this?
top