By Alan on 2010-02-08 07:53:48Z

Before I go and make a claim like I can invent some new pattern language that best serves the purpose I’m striving to fulfill, I thought it would be only prudent to look around and see whether anyone else has already done this.

So far, I haven’t discovered anyone doing it. Not much else to report, so these posts will be short for a while.

By Alan on 2010-02-06 11:32:18Z

I’m trying to lay my hands on a copy of the book A Pattern Language: Towns, Buildings, Construction right now.

Meanwhile, I’m going to conjecture about how a pattern language might be better constructed so as to significantly improve the ability for the person who wants to create some software to communicate with the person who wants to build some software about the (and I’ll use this word loosely) architectural vision of the software to be created.

In other words, it’s all well and good for patterns to be used in the design and construction of software. I’m not saying the Gang-of-Four got it wrong when they began identifying patterns for use in the design and building of software. In fact, if I recall, they specifically admonish readers not to stop with their patterns but to build upon them and create an ever larger vocabulary of patterns.

Turns out this can be very difficult to do. But let’s try, anyway, and see what we come up with.

Typically, software is classified by the people who create it. That is, we have embedded systems and operating systems. We have enterprise information systems, applications, games, libraries, frameworks. Certainly we have others. I’m most familiar with enterprise information systems (EIS), but I’ll maintain that the technique I’ll try to apply can also apply to other classes of software.

Within each of those classifications are further classifications. Within EIS, we have database management systems, messaging systems, logging systems, operations management systems, … you get the point.

I’m going to play with the building construction analogy to see if it works.

Operating systems are like public infrastructure. Roads, bridges, levees, fresh water, drainage and sewage, sanitation, etc.

Embedded systems are like private, miniature versions of the public infrastructure, along with the buildings connected to that infrastructure.

Enterprise information systems are like corporate facilities and all the means of connecting them together. Large skyscraper headquarters buildings, warehouses, factories, campuses, distribution centers, leased office spaces, shopping centers, retail stores, and of course fleets of cars and trucks, jets, helicopters.

Applications are like houses, apartment buildings, hotels, garages, mobile homes, recreational vehicles, campers, and tents.

Games are like amusement theme parks, theaters, arcades. cruise ships, restaurants, casinos.

Libraries and frameworks are like laws, rules, standards, traditions, building codes, common practices, policies.

Social network applications are like television, cable, radio, telephones, walkie-talkies, intercoms, and the post office.

So, if I were going to try to describe the application I think my wife was trying to describe, then it would need to feel like I was trying to describe some kind of structure in which people would live and keep their stuff.

Was she describing a hotel or a tent? There’s a very large difference in the scale and scope of the two, even though functionally they both serve similar purposes.

She said she’s spend $20,000 to have it made. You can’t really by a house for $20,000, even in this economy. And yes, I know I’m comparing apples to oranges. But $20,000 indicates not just what the market price is for property; it’s the value that we place on whatever we’re getting. I’ll leave alone for the moment any calculations of rate of return; I’ll have to come back to that later.

For $20,000, you could probably get a used pickup truck with a camper mounted on it, or an exceptionally nice tent. Neither one is quite exactly what you want to spend $20,000 on, though, right? The pickup truck camper will be tiny, cramped, and have been designed and built for someone else. The tent, nice though it may be, is still a tent with thin walls, only so much protection from the rain or predators, or places to hang your pictures. And don’t forget it’s still not connected to drains or sewers, or even electricity (although, for $20,000 you’d think it would come with a generator—and even that will be noisy).

I’ll keep sussing out the metaphor tomorrow.

By Alan on 2010-02-05 08:16:47Z

Partly because I am trying to solve with my right-brain what I mean when talk about patterns the way I think Christopher Alexander meant them to be applied in fields like computer science. (As I stated previously, the way we do it most of the time is definitely not what he meant.)

And …

Partly because I stayed up too late last night on my guilty pleasure: catching up on past episodes of Lost so that I can tune in to current episodes. So sue me ;-).

By Alan on 2010-02-04 07:29:02Z

While I’m not the foremost expert on patterns, I have a little familiarity with them. Chiefly, I’m aware they were adopted and adapted from a work by Christopher Alexander, an architect, in which he attempted to create a language with which anyone—not merely architects and builders—could efficiently and effectively communicate about built aspects of our world, and what works well, and why. And I’ve read many books which more or less acknowledge Alexander’s influence on their own ideas.

And I’m left with this thought. The scale is way, way the hell off.

Alexander promoted patterns for a metropolis, and patterns for a small part of a building, and everything in between.

But most software design patterns are focused around the code or processes used to create software.

To me, it’s as though software design patterns, if they were directly transferred to their counterparts in the building and architecture world, were describing minutiae. The seminal work, the Gang-of-Four’s book Design Patterns: Elements of Reusable Object-Oriented Software, focuses on object-oriented coding patterns. In buildings, that would be like patterns for attaching boards to each other, for leveling a floor, for digging a square hole in which to put a foundation, for connecting ducts, for making sure screws and bolts turn “righty-tighty, lefty-loosey.”

If I’m about to buy a house, I really don’t care that much about those things. That’s stuff that the carpenters, the masons, the plumbers, and the heating/cooling technicians need to know. Should they know those things? Sure! I’m not saying patterns at those levels aren’t important. But they’re worthless for me, as the user of the house, to know about.

If I’m about to buy a house, I really need to be able to evaluate a house or express my ideas for a house in a much larger, much broader way that really doesn’t care about how it gets done or what construction patterns were used. I need a language where I can share my thoughts with the architect or with the general contractor.

We have no such language right now. When I try to describe potential software solutions to people, it’s almost like I’m trying to describe the materials or processes by which the building gets built, but not really the functional aspects of how it will work for the user. Or, I’m describing the functional aspects of the software, but it’s not clear from that description whether I’m describing something simple, commoditized, and cheap like a kitchen sink (“the function is to wash dirt off hands, dishes, and food”), or something large, intricate, built-to-order, and expensive like an automatic cat and dog groomer (“the function is to wash dirt off dogs and cats”).

We need that language in order to more effectively discuss solutions with our customers.

By Alan on 2010-02-03 07:35:56Z

I was trying to explain to my wife what I felt was a need for a better way to customers and developers to share ideas and reach a mutual understanding that could realistically lead to a plan.

To make the point, I asked her to think of an idea for new software that she would find valuable. Right away, she said that a program that managed her food inventory would be useful. She even knew some of the features it should have. It would remind her when something is running low. It would update itself when she took something out of the pantry or refrigerator. She could quickly check to see whether she had all the things she needed for a given recipe.

At this stage, she certainly doesn’t care what it looks like, whether it’s on the Web, or an embedded device, or whatever. She doesn’t care whether it’s Windows or Mac or *nix. She probably does care about how she’d have to interact with it, but in her mind, all she knows for sure at this point is that it should be easy and quick and as much of it should be automated as can be.

To get back to my point, I claimed that I could think of a spectrum of solutions varying quite widely in expense. At the low end would be a Microsoft Excel spreadsheet, with a few special formula cells and VBA modules, costing a few hundred dollars to develop. At the high end would be something like MIT Media Labs context-aware kitchen, or any one of a variety of high-tech kitchens, costing hundreds of thousands if not millions to develop.

I said it was important to make sure she got value back out of the software. She said she thought it could be worth $20,000. I’m sure she meant Monopoly money, because there’s no way she’d actually fork over that much for a food inventory program. For the sake of argument, though, I took her at her word.

And then we reached the crux of the matter. It was certainly possible to create something for that much money that at least partly reached the functionality she wanted. But neither of us knew whether the final result would be actually valuable to her, enough that she would feel that her money had been well-spent.

Had I been talking about something that she understood better, she would have been able to quickly evaluate a plan. Instead of new software, I might have suggested buying a kitchen makeover, or a visit to famous kitchens around the world, like Julia Child’s kitchen in the Smithsonian’s museum, or several years’ subscription to a home food delivery service. We’ve never bought any of those, so we have no direct experience to apply in determining whether any of those things would be worth $20,000. But we can pretty easily and quickly make that evaluation anyway. Why? And why can’t we do the same thing with software?

By Alan on 2010-02-02 07:12:32Z

Like we don’t have enough languages to work with in software development. There’s the language of the business, there’s the jargon we use, there’s the handful of programming languages. Things like domain-specific languages (DSLs) are gaining in popularity. Do we really need another language? Did I really need to use a rhetorical question just there?

Actually, I don’t know. I’m exploring this in my mind right now. Here’s what I’m thinking so far.

I tried to explain the problem I think we face when we’re in the envisioning process of creating software – agreeing to fulfill a fairly specific outcome on a budget without being certain what the final result will really need.

I know that more experienced software developers can use their previous projects as frames of reference from which to cull an estimate for a new project. We know generally how long it takes for us to design the code, write and test the code, assemble and deploy the code. We know what parts to put under the covers that we’ll need when we find out something has to change. We know what parts to put in that the customer never would have thought to ask for but are essential to having it run well. And so we can look at a new project and make off-the-cuff estimates.

But we face all sorts of complications going down this path. Rapidly changing business concerns. Parkinson’s law. Minor catastrophes, like having a key developer leave the team early in a project.

And I’m not convinced that either traditional or agile developers really have cornered the market on a way to get the customer to really understand what he or she is buying, how much it will cost, and when they will have it before he or she makes the commitment to begin the project.

I’m going to spend a little time reading up Steve McConnell’s works and seeing what I can apply here. But Steve focuses largely on software estimation. That’s only half the picture. The other part is getting a clear vision back into the customer’s head about exactly what will be produced so that everyone is happy and calls the outcome successful.

By Alan on 2010-02-01 07:16:09Z

When trying to come up with a solution to a problem, the technique I’ve been taught most of the time, but not necessarily the one I see employed most of the time, is to brainstorm first. Perhaps using mind-mapping to capture the thoughts and suggest new ones. Then, once the ideas have been captured, prioritize them and come up with a plan.

That’s not a bad way to start, but doesn’t really solve the more intractable problem. What if the problem being solved isn’t the right problem?

Consider our two earlier people container developers. I think both of them came up with what would ordinarily be regarded as reasonably smart, clever, and cost-effective solutions for the problems they were trying to solve—protecting a man and woman from predators with a budget of about $10,000 and a schedule of three months. The traditional developer came up with a metal cage, and the agile developer came up with a large upside-down bowl woven from saplings surrounded by bells on wires. The agile developer would have criticized the metal cage as having been over-engineered. The traditional developer would have criticized the wood-and-leaf bowl as being substantially less appropriate for the customer’s long-term needs, or for larger projects.

However, both projects were, in my opinion, over budget and over schedule. In neither case did the customer have the basic need truly met by his deadline and without asking for more money. The traditional developer simply underestimated his job; that happens very frequently with software development. The agile developer technically met his deadline, if we use an extremely generous perspective, but exceeded his budget. (I believe agile developers would generally try to avoid being stuck with a long-term deadline.)

Is there a way to get the customer to express his needs to either developer, or both, in such a way that invented solutions could be quickly estimated fairly accurately? I’m not sure there is. How can anyone know what it will cost to solve a problem which hasn’t really been asked clearly yet?

I should point out about now that I will have to totally abandon the metaphor linking software development with building construction. Software is a continually evolving mechanism, having to deal with extraordinary changes to its environment as well as dramatic changes (or clarification) of the customer’s needs, whereas buildings tend to need much less evolution over time.

What could we compare it to, then, in order to learn from previous experience?

Art? No. While its true that the production of an idea generally ranges from moments to days, and, more rarely weeks, the gestation of that idea has no direct correlation. It could come in a flash, it may never come at all.

Medicine? No. Medicine is chiefly concerned with diagnosing, treating, and researching problems. The treatments would be most closely associated with creating software, but rarely are treatments an enterprise in constructing something completely new from imagination.

Producing movies? No. While the creative elements of making a movie are difficult to predict—getting the script right, finding or making the music, set design, etc.—the structural process of creating a movie remains generally the same from one movie to another, large or small, long or short.

It’s a little like all fields of work, and yet not quite like any of them. There are creative aspects, even when we assemble enterprise information systems. There are basic constructive aspects, even when we’re experimenting with prototypes. There are diagnostic aspects, even when we’re creating greenfield software.

I suspect that the eventual change that will have to happen is that we begin systematically, professionally, and uniformly presenting the creation of software as a completely different endeavor than is traditionally used to solve problems like constructing a bridge, publishing a book, fixing a watch, whatever.

We really are tasked with reading the dreams of our customers, understanding where the opportunity is present, suggesting ways to solve it while having a very good ability to say what the costs of solving it should be.

Makes me wonder if I should have been a professional bowler, instead.

By Alan on 2010-01-31 12:25:57Z

So what was my point over the last three days? Basically, that the people who will be paying for software still don’t have a very effective way of understanding what they’re asking for and how much it will cost and how long it will take.

If you’re an agile developer, you probably read what I wrote as though I were more critical of agile developers than traditional developers. Conversely, if you’re a traditional developer, you probably read what I wrote as though I were more critical of traditional developers than agile developers. I hope that’s what happened, anyway.

The truth is that I think both ways have a fundamental failure at solving the basic reason many software projects are still over budget or late. We all that that projects, by definition, involve imposing a change in the status quo, and therefore carry risks. But every project carries this risk. And yet many projects are successful, while many others are not as successful, and still others fail outright.

In general, when money is committed to constructing something, there is a plan to recover the cost, plus some profit. Perhaps by selling the thing, or by leasing it. Buildings, cars, watches, beds, traffic lights, whatever. The important part is to make sure that, on average, the costs are kept below the amount of money you’ll demand in exchange for the product. It’s usually not all that difficult to determine what someone would pay for a building, a car, a watch, a bed, or a traffic light. All of those things are commodities. And while creating a valuable label, or brand, for those products helps you charge more (thus allowing you to either profit more or to increase the quality of the product), they rarely allow you to be capricious when setting a price. For example, while The Coca-Cola Company certainly charges far more per can of cola than the pennies it costs to manufacture it, they will still be bound to prices similar to what PepsiCo charge, and to a lesser extent what other beverages manufacturers charge.

Customers know what to expect when they open a can of soda, regardless of the brand. They know what buildings should be able to do, what cars are capable of, the core features of a watch, their requirements for beds, and what constitutes a minimally-functional traffic light.

But software is another matter altogether. You and I, as software developers, know that these beasties can do anything. Anything!! Even the most ambitious types of software developments to date, around artificial intelligence, are not constrained so much by the limits of software as they are by the hardware. Only so much memory can be made available, the CPU requires so much time to process its instructions, the bus can only transfer so much data at a time…. Imagine how much closer we would be to artificial intelligence if we had networks of computers that performed 1 trillion teraFLOPS (amusingly called a yottaFLOPS), and if they had instantaneous access to 1 trillion tebibytes (called a yobibyte)? With that kind of power, it may even be possible to mimic intelligence using a program written in COBOL, although it would undoubtedly take many years to write such a program.

So, the challenge really comes down to this. Every new software creation is a new invention as yet unimagined by the people who want it, of complexity and sophistication which is incomprehensible to most people.

I wager that most people could think of a hundred ways that they would like to improve the tools they use, but only if they were free from the constraints of physics. Imagine a box that held more on the inside than it consumed on the outside. Imagine a battery that lasted for a thousand years. A knife which could cut through a steak but not a even scratch your finger. Now, imagine being asked to estimate the cost and schedule for a project to create something like that. You’d have no idea!

Every software project is a little bit like trying to create what seems like magic. All of software is, in a sense, imaginary. You can’t actually touch software. You can’t weigh it. It can’t wear out with overuse. And it costs nearly the same to make billions of copies of it as it does to make just one copy (if you’re not transmitting or storing those copies inside other things that cost more money per copy). And even when you’re creating software by assembling other software creations, we’re still largely working with tools whose existence is almost entirely within our imaginations, whose capabilities are limited only by our imagination—which is to say, not really limited at all.

So how do we get clear about our imaginations? How do we clearly share our expectations of some imagined solution that itself is created from other solutions which were created from imaginations? And, more importantly, how do we balance our imaginary creations with a very real limit on the ability of developers to fabricate the creation?

I think I’ll spend the next few days on this blog addressing this. I know there are many techniques for getting requirements, but maybe not so many on how to get the right balance of limits on the costs with what is possible in our imaginations and what is possible with the skills of the developers. I don’t know whether I have an answer, but I am very sure it’s a question that must be answered eventually.

By Alan on 2010-01-30 12:49:53Z

Yesterday, I imagined a dialogue between a customer consulting with an engineer/architect as it might be applied to constructing a building. Only I reversed the order of history, and wondered how it would be different if the science and art of building construction was as new as software engineering. Yesterday, we listened to the conversation between the customer and a traditional developer (of buildings). Today, we’ll listen to the conversation between a different customer and an agile developer.

The Agile Process applied to People Containers

Customer: “I’m getting married in three months, and I need you to put together a container where I can protect my wife and myself from the wolves and mountain lions. But first, I’ll have to get money from my dad, so I need to know how much this will cost.”

Agile Developer: “Well, we know that upfront designs are usually estimated poorly. I know about this one developer that was asked to build a people container, and he made a big upfront design—the old-school way—and when they finally delivered the container, that customer was really upset to find out that he’d bought a big hanging birdcage to put his family in! It was a total disaster. The customer had to get it lowered from the tree, and cover it with wood panels, put in vents, and so on. The original estimate was $10,000, and he wound up paying three times that, and not only that, but the project took twice as long as the original estimate of three months. So, the way I work is this: We’ll pick a period of time, let’s say about two weeks, and each period we’ll commit to completing something which is in line with your needs. Each period, you’ll get a clearer idea of what we’re building, and you can refine it until it’s exactly what you need.”

Customer: “What? Why can’t I just tell you what I want, and then you can go away and make it? You don’t really need me to help make it, right?”

Agile Developer: “Wrong. I totally need you to help make it. In fact, I will predict that if I do it that way you just suggested, you’ll get something that isn’t quite right, and then you’ll have to spend more money than you planned to fix it. Look, I’m not saying you need to do any of the actual construction. But every day, we’ll get together for a very, very short meeting so you can see how we’re doing and meeting our goals. Every two weeks, we’ll actually have a finished product. Now, naturally, the first couple of times we do this, the finished product won’t really be useful, the way the final product will, but it will definitely be a step in the right direction, because you’ll be there to help us understand what it is you really want.”

Customer: “That sounds great and all, except for one thing. I don’t know what I want yet. I just know what I want it to do.”

Agile Developer: “Perfect! That’s all we need from you. You tell us all the things you need it to do. Then we’ll prioritize, and make sure it takes care of the most important things first. When you think it’s finally doing everything you asked, then it’s done.”

Customer: “So it could go on for a really long time?”

Agile Developer: “That’s up to you.”

Customer: “But I said I need it before I get married in three months! How do I know it will be done before then?”

Agile Developer: “Because you are really the force driving what we do in each two-week cycle. You know your priorities better than we do, so you can make sure that each two-week’s goals will get us closer to your final goal, and since you know your schedule better than us, too, you can decide which goals to keep and which ones you can do without.”

Customer: “Yeah, that makes some sense. Only, how much is this going to cost?”

Agile Developer: “Again, that’s something you control. You should know how much value you need to get from this container."

Customer: “You mean, like how much this is worth to me? Well, see the problem is that it’s worth everything to me. It’s for my wife. But like I said, I’ll need to get the money from my dad. He’s an old-fashioned guy, and has spent his whole life in caves or up in trees. He thinks people containers are just really expensive problems waiting to happen, but we’re running out of room in the cave, so he’s agreed to let me do this. But he says I have to keep the cost down. All my friends tell me that agile developers are great at that.”

Agile Developer: “I see. So, then we have to stick with how much money you think you can get from your dad.”

Customer: “Uh… yeah, only I’m not sure how well he’ll take me saying ‘Dad, I need money, only I don’t know exactly what I’m going to get when it’s done.’ “

Agile Developer: “Well, look, you have a fixed deadline, right? And our average rate is about $1,000 per week. You want to finish just a little early, rather than late. So we can plan for ten weeks, five cycles of building. That’s $10,000.”

Customer: “That helps a little. But I still don’t know what I’ll get, other than that I’ll know what it will do, right? My dad is going to want to know I’m getting something more than a couple of those wooden boxes like they use for burying people.”

Agile Developer: “I can see you’re a little apprehensive about this. But rest assured, even by the first or second cycle, you’ll have a much clearer picture of what you’re getting than you could have right now, even if we tried our best to figure it out now.”

Epilogue

Near the beginning of this conversation, I suggested an ending to the story with the customer and the traditional developer. It wasn’t very flattering to the traditional developer, but the truth is that most traditional software projects are delivered over budget and late. See almost any of Steve McConnell’s works if you would like to see abundant research supporting that claim.

But what was the outcome of the agile developer? Bear in mind, both of these stories are totally fabricated from my imagination, but I believe I can predict a likely outcome.

Before beginning the project, the customer was asked to list his priorities for success. He came up with this list:

1. Big enough for him and his wife
2. Protection from predators like wolves and mountain lions.
3. Insulation from the elements (rain, cold, sunburn, wind)
4. Comfortable enough for sleep
5. Could be expanded for kids

Iteration 1: The team decided that the “big enough” requirement was largely an architectural requirement, and not something they could actually build and deliver on its own. However, they could certainly start to deliver something that offered protection from predators. They agreed, after discussing the possibilities amongst themselves and the customer, that an obstacle like a very high wall or very deep ditch could not be built in time and would be prohibitively expensive. The suggestion was raised to create an alarm system instead which could wake the customer (if he was sleeping) with enough time to take action. The customer reluctantly agreed, since he couldn’t really afford what he wanted.

The team committed to a “continuously constructed” tactic. Rather than having large chunks of work in an “under construction” state, they planned to have an actual container by the end of the first iteration, and to keep it inhabitable every day. By the end of the first two weeks, the team had built a meter long section of bells hanging from trip wires, and had placed a large flat rock on the ground about 50 meters from the bells. The customer was able to stand on the rock and hear the bells, and judged that he could probably keep his sleep light enough to be awakened by the bells and take action. The customer was both a little perplexed and pleased when the team celebrated their success at creating an inhabitable container. It didn’t look much like a container to him.

The cost for this iteration was $3,000, 50% more than what was originally estimated, but that was because $1,000 went to securing the large area of land (10,000 square meters, or about 2.5 acres) where the container would be built. The team assured the customer that they were making great progress and that, in all likelihood, they would finish under budget and ahead of schedule.

Iteration 2: Part of the team continued working on the alarm wires and bells. They needed to build over 300 meters of wires to create a circle that was at least 50 meters from the center. They determined they were completing 10 meters a day, so they focused on working the part of the circle deemed most likely to encounter predators, and set the goal for this iteration at 100 meters (they were not working weekends!).

Meanwhile, the rest of the team focused on the container. Clearly, the rock would not work to insulate the customer from any elements, but it would work well to anchor the actual container. The team and the customer discussed ways to insulate a person from the elements. The rain was seen as the hardest and most important problem to solve. Several members of the team had experience working on water container projects, designed to catch and keep rainwater for later use. They knew the best way to create a large water container was to dig a large hold in the ground and line it with something waterproof. The best waterproof material created so far was concrete. Unfortunately, it would be too expensive and difficult to apply for this project. But the team felt that the same principle could be applied here. The plan was to create a large upside-down bowl, and cover it with a waterproof liner.

So the team, sticking with their “continuously constructed” plan, began building the frame of a large, upside-down bowl. The goal for this iteration was to have the general frame of the bowl complete. At the end of the two weeks, they had created a circle three meters wide of stripped saplings on the ground lashed together with rope, and three arcs of saplings tied onto the circle and joining at the top. The customer could now begin to see something that seemed a little like a people container, and was pleased.

The cost for this iteration was $2,500, or 25% more than what was originally estimated. $500 went to purchasing stripped saplings, because the team did not have time to cut and strip them themselves. So far, the customer was delighted that he chose an agile developer.

Iteration 3: The wires and bells team had built 100 meters of the alarm system. With a little over 200 meters to go, they predicted they would speed up their progress with experience and could complete it by the end of the next iteration, but only committed to completing 100 meters this iteration.

The container team set a goal of completing the upright arcs of saplings around the circle. They were still trying to decide which waterproof material to use. They’d considered tanned animal hides, large tree leaves, and woven corn husks. They settled on tree leaves as the most practical given the time and expense allotted. However, the largest tree leaves available were only a little over 1/4 meter large. The team realized that meant the arcs around the circle would have to be spaced about 1/4 meter apart. With nearly 10 meters around the edge of the circle, that came to 40 half arcs, or 20 full arcs.

Three full arcs had already been built, so seventeen more needed building. It was taking the team a full day to build each arc, and they still needed to start fastening the leaves to the arcs. So they cleverly decided to start attach each new arc within 1/4 meter of an existing arc. That would allow another member of the team to start attaching leaves to a pair of adjacent arcs sooner than if they had waited until all the arcs were in place. They committed to building nine arcs for this iteration, leaving the remaining eight for the next iteration.

Unfortunately, there were not enough people on the team to work simultaneously on the wires and bells, the arcs, and the leaves. And they still hadn’t tried to tackle the requirement for comfortable sleep. They couldn’t push out the schedule, so they decided to hire more people to fasten on the leaves. The new team was brought on board and shown what to do.

The cost for this iteration was $3,000. The unplanned $1,000 was used to bring in the additional workers to keep the project on schedule. Nevertheless, the customer was getting very excited. He could see the outline of his new container being formed. He could hear the alarm bells periodically tested. He started telling people what a great move it was to use the agile developer. He had to spend some time convincing his dad that it would be worth more than the $10,000 he’d originally agreed to, but because he could actually show some good results every two weeks, his dad agreed to another $5,000. After all, every day, he’d been able to walk inside this new container and imagine living in it in safety and comfort with his bride.

Iteration 4: The wires and bells team completed 105 meters during the last iteration, and felt they could continue to accelerate their pace for this iteration. So they set a goal of 110 meters, which would bring them to completion.

The container team had completed twelve arcs so far, and had planned to complete the remaining eight this iteration. However, as they began, the customer noticed a new requirement that he hadn’t noticed before. He needed a way to easily get into or out of the container. The agile developer took this problem in stride. It was exactly to address this sort of thing that agile development had been created. Everyone had known that the customer would probably get clearer about his requirements as he was able to more clearly see the solution every day.

So the team brainstormed, and came up with a few solutions. One was just to cut some arcs a little short of where they would come down to the circle, but nobody was sure they could properly fasten a beam that would hold them in place since they wouldn’t be fastened to the circle at the bottom. Another was to bend the arcs at one section so that they left an opening wide enough to walk through. A third suggestion was to dig a tunnel under the circle through which the customer could crawl into and out of the container; it was discarded as soon as someone pointed out that it would fill with water at the first rain. The best and cheapest option seemed to be leaving a gap around the outside by bending the arcs away from the opening.

A few big problems came up during this iteration. First, during one of their daily meetings, the team realized that the leaf-fastening team was going to have a problem. The arcs were parabolic, and fastened to a circle that was three meters wide. While this had created a serendipitous outcome of yielding a stable structure, it meant that the top of the structure was about four meters off the ground. The leaf-fasteners had been working from the base of the structure, and working up as high as they could get—about two meters. They would need to find a way to reach the higher parts of the arc.

Second, the wire and bells team were having another issue. Birds kept landing on the wires or the posts holding up the wires and ringing the bells. They’d been continuous testing the bells to make sure they kept working, but hadn’t foreseen the need to test against false alarms.

Third, just as the arc-fastening team was fastening the fourth of the last eight arcs, some things happened unexpectedly and suddenly. At the top of the structure, the arcs had been merely laid across each other at the center where they all crossed. They were too far off the ground to lash them together. Now, with sixteen saplings laying on top of each other, a few slipped sideways a little. With all the tension present in the bent saplings, and with the pulling apart of some of the arcs to make space for the opening, what had been a flat circle on the ground suddenly became a wobbly oval. Furthermore, the gap for the opening was folded in, and what had been planned to be a meter-wide opening was now about one-half meter wide.

The cost for this iteration was $2,000. Although no new unplanned costs came into play, the customer was a little alarmed to see that he wasn’t as close as he thought he was to the outcome he’d planned. It wasn’t making him feel much better that the agile developer seemed not at all upset by the setbacks. The agile developer reminded the customer that all projects encounter setbacks; what made agile better than traditional methods is that the customer and the agile team could more easily adapt and respond to these setbacks appropriately.

Iteration 5: With only a month to go before his wedding, the customer was getting nervous. The alarms bells were going off all the time, even when they shouldn’t, while the wires and bells team tried to figure out how to solve the false alarm problem. The top half of the container was still uncovered, and the only plausible way to cover it would be to roll the whole container onto its side, which presented an additional risk of bending the container even more than it was. And the container would probably never sit completely flat on the ground.

Iteration 6: The wires and bells team continually tried to make the case that they’d met their requirement, and that the new requirement to avoid false alarms would need new planning, more money, and probably a newer technology.

The leaf-fastening team came up with a clever way to fasten the leaves from the inside of the container by constructing a crude scaffold. They managed to complete their task, but everyone on the team noticed that the leaves were beginning to dry and would become brittle before long. While they would still keep out the sun, rain, and wind while brittle, they would clearly not last more than a few months. They said nothing, deciding to let the next team deal with the customer when he noticed that he would have a requirement for durability in the waterproofing.

The arc-building team, which was still the core team, worked long and hard (including weekends) to fasten lashings across the arcs inside the container. Eventually, the pulled the container back into more-or-less the shape it had been. The opening was covered with a tanned animal hide.

The customer was out of time. He realized had decided to postpone the requirement of a comfortable place to sleep. The lashings across the insides of the container made it difficult to walk around in, let alone lay down comfortably. When he presented the container to his bride, she was pleased, but not very impressed.

Iteration 8: The customer stopped the project. Although he could have stopped the project at iteration 6 and called it successful at meeting the schedule, he really was not happy with what he would have had to live with. He’d spent nearly $20,000 so far, and although it was technically true that the container fulfilled the requirements he’d set early in the project, he realized that he was still a long ways from getting what he’d dreamed of getting two months ago.

The agile developer maintained that the project had been successfully executed, and implored the customer not to give up too quickly. After all, they were still much further ahead than that other customer who’d hired the traditional developer and got a bird cage. They’d spent less, they’d done it in less time, and they were much closer to completing the solution that the customer really wanted.

The container did what it was supposed to. It protected them from predators by ringing bells when wolves or mountain lions brushed against the wires surrounding the container. It kept out the sun, the rain, the wind, and the cold. And it was possible to find a spot to lie down and get a comfortable night’s sleep on some animal furs and hay.

The customer was familiar with the basic problems. The leaves were starting to crack, and had to be replaced frequently. The alarm bells were routinely ignored; the customer and his new wife would take turns keeping watch at night instead. And the container was crisscrossed with lashings that had to be avoided and retightened periodically. Plus, it wobbled and rattled loudly when the wind blew even a little bit.

But it also had revealed many more problems that the customer would have to resolve some other day. For one thing, it was extremely dark inside, especially when the opening was covered. Fortunately, they thought of how flammable the leaves and saplings were before they tried to bring a fire inside to illuminate it. Even if they could have safely made a fire inside, there would be no place for the smoke to go.

After all this, the customer wished he’d known from the beginning that the container project really would cost him the $50,000 and six months that his brother had paid for his container. Even though he wouldn’t have been able to afford the $50,000, he would have felt better going to sleep in the cheap wooden body-burying boxes that would allow him to save his money for what he really wanted.

By Alan on 2010-01-29 07:10:02Z

As I was saying yesterday, most customers can not fully and vividly grasp the complexity of the software required to deliver even a rudimentary solution to their problems. Most customers understand this by now, and are willing to accommodate whatever path we recommend towards solving their problem. They don’t care too much whether we prefer traditional or agile methods. Proponents of agile methods tend to suggest that at least with their method, customers with the same money and the same initial vision wind up with their needs met most closely than with traditional methods, and I agree that seems more likely to be true.

On the other hand, in the real world, most customers need to be able to justify large expenses before they commit the money, not after. And no matter whether we start with agile or traditional methods, they won’t have a clear idea of the actual outcome by the time they need to make the decision whether to spend the money.

On an agile project, we could ask the customer to commit to only small steps at a time. For what will ultimately be a $100,000 piece of software, we ask for commitment to only $10,000 at a time. We’ll deliver exactly what we said we would with the first the $10,000, and the second $10,000. The customer will develop faith in our ability to produce what they want, and we repeat until we’re done, eight iterations later.

But before that customer agreed to pay the first $10,000, we will have had to explain that $10,000 wouldn’t really meet their needs, and that several more iterations would be needed. How many? We don’t really know. We guess, and we point out that it’s more likely to cost less and work better than traditional methods. How much? The customer really needs to know whether they are embarking on a $25,000 project or a $400,000 project.

Steve McConnell is an expert that every software developer should get to know. As he writes in his book, Software Estimation: Demystifying the Black Art (Best Practices (Microsoft)), “Many businesses value predictability more than development time, cost, or flexibility. Be sure you understand what your business values the most.”

Most businesses—for that matter, most people—want to know before they agree to commit any money to a project at all that they will get the value from the project that they need to get in order to make it worth the money. An agile process can’t guarantee that outcome from the beginning with any more certainty than a traditional process. It simply promises to deliver successive versions of something that the customer agrees is more valuable than previous versions.

I’ve compared building software to building actual buildings, but there’s a great big discrepancy in that metaphor. It’s this discrepancy that agile methods attempt to address, and it’s because of this discrepancy that no actual buildings are ever built using agile processes. Creating a building involves the aggregation of a number of tasks, each of which have already been done many times before by a wide variety of people. It’s well known how long it takes to get permits, draw up blueprints, hire subcontractors, prepare the land, lay the foundation, erect the frame, cover the roof, install wiring, ductwork, and plumbing, put in sheetrock, tiles, and ceiling panels, lighting, windows, lay carpet, and paint. Those tasks have to be done for virtually every building ever made. The tools used for one building are the same tools for all buildings. And nearly every part of the building is something that the customer can tangibly grasp.

In contrast, building software is frequently filled with tasks which the developers have, at best, only read about. A new development technology is not like an improvement on a screwdriver; better, but you still use it the same way when turning screws. It’s more like switching from a screwdriver to a rivet gun. And then from a rivet gun to a laser welder. And more often than not, we have to find a way to use the laser welder when an ordinary screwdriver might have worked better.

To complicate it, the relationships between components in software are not as obvious or well understood as the relationships between components in a building. That has to be worked out each and every time with each software project. And finally, most of what makes a software project complete is never directly seen or controlled by the customer.

If we reverse these comparisons—that is, if we had not been making building for thousands of years, but had been building software the way we do for that long, and were now trying to apply software development concepts into building construction—then buyers of houses and buildings would face a pretty crazy process. The customer would come to you and say they wanted a building that they could put their family and their stuff in, and they wanted to know how much it would cost.

The Traditional Process applied to People Containers

Customer: “I’m getting married in three months, and I need you to put together a container where I can protect my wife and myself from the wolves and mountain lions. But first, I’ll have to get money from my dad, so I need to know how much this will cost.”

Traditional Developer: “A container in three months? I need to know a few things first. How much time each day will you and your wife need to spend in the container? And how long does the container have to last?”

Customer: “Well, mostly we need it at night when we’re asleep and can’t be on guard for predators. How long? I don’t know. I guess a couple of years, when I will have money of my own and can get a better one built.”

Traditional Developer: “There a lot of options, and the costs range widely. For example, I could make a couple of small wooden boxes with padded lining for you to sleep in. They’re only about $2,500 a pop. Less if you use cheap wood, like pine, which you might be able to do since you only need it for 2 years. But some people get creeped out with those, because they’re basically the same thing they’ve been using when they put dead people in the ground. Or, we could go to the high end, and put together a large ball made out of cast iron with a watertight door, put in an airhose and a pneumatic pump, a mechanism that can raise and lower it every night into the lake. But that’s kinda pricey; it could cost about $1,500,000. I think that’s what Bill Gates uses. Maybe I’m thinking of Larry Ellison.”

Customer: “I can’t spend that much, but my wife will hate the wooden boxes. What can I get for about $10,000?”

Traditional Developer: “I think we can build you a platform that has been suspended from a large tree, and we’ll put a bunch of iron bars around the sides and top to keep out the mountain lions that might try to jump from the tree. You’ll have a rope ladder you can pull up when you don’t need it. Although, you should know that we’re trying a new thing with the iron bars. We’d been just tying them together with rope, but the rope tended to fray in a few months and the bars would come loose. So we’re planning to use a new technology to keep them together. I think it’s called “wedding.” No, it’s “welding.” They’re using it in those other projects – the ones that connect roads on both sides of a river. Bridges. We haven’t tried it with people containers yet, but I’m sure it will work great here.”

Customer: “And I can get that in three months for $10,000?”

Traditional Developer: “Um… yeah, that sounds do-able. Obviously, I’ll need to get more requirements from you before we really commit to the schedule.”

Customer: “That’s fine, I just needed to know how much I needed to ask from my Dad. I’ll call you back when I have the money.”

 

Out of time for today. I’ll be back tomorrow with the same conversation, only this time with the Agile People Container Developer.