Code Charm

Lapse

Alan@TwilightSoul.com — Sun, 21 Feb 2010 01:44:54 GMT

I’ve been slowly working on a different entry which I’ve kept as a draft for the past several days. But it’s clear to me that I’m procrastinating finishing it and making progress on this blog. I don’t know why yet, and maybe it doesn’t matter.

Listening to an interview on one of my favorite shows, Fresh Air, Terry got Dan Dante to talk about writing. Although his father was a successful author, Dan is only now successful after living the life of a ne'er-do-well. In his interview, he makes a claim that I think was profound, and one from which I might profit. He says, “… it occurred to me … that I couldn't write a novel, but I might be able to write a page a day…. And you know, I'm working on my 10th book right now. So I don't write books. I write pages."

I’d focused too much on trying to write a whole book, I think I got overwhelmed with trying to make everything work towards that end.

So, Ctrl-Alt-Del, I discard my previous distracted meandering, and get back to writing things I think are important to share.

Still researching…

Alan@TwilightSoul.com — Mon, 08 Feb 2010 15:53:48 GMT

Before I go and make a claim like I can invent some new pattern language that best serves the purpose I’m striving to fulfill, I thought it would be only prudent to look around and see whether anyone else has already done this.

So far, I haven’t discovered anyone doing it. Not much else to report, so these posts will be short for a while.

Of Mice and Patterns

Alan@TwilightSoul.com — Sat, 06 Feb 2010 19:32:18 GMT

I’m trying to lay my hands on a copy of the book A Pattern Language: Towns, Buildings, Construction right now.

Meanwhile, I’m going to conjecture about how a pattern language might be better constructed so as to significantly improve the ability for the person who wants to create some software to communicate with the person who wants to build some software about the (and I’ll use this word loosely) architectural vision of the software to be created.

In other words, it’s all well and good for patterns to be used in the design and construction of software. I’m not saying the Gang-of-Four got it wrong when they began identifying patterns for use in the design and building of software. In fact, if I recall, they specifically admonish readers not to stop with their patterns but to build upon them and create an ever larger vocabulary of patterns.

Turns out this can be very difficult to do. But let’s try, anyway, and see what we come up with.

Typically, software is classified by the people who create it. That is, we have embedded systems and operating systems. We have enterprise information systems, applications, games, libraries, frameworks. Certainly we have others. I’m most familiar with enterprise information systems (EIS), but I’ll maintain that the technique I’ll try to apply can also apply to other classes of software.

Within each of those classifications are further classifications. Within EIS, we have database management systems, messaging systems, logging systems, operations management systems, … you get the point.

I’m going to play with the building construction analogy to see if it works.

Operating systems are like public infrastructure. Roads, bridges, levees, fresh water, drainage and sewage, sanitation, etc.

Embedded systems are like private, miniature versions of the public infrastructure, along with the buildings connected to that infrastructure.

Enterprise information systems are like corporate facilities and all the means of connecting them together. Large skyscraper headquarters buildings, warehouses, factories, campuses, distribution centers, leased office spaces, shopping centers, retail stores, and of course fleets of cars and trucks, jets, helicopters.

Applications are like houses, apartment buildings, hotels, garages, mobile homes, recreational vehicles, campers, and tents.

Games are like amusement theme parks, theaters, arcades. cruise ships, restaurants, casinos.

Libraries and frameworks are like laws, rules, standards, traditions, building codes, common practices, policies.

Social network applications are like television, cable, radio, telephones, walkie-talkies, intercoms, and the post office.

So, if I were going to try to describe the application I think my wife was trying to describe, then it would need to feel like I was trying to describe some kind of structure in which people would live and keep their stuff.

Was she describing a hotel or a tent? There’s a very large difference in the scale and scope of the two, even though functionally they both serve similar purposes.

She said she’s spend $20,000 to have it made. You can’t really by a house for $20,000, even in this economy. And yes, I know I’m comparing apples to oranges. But $20,000 indicates not just what the market price is for property; it’s the value that we place on whatever we’re getting. I’ll leave alone for the moment any calculations of rate of return; I’ll have to come back to that later.

For $20,000, you could probably get a used pickup truck with a camper mounted on it, or an exceptionally nice tent. Neither one is quite exactly what you want to spend $20,000 on, though, right? The pickup truck camper will be tiny, cramped, and have been designed and built for someone else. The tent, nice though it may be, is still a tent with thin walls, only so much protection from the rain or predators, or places to hang your pictures. And don’t forget it’s still not connected to drains or sewers, or even electricity (although, for $20,000 you’d think it would come with a generator—and even that will be noisy).

I’ll keep sussing out the metaphor tomorrow.

No post today

Alan@TwilightSoul.com — Fri, 05 Feb 2010 16:16:47 GMT

Partly because I am trying to solve with my right-brain what I mean when talk about patterns the way I think Christopher Alexander meant them to be applied in fields like computer science. (As I stated previously, the way we do it most of the time is definitely not what he meant.)

And …

Partly because I stayed up too late last night on my guilty pleasure: catching up on past episodes of Lost so that I can tune in to current episodes. So sue me ;-).

Patterns revisited

Alan@TwilightSoul.com — Thu, 04 Feb 2010 15:29:02 GMT

While I’m not the foremost expert on patterns, I have a little familiarity with them. Chiefly, I’m aware they were adopted and adapted from a work by Christopher Alexander, an architect, in which he attempted to create a language with which anyone—not merely architects and builders—could efficiently and effectively communicate about built aspects of our world, and what works well, and why. And I’ve read many books which more or less acknowledge Alexander’s influence on their own ideas.

And I’m left with this thought. The scale is way, way the hell off.

Alexander promoted patterns for a metropolis, and patterns for a small part of a building, and everything in between.

But most software design patterns are focused around the code or processes used to create software.

To me, it’s as though software design patterns, if they were directly transferred to their counterparts in the building and architecture world, were describing minutiae. The seminal work, the Gang-of-Four’s book Design Patterns: Elements of Reusable Object-Oriented Software, focuses on object-oriented coding patterns. In buildings, that would be like patterns for attaching boards to each other, for leveling a floor, for digging a square hole in which to put a foundation, for connecting ducts, for making sure screws and bolts turn “righty-tighty, lefty-loosey.”

If I’m about to buy a house, I really don’t care that much about those things. That’s stuff that the carpenters, the masons, the plumbers, and the heating/cooling technicians need to know. Should they know those things? Sure! I’m not saying patterns at those levels aren’t important. But they’re worthless for me, as the user of the house, to know about.

If I’m about to buy a house, I really need to be able to evaluate a house or express my ideas for a house in a much larger, much broader way that really doesn’t care about how it gets done or what construction patterns were used. I need a language where I can share my thoughts with the architect or with the general contractor.

We have no such language right now. When I try to describe potential software solutions to people, it’s almost like I’m trying to describe the materials or processes by which the building gets built, but not really the functional aspects of how it will work for the user. Or, I’m describing the functional aspects of the software, but it’s not clear from that description whether I’m describing something simple, commoditized, and cheap like a kitchen sink (“the function is to wash dirt off hands, dishes, and food”), or something large, intricate, built-to-order, and expensive like an automatic cat and dog groomer (“the function is to wash dirt off dogs and cats”).

We need that language in order to more effectively discuss solutions with our customers.

Field Research

Alan@TwilightSoul.com — Wed, 03 Feb 2010 15:35:56 GMT

I was trying to explain to my wife what I felt was a need for a better way to customers and developers to share ideas and reach a mutual understanding that could realistically lead to a plan.

To make the point, I asked her to think of an idea for new software that she would find valuable. Right away, she said that a program that managed her food inventory would be useful. She even knew some of the features it should have. It would remind her when something is running low. It would update itself when she took something out of the pantry or refrigerator. She could quickly check to see whether she had all the things she needed for a given recipe.

At this stage, she certainly doesn’t care what it looks like, whether it’s on the Web, or an embedded device, or whatever. She doesn’t care whether it’s Windows or Mac or *nix. She probably does care about how she’d have to interact with it, but in her mind, all she knows for sure at this point is that it should be easy and quick and as much of it should be automated as can be.

To get back to my point, I claimed that I could think of a spectrum of solutions varying quite widely in expense. At the low end would be a Microsoft Excel spreadsheet, with a few special formula cells and VBA modules, costing a few hundred dollars to develop. At the high end would be something like MIT Media Labs context-aware kitchen, or any one of a variety of high-tech kitchens, costing hundreds of thousands if not millions to develop.

I said it was important to make sure she got value back out of the software. She said she thought it could be worth $20,000. I’m sure she meant Monopoly money, because there’s no way she’d actually fork over that much for a food inventory program. For the sake of argument, though, I took her at her word.

And then we reached the crux of the matter. It was certainly possible to create something for that much money that at least partly reached the functionality she wanted. But neither of us knew whether the final result would be actually valuable to her, enough that she would feel that her money had been well-spent.

Had I been talking about something that she understood better, she would have been able to quickly evaluate a plan. Instead of new software, I might have suggested buying a kitchen makeover, or a visit to famous kitchens around the world, like Julia Child’s kitchen in the Smithsonian’s museum, or several years’ subscription to a home food delivery service. We’ve never bought any of those, so we have no direct experience to apply in determining whether any of those things would be worth $20,000. But we can pretty easily and quickly make that evaluation anyway. Why? And why can’t we do the same thing with software?

Prototyping language?

Alan@TwilightSoul.com — Tue, 02 Feb 2010 15:12:32 GMT

Like we don’t have enough languages to work with in software development. There’s the language of the business, there’s the jargon we use, there’s the handful of programming languages. Things like domain-specific languages (DSLs) are gaining in popularity. Do we really need another language? Did I really need to use a rhetorical question just there?

Actually, I don’t know. I’m exploring this in my mind right now. Here’s what I’m thinking so far.

I tried to explain the problem I think we face when we’re in the envisioning process of creating software – agreeing to fulfill a fairly specific outcome on a budget without being certain what the final result will really need.

I know that more experienced software developers can use their previous projects as frames of reference from which to cull an estimate for a new project. We know generally how long it takes for us to design the code, write and test the code, assemble and deploy the code. We know what parts to put under the covers that we’ll need when we find out something has to change. We know what parts to put in that the customer never would have thought to ask for but are essential to having it run well. And so we can look at a new project and make off-the-cuff estimates.

But we face all sorts of complications going down this path. Rapidly changing business concerns. Parkinson’s law. Minor catastrophes, like having a key developer leave the team early in a project.

And I’m not convinced that either traditional or agile developers really have cornered the market on a way to get the customer to really understand what he or she is buying, how much it will cost, and when they will have it before he or she makes the commitment to begin the project.

I’m going to spend a little time reading up Steve McConnell’s works and seeing what I can apply here. But Steve focuses largely on software estimation. That’s only half the picture. The other part is getting a clear vision back into the customer’s head about exactly what will be produced so that everyone is happy and calls the outcome successful.

Dream Costing

Alan@TwilightSoul.com — Mon, 01 Feb 2010 15:16:09 GMT

When trying to come up with a solution to a problem, the technique I’ve been taught most of the time, but not necessarily the one I see employed most of the time, is to brainstorm first. Perhaps using mind-mapping to capture the thoughts and suggest new ones. Then, once the ideas have been captured, prioritize them and come up with a plan.

That’s not a bad way to start, but doesn’t really solve the more intractable problem. What if the problem being solved isn’t the right problem?

Consider our two earlier people container developers. I think both of them came up with what would ordinarily be regarded as reasonably smart, clever, and cost-effective solutions for the problems they were trying to solve—protecting a man and woman from predators with a budget of about $10,000 and a schedule of three months. The traditional developer came up with a metal cage, and the agile developer came up with a large upside-down bowl woven from saplings surrounded by bells on wires. The agile developer would have criticized the metal cage as having been over-engineered. The traditional developer would have criticized the wood-and-leaf bowl as being substantially less appropriate for the customer’s long-term needs, or for larger projects.

However, both projects were, in my opinion, over budget and over schedule. In neither case did the customer have the basic need truly met by his deadline and without asking for more money. The traditional developer simply underestimated his job; that happens very frequently with software development. The agile developer technically met his deadline, if we use an extremely generous perspective, but exceeded his budget. (I believe agile developers would generally try to avoid being stuck with a long-term deadline.)

Is there a way to get the customer to express his needs to either developer, or both, in such a way that invented solutions could be quickly estimated fairly accurately? I’m not sure there is. How can anyone know what it will cost to solve a problem which hasn’t really been asked clearly yet?

I should point out about now that I will have to totally abandon the metaphor linking software development with building construction. Software is a continually evolving mechanism, having to deal with extraordinary changes to its environment as well as dramatic changes (or clarification) of the customer’s needs, whereas buildings tend to need much less evolution over time.

What could we compare it to, then, in order to learn from previous experience?

Art? No. While its true that the production of an idea generally ranges from moments to days, and, more rarely weeks, the gestation of that idea has no direct correlation. It could come in a flash, it may never come at all.

Medicine? No. Medicine is chiefly concerned with diagnosing, treating, and researching problems. The treatments would be most closely associated with creating software, but rarely are treatments an enterprise in constructing something completely new from imagination.

Producing movies? No. While the creative elements of making a movie are difficult to predict—getting the script right, finding or making the music, set design, etc.—the structural process of creating a movie remains generally the same from one movie to another, large or small, long or short.

It’s a little like all fields of work, and yet not quite like any of them. There are creative aspects, even when we assemble enterprise information systems. There are basic constructive aspects, even when we’re experimenting with prototypes. There are diagnostic aspects, even when we’re creating greenfield software.

I suspect that the eventual change that will have to happen is that we begin systematically, professionally, and uniformly presenting the creation of software as a completely different endeavor than is traditionally used to solve problems like constructing a bridge, publishing a book, fixing a watch, whatever.

We really are tasked with reading the dreams of our customers, understanding where the opportunity is present, suggesting ways to solve it while having a very good ability to say what the costs of solving it should be.

Makes me wonder if I should have been a professional bowler, instead.

Professional Imagining

Alan@TwilightSoul.com — Sun, 31 Jan 2010 20:25:57 GMT

So what was my point over the last three days? Basically, that the people who will be paying for software still don’t have a very effective way of understanding what they’re asking for and how much it will cost and how long it will take.

If you’re an agile developer, you probably read what I wrote as though I were more critical of agile developers than traditional developers. Conversely, if you’re a traditional developer, you probably read what I wrote as though I were more critical of traditional developers than agile developers. I hope that’s what happened, anyway.

The truth is that I think both ways have a fundamental failure at solving the basic reason many software projects are still over budget or late. We all that that projects, by definition, involve imposing a change in the status quo, and therefore carry risks. But every project carries this risk. And yet many projects are successful, while many others are not as successful, and still others fail outright.

In general, when money is committed to constructing something, there is a plan to recover the cost, plus some profit. Perhaps by selling the thing, or by leasing it. Buildings, cars, watches, beds, traffic lights, whatever. The important part is to make sure that, on average, the costs are kept below the amount of money you’ll demand in exchange for the product. It’s usually not all that difficult to determine what someone would pay for a building, a car, a watch, a bed, or a traffic light. All of those things are commodities. And while creating a valuable label, or brand, for those products helps you charge more (thus allowing you to either profit more or to increase the quality of the product), they rarely allow you to be capricious when setting a price. For example, while The Coca-Cola Company certainly charges far more per can of cola than the pennies it costs to manufacture it, they will still be bound to prices similar to what PepsiCo charge, and to a lesser extent what other beverages manufacturers charge.

Customers know what to expect when they open a can of soda, regardless of the brand. They know what buildings should be able to do, what cars are capable of, the core features of a watch, their requirements for beds, and what constitutes a minimally-functional traffic light.

But software is another matter altogether. You and I, as software developers, know that these beasties can do anything. Anything!! Even the most ambitious types of software developments to date, around artificial intelligence, are not constrained so much by the limits of software as they are by the hardware. Only so much memory can be made available, the CPU requires so much time to process its instructions, the bus can only transfer so much data at a time…. Imagine how much closer we would be to artificial intelligence if we had networks of computers that performed 1 trillion teraFLOPS (amusingly called a yottaFLOPS), and if they had instantaneous access to 1 trillion tebibytes (called a yobibyte)? With that kind of power, it may even be possible to mimic intelligence using a program written in COBOL, although it would undoubtedly take many years to write such a program.

So, the challenge really comes down to this. Every new software creation is a new invention as yet unimagined by the people who want it, of complexity and sophistication which is incomprehensible to most people.

I wager that most people could think of a hundred ways that they would like to improve the tools they use, but only if they were free from the constraints of physics. Imagine a box that held more on the inside than it consumed on the outside. Imagine a battery that lasted for a thousand years. A knife which could cut through a steak but not a even scratch your finger. Now, imagine being asked to estimate the cost and schedule for a project to create something like that. You’d have no idea!

Every software project is a little bit like trying to create what seems like magic. All of software is, in a sense, imaginary. You can’t actually touch software. You can’t weigh it. It can’t wear out with overuse. And it costs nearly the same to make billions of copies of it as it does to make just one copy (if you’re not transmitting or storing those copies inside other things that cost more money per copy). And even when you’re creating software by assembling other software creations, we’re still largely working with tools whose existence is almost entirely within our imaginations, whose capabilities are limited only by our imagination—which is to say, not really limited at all.

So how do we get clear about our imaginations? How do we clearly share our expectations of some imagined solution that itself is created from other solutions which were created from imaginations? And, more importantly, how do we balance our imaginary creations with a very real limit on the ability of developers to fabricate the creation?

I think I’ll spend the next few days on this blog addressing this. I know there are many techniques for getting requirements, but maybe not so many on how to get the right balance of limits on the costs with what is possible in our imaginations and what is possible with the skills of the developers. I don’t know whether I have an answer, but I am very sure it’s a question that must be answered eventually.

Project Cheeky Moose Kick-off Meeting (cont’d)

Alan@TwilightSoul.com — Sat, 30 Jan 2010 20:49:53 GMT

Yesterday, I imagined a dialogue between a customer consulting with an engineer/architect as it might be applied to constructing a building. Only I reversed the order of history, and wondered how it would be different if the science and art of building construction was as new as software engineering. Yesterday, we listened to the conversation between the customer and a traditional developer (of buildings). Today, we’ll listen to the conversation between a different customer and an agile developer.

The Agile Process applied to People Containers

Customer: “I’m getting married in three months, and I need you to put together a container where I can protect my wife and myself from the wolves and mountain lions. But first, I’ll have to get money from my dad, so I need to know how much this will cost.”

Agile Developer: “Well, we know that upfront designs are usually estimated poorly. I know about this one developer that was asked to build a people container, and he made a big upfront design—the old-school way—and when they finally delivered the container, that customer was really upset to find out that he’d bought a big hanging birdcage to put his family in! It was a total disaster. The customer had to get it lowered from the tree, and cover it with wood panels, put in vents, and so on. The original estimate was $10,000, and he wound up paying three times that, and not only that, but the project took twice as long as the original estimate of three months. So, the way I work is this: We’ll pick a period of time, let’s say about two weeks, and each period we’ll commit to completing something which is in line with your needs. Each period, you’ll get a clearer idea of what we’re building, and you can refine it until it’s exactly what you need.”

Customer: “What? Why can’t I just tell you what I want, and then you can go away and make it? You don’t really need me to help make it, right?”

Agile Developer: “Wrong. I totally need you to help make it. In fact, I will predict that if I do it that way you just suggested, you’ll get something that isn’t quite right, and then you’ll have to spend more money than you planned to fix it. Look, I’m not saying you need to do any of the actual construction. But every day, we’ll get together for a very, very short meeting so you can see how we’re doing and meeting our goals. Every two weeks, we’ll actually have a finished product. Now, naturally, the first couple of times we do this, the finished product won’t really be useful, the way the final product will, but it will definitely be a step in the right direction, because you’ll be there to help us understand what it is you really want.”

Customer: “That sounds great and all, except for one thing. I don’t know what I want yet. I just know what I want it to do.”

Agile Developer: “Perfect! That’s all we need from you. You tell us all the things you need it to do. Then we’ll prioritize, and make sure it takes care of the most important things first. When you think it’s finally doing everything you asked, then it’s done.”

Customer: “So it could go on for a really long time?”

Agile Developer: “That’s up to you.”

Customer: “But I said I need it before I get married in three months! How do I know it will be done before then?”

Agile Developer: “Because you are really the force driving what we do in each two-week cycle. You know your priorities better than we do, so you can make sure that each two-week’s goals will get us closer to your final goal, and since you know your schedule better than us, too, you can decide which goals to keep and which ones you can do without.”

Customer: “Yeah, that makes some sense. Only, how much is this going to cost?”

Agile Developer: “Again, that’s something you control. You should know how much value you need to get from this container."

Customer: “You mean, like how much this is worth to me? Well, see the problem is that it’s worth everything to me. It’s for my wife. But like I said, I’ll need to get the money from my dad. He’s an old-fashioned guy, and has spent his whole life in caves or up in trees. He thinks people containers are just really expensive problems waiting to happen, but we’re running out of room in the cave, so he’s agreed to let me do this. But he says I have to keep the cost down. All my friends tell me that agile developers are great at that.”

Agile Developer: “I see. So, then we have to stick with how much money you think you can get from your dad.”

Customer: “Uh… yeah, only I’m not sure how well he’ll take me saying ‘Dad, I need money, only I don’t know exactly what I’m going to get when it’s done.’ “

Agile Developer: “Well, look, you have a fixed deadline, right? And our average rate is about $1,000 per week. You want to finish just a little early, rather than late. So we can plan for ten weeks, five cycles of building. That’s $10,000.”

Customer: “That helps a little. But I still don’t know what I’ll get, other than that I’ll know what it will do, right? My dad is going to want to know I’m getting something more than a couple of those wooden boxes like they use for burying people.”

Agile Developer: “I can see you’re a little apprehensive about this. But rest assured, even by the first or second cycle, you’ll have a much clearer picture of what you’re getting than you could have right now, even if we tried our best to figure it out now.”

Epilogue

Near the beginning of this conversation, I suggested an ending to the story with the customer and the traditional developer. It wasn’t very flattering to the traditional developer, but the truth is that most traditional software projects are delivered over budget and late. See almost any of Steve McConnell’s works if you would like to see abundant research supporting that claim.

But what was the outcome of the agile developer? Bear in mind, both of these stories are totally fabricated from my imagination, but I believe I can predict a likely outcome.

Before beginning the project, the customer was asked to list his priorities for success. He came up with this list:

Big enough for him and his wife
Protection from predators like wolves and mountain lions.
Insulation from the elements (rain, cold, sunburn, wind)
Comfortable enough for sleep
Could be expanded for kids

Iteration 1: The team decided that the “big enough” requirement was largely an architectural requirement, and not something they could actually build and deliver on its own. However, they could certainly start to deliver something that offered protection from predators. They agreed, after discussing the possibilities amongst themselves and the customer, that an obstacle like a very high wall or very deep ditch could not be built in time and would be prohibitively expensive. The suggestion was raised to create an alarm system instead which could wake the customer (if he was sleeping) with enough time to take action. The customer reluctantly agreed, since he couldn’t really afford what he wanted.

The team committed to a “continuously constructed” tactic. Rather than having large chunks of work in an “under construction” state, they planned to have an actual container by the end of the first iteration, and to keep it inhabitable every day. By the end of the first two weeks, the team had built a meter long section of bells hanging from trip wires, and had placed a large flat rock on the ground about 50 meters from the bells. The customer was able to stand on the rock and hear the bells, and judged that he could probably keep his sleep light enough to be awakened by the bells and take action. The customer was both a little perplexed and pleased when the team celebrated their success at creating an inhabitable container. It didn’t look much like a container to him.

The cost for this iteration was $3,000, 50% more than what was originally estimated, but that was because $1,000 went to securing the large area of land (10,000 square meters, or about 2.5 acres) where the container would be built. The team assured the customer that they were making great progress and that, in all likelihood, they would finish under budget and ahead of schedule.

Iteration 2: Part of the team continued working on the alarm wires and bells. They needed to build over 300 meters of wires to create a circle that was at least 50 meters from the center. They determined they were completing 10 meters a day, so they focused on working the part of the circle deemed most likely to encounter predators, and set the goal for this iteration at 100 meters (they were not working weekends!).

Meanwhile, the rest of the team focused on the container. Clearly, the rock would not work to insulate the customer from any elements, but it would work well to anchor the actual container. The team and the customer discussed ways to insulate a person from the elements. The rain was seen as the hardest and most important problem to solve. Several members of the team had experience working on water container projects, designed to catch and keep rainwater for later use. They knew the best way to create a large water container was to dig a large hold in the ground and line it with something waterproof. The best waterproof material created so far was concrete. Unfortunately, it would be too expensive and difficult to apply for this project. But the team felt that the same principle could be applied here. The plan was to create a large upside-down bowl, and cover it with a waterproof liner.

So the team, sticking with their “continuously constructed” plan, began building the frame of a large, upside-down bowl. The goal for this iteration was to have the general frame of the bowl complete. At the end of the two weeks, they had created a circle three meters wide of stripped saplings on the ground lashed together with rope, and three arcs of saplings tied onto the circle and joining at the top. The customer could now begin to see something that seemed a little like a people container, and was pleased.

The cost for this iteration was $2,500, or 25% more than what was originally estimated. $500 went to purchasing stripped saplings, because the team did not have time to cut and strip them themselves. So far, the customer was delighted that he chose an agile developer.

Iteration 3: The wires and bells team had built 100 meters of the alarm system. With a little over 200 meters to go, they predicted they would speed up their progress with experience and could complete it by the end of the next iteration, but only committed to completing 100 meters this iteration.

The container team set a goal of completing the upright arcs of saplings around the circle. They were still trying to decide which waterproof material to use. They’d considered tanned animal hides, large tree leaves, and woven corn husks. They settled on tree leaves as the most practical given the time and expense allotted. However, the largest tree leaves available were only a little over 1/4 meter large. The team realized that meant the arcs around the circle would have to be spaced about 1/4 meter apart. With nearly 10 meters around the edge of the circle, that came to 40 half arcs, or 20 full arcs.

Three full arcs had already been built, so seventeen more needed building. It was taking the team a full day to build each arc, and they still needed to start fastening the leaves to the arcs. So they cleverly decided to start attach each new arc within 1/4 meter of an existing arc. That would allow another member of the team to start attaching leaves to a pair of adjacent arcs sooner than if they had waited until all the arcs were in place. They committed to building nine arcs for this iteration, leaving the remaining eight for the next iteration.

Unfortunately, there were not enough people on the team to work simultaneously on the wires and bells, the arcs, and the leaves. And they still hadn’t tried to tackle the requirement for comfortable sleep. They couldn’t push out the schedule, so they decided to hire more people to fasten on the leaves. The new team was brought on board and shown what to do.

The cost for this iteration was $3,000. The unplanned $1,000 was used to bring in the additional workers to keep the project on schedule. Nevertheless, the customer was getting very excited. He could see the outline of his new container being formed. He could hear the alarm bells periodically tested. He started telling people what a great move it was to use the agile developer. He had to spend some time convincing his dad that it would be worth more than the $10,000 he’d originally agreed to, but because he could actually show some good results every two weeks, his dad agreed to another $5,000. After all, every day, he’d been able to walk inside this new container and imagine living in it in safety and comfort with his bride.

Iteration 4: The wires and bells team completed 105 meters during the last iteration, and felt they could continue to accelerate their pace for this iteration. So they set a goal of 110 meters, which would bring them to completion.

The container team had completed twelve arcs so far, and had planned to complete the remaining eight this iteration. However, as they began, the customer noticed a new requirement that he hadn’t noticed before. He needed a way to easily get into or out of the container. The agile developer took this problem in stride. It was exactly to address this sort of thing that agile development had been created. Everyone had known that the customer would probably get clearer about his requirements as he was able to more clearly see the solution every day.

So the team brainstormed, and came up with a few solutions. One was just to cut some arcs a little short of where they would come down to the circle, but nobody was sure they could properly fasten a beam that would hold them in place since they wouldn’t be fastened to the circle at the bottom. Another was to bend the arcs at one section so that they left an opening wide enough to walk through. A third suggestion was to dig a tunnel under the circle through which the customer could crawl into and out of the container; it was discarded as soon as someone pointed out that it would fill with water at the first rain. The best and cheapest option seemed to be leaving a gap around the outside by bending the arcs away from the opening.

A few big problems came up during this iteration. First, during one of their daily meetings, the team realized that the leaf-fastening team was going to have a problem. The arcs were parabolic, and fastened to a circle that was three meters wide. While this had created a serendipitous outcome of yielding a stable structure, it meant that the top of the structure was about four meters off the ground. The leaf-fasteners had been working from the base of the structure, and working up as high as they could get—about two meters. They would need to find a way to reach the higher parts of the arc.

Second, the wire and bells team were having another issue. Birds kept landing on the wires or the posts holding up the wires and ringing the bells. They’d been continuous testing the bells to make sure they kept working, but hadn’t foreseen the need to test against false alarms.

Third, just as the arc-fastening team was fastening the fourth of the last eight arcs, some things happened unexpectedly and suddenly. At the top of the structure, the arcs had been merely laid across each other at the center where they all crossed. They were too far off the ground to lash them together. Now, with sixteen saplings laying on top of each other, a few slipped sideways a little. With all the tension present in the bent saplings, and with the pulling apart of some of the arcs to make space for the opening, what had been a flat circle on the ground suddenly became a wobbly oval. Furthermore, the gap for the opening was folded in, and what had been planned to be a meter-wide opening was now about one-half meter wide.

The cost for this iteration was $2,000. Although no new unplanned costs came into play, the customer was a little alarmed to see that he wasn’t as close as he thought he was to the outcome he’d planned. It wasn’t making him feel much better that the agile developer seemed not at all upset by the setbacks. The agile developer reminded the customer that all projects encounter setbacks; what made agile better than traditional methods is that the customer and the agile team could more easily adapt and respond to these setbacks appropriately.

Iteration 5: With only a month to go before his wedding, the customer was getting nervous. The alarms bells were going off all the time, even when they shouldn’t, while the wires and bells team tried to figure out how to solve the false alarm problem. The top half of the container was still uncovered, and the only plausible way to cover it would be to roll the whole container onto its side, which presented an additional risk of bending the container even more than it was. And the container would probably never sit completely flat on the ground.

Iteration 6: The wires and bells team continually tried to make the case that they’d met their requirement, and that the new requirement to avoid false alarms would need new planning, more money, and probably a newer technology.

The leaf-fastening team came up with a clever way to fasten the leaves from the inside of the container by constructing a crude scaffold. They managed to complete their task, but everyone on the team noticed that the leaves were beginning to dry and would become brittle before long. While they would still keep out the sun, rain, and wind while brittle, they would clearly not last more than a few months. They said nothing, deciding to let the next team deal with the customer when he noticed that he would have a requirement for durability in the waterproofing.

The arc-building team, which was still the core team, worked long and hard (including weekends) to fasten lashings across the arcs inside the container. Eventually, the pulled the container back into more-or-less the shape it had been. The opening was covered with a tanned animal hide.

The customer was out of time. He realized had decided to postpone the requirement of a comfortable place to sleep. The lashings across the insides of the container made it difficult to walk around in, let alone lay down comfortably. When he presented the container to his bride, she was pleased, but not very impressed.

Iteration 8: The customer stopped the project. Although he could have stopped the project at iteration 6 and called it successful at meeting the schedule, he really was not happy with what he would have had to live with. He’d spent nearly $20,000 so far, and although it was technically true that the container fulfilled the requirements he’d set early in the project, he realized that he was still a long ways from getting what he’d dreamed of getting two months ago.

The agile developer maintained that the project had been successfully executed, and implored the customer not to give up too quickly. After all, they were still much further ahead than that other customer who’d hired the traditional developer and got a bird cage. They’d spent less, they’d done it in less time, and they were much closer to completing the solution that the customer really wanted.

The container did what it was supposed to. It protected them from predators by ringing bells when wolves or mountain lions brushed against the wires surrounding the container. It kept out the sun, the rain, the wind, and the cold. And it was possible to find a spot to lie down and get a comfortable night’s sleep on some animal furs and hay.

The customer was familiar with the basic problems. The leaves were starting to crack, and had to be replaced frequently. The alarm bells were routinely ignored; the customer and his new wife would take turns keeping watch at night instead. And the container was crisscrossed with lashings that had to be avoided and retightened periodically. Plus, it wobbled and rattled loudly when the wind blew even a little bit.

But it also had revealed many more problems that the customer would have to resolve some other day. For one thing, it was extremely dark inside, especially when the opening was covered. Fortunately, they thought of how flammable the leaves and saplings were before they tried to bring a fire inside to illuminate it. Even if they could have safely made a fire inside, there would be no place for the smoke to go.

After all this, the customer wished he’d known from the beginning that the container project really would cost him the $50,000 and six months that his brother had paid for his container. Even though he wouldn’t have been able to afford the $50,000, he would have felt better going to sleep in the cheap wooden body-burying boxes that would allow him to save his money for what he really wanted.

Project Cheeky Moose Kick-off Meeting (cont’d)

Alan@TwilightSoul.com — Fri, 29 Jan 2010 15:10:02 GMT

As I was saying yesterday, most customers can not fully and vividly grasp the complexity of the software required to deliver even a rudimentary solution to their problems. Most customers understand this by now, and are willing to accommodate whatever path we recommend towards solving their problem. They don’t care too much whether we prefer traditional or agile methods. Proponents of agile methods tend to suggest that at least with their method, customers with the same money and the same initial vision wind up with their needs met most closely than with traditional methods, and I agree that seems more likely to be true.

On the other hand, in the real world, most customers need to be able to justify large expenses before they commit the money, not after. And no matter whether we start with agile or traditional methods, they won’t have a clear idea of the actual outcome by the time they need to make the decision whether to spend the money.

On an agile project, we could ask the customer to commit to only small steps at a time. For what will ultimately be a $100,000 piece of software, we ask for commitment to only $10,000 at a time. We’ll deliver exactly what we said we would with the first the $10,000, and the second $10,000. The customer will develop faith in our ability to produce what they want, and we repeat until we’re done, eight iterations later.

But before that customer agreed to pay the first $10,000, we will have had to explain that $10,000 wouldn’t really meet their needs, and that several more iterations would be needed. How many? We don’t really know. We guess, and we point out that it’s more likely to cost less and work better than traditional methods. How much? The customer really needs to know whether they are embarking on a $25,000 project or a $400,000 project.

Steve McConnell is an expert that every software developer should get to know. As he writes in his book, Software Estimation: Demystifying the Black Art (Best Practices (Microsoft)), “Many businesses value predictability more than development time, cost, or flexibility. Be sure you understand what your business values the most.”

Most businesses—for that matter, most people—want to know before they agree to commit any money to a project at all that they will get the value from the project that they need to get in order to make it worth the money. An agile process can’t guarantee that outcome from the beginning with any more certainty than a traditional process. It simply promises to deliver successive versions of something that the customer agrees is more valuable than previous versions.

I’ve compared building software to building actual buildings, but there’s a great big discrepancy in that metaphor. It’s this discrepancy that agile methods attempt to address, and it’s because of this discrepancy that no actual buildings are ever built using agile processes. Creating a building involves the aggregation of a number of tasks, each of which have already been done many times before by a wide variety of people. It’s well known how long it takes to get permits, draw up blueprints, hire subcontractors, prepare the land, lay the foundation, erect the frame, cover the roof, install wiring, ductwork, and plumbing, put in sheetrock, tiles, and ceiling panels, lighting, windows, lay carpet, and paint. Those tasks have to be done for virtually every building ever made. The tools used for one building are the same tools for all buildings. And nearly every part of the building is something that the customer can tangibly grasp.

In contrast, building software is frequently filled with tasks which the developers have, at best, only read about. A new development technology is not like an improvement on a screwdriver; better, but you still use it the same way when turning screws. It’s more like switching from a screwdriver to a rivet gun. And then from a rivet gun to a laser welder. And more often than not, we have to find a way to use the laser welder when an ordinary screwdriver might have worked better.

To complicate it, the relationships between components in software are not as obvious or well understood as the relationships between components in a building. That has to be worked out each and every time with each software project. And finally, most of what makes a software project complete is never directly seen or controlled by the customer.

If we reverse these comparisons—that is, if we had not been making building for thousands of years, but had been building software the way we do for that long, and were now trying to apply software development concepts into building construction—then buyers of houses and buildings would face a pretty crazy process. The customer would come to you and say they wanted a building that they could put their family and their stuff in, and they wanted to know how much it would cost.

The Traditional Process applied to People Containers

Traditional Developer: “A container in three months? I need to know a few things first. How much time each day will you and your wife need to spend in the container? And how long does the container have to last?”

Customer: “Well, mostly we need it at night when we’re asleep and can’t be on guard for predators. How long? I don’t know. I guess a couple of years, when I will have money of my own and can get a better one built.”

Traditional Developer: “There a lot of options, and the costs range widely. For example, I could make a couple of small wooden boxes with padded lining for you to sleep in. They’re only about $2,500 a pop. Less if you use cheap wood, like pine, which you might be able to do since you only need it for 2 years. But some people get creeped out with those, because they’re basically the same thing they’ve been using when they put dead people in the ground. Or, we could go to the high end, and put together a large ball made out of cast iron with a watertight door, put in an airhose and a pneumatic pump, a mechanism that can raise and lower it every night into the lake. But that’s kinda pricey; it could cost about $1,500,000. I think that’s what Bill Gates uses. Maybe I’m thinking of Larry Ellison.”

Customer: “I can’t spend that much, but my wife will hate the wooden boxes. What can I get for about $10,000?”

Traditional Developer: “I think we can build you a platform that has been suspended from a large tree, and we’ll put a bunch of iron bars around the sides and top to keep out the mountain lions that might try to jump from the tree. You’ll have a rope ladder you can pull up when you don’t need it. Although, you should know that we’re trying a new thing with the iron bars. We’d been just tying them together with rope, but the rope tended to fray in a few months and the bars would come loose. So we’re planning to use a new technology to keep them together. I think it’s called “wedding.” No, it’s “welding.” They’re using it in those other projects – the ones that connect roads on both sides of a river. Bridges. We haven’t tried it with people containers yet, but I’m sure it will work great here.”

Customer: “And I can get that in three months for $10,000?”

Traditional Developer: “Um… yeah, that sounds do-able. Obviously, I’ll need to get more requirements from you before we really commit to the schedule.”

Customer: “That’s fine, I just needed to know how much I needed to ask from my Dad. I’ll call you back when I have the money.”

Out of time for today. I’ll be back tomorrow with the same conversation, only this time with the Agile People Container Developer.

Meeting Invitation: Project Cheeky Moose Kick-off

Alan@TwilightSoul.com — Thu, 28 Jan 2010 15:16:39 GMT

I’m not a big fan of meetings usually, but there are some times when they’re absolutely essential. Unfortunately, I’ll be the only one attending this project kick-off meeting, so I get to have my cake and eat it too.

The agenda for this meeting is fairly simple. The goal I have today is to set the next goal.

If I were following a traditional large project model (waterfall, RUP, MSF), I’d probably set the next goal so that it was as far away as I could reasonably make it. I would have whoever is going to pay for this project describe their aspirations, and how quickly they wanted it, and how much resources they would devote to get it. I’d want them to be fairly clear about those aspirations, because ultimately, I’ll be attempting to guess (only I’ll call it “estimating”) how much work it will take to deliver a solution in the time they want it, and whether the available resources are enough. Usually, this means telling the customer that they will either have to scale back their ambition, or give it more time, or spend more on resources, none of which makes anyone happy. I’ll need a lot of experience at doing similar work if I want to be any good at all at guessing the time frame and work required.

If were following something more like an agile project model, I’d probably meet with whoever is going to pay for this project, have them describe their aspirations, the time frame by which they’d like to see what they might call a finished product, and find out how many resources would be available. I would not spend much time getting clear on the goals right now; rather, I will trust that the customer will be able to decide when they’ve been met. I’ll probably give some feedback about my guess for the success we might have at meeting the expectations in time with the available resources. But my main goal will be to make sure the customer understands that we’ll take incremental steps towards that goal, without looking much beyond the immediate step we’re on. I want to make sure the customer understands that our ability to predict project outcomes is extremely limited early in the project, but will get progressively better.

Probably the biggest problem with the first approach is that my experience has taught me that over the lifetime of the project the requirements will grow. the resources will turn out to be not as valuable as I’d hoped, but the schedule won’t change to accommodate those problems. I’ll share my observations with the customer; some will be okay with that and some won’t. To counter it, what I usually try to do is estimate on the very high side, just to see how much tolerance the customer has for finding out the project will cost more or take longer or deliver less than they wanted. Interestingly, those very high estimates usually turn out to be close to what happens. (Perhaps I should be estimating even higher?)

This is the sort of quandary that agile methodology seeks to avoid. By avoiding committing to a fully worked out plan early on, I’m pushing the management of expectations back onto the customer.

I think software development processes has similarities to building construction. I’m the general contractor, responsible for delivering a building which is only in someone’s imagination right now. Let’s say we’re building a house.

If I followed traditional process models, then I’d sit with the customer, maybe bring an architect (or the architect could bring me to sit with the customer), and we’d work out how much the customer was willing to spend, what kinds of features and qualities they wanted in the house, and how quickly they wanted it. In houses, I would probably tell the customer when I could deliver it. In software, the customer usually has a quarterly or annual goal to make and will know how quickly they must have it. While the first meeting would not accomplish much, it would only take a few meetings to gel an idea of the scope of the house and the budget. Ultimately, I’m going to give an estimate to the customer for the cost of the house and the time it will take to build it.

If I followed an agile process model, then one of the first things I’d do with the customer is to make it clear that I won’t be giving much of an estimate. Rather, I’ll lay out the plan to just start building a house, a chunk of work at a time. When the customer decides that enough time and money has been spent on building the house, and that it’s complete enough for their satisfaction, then we’ll end the project. I can assuage the feelings of uncertainty the customer will undoubtedly have by referring to previous successful projects and having previous customers describe how I solidly met my intermediate goals.

Software development has a lot that is different from building construction, however, and this is why agile methods work better than my brief descriptions might suggest.

Most people can vividly imagine a house that hasn’t been built yet. We’ve all lived in at least one, we’ve all been inside dozens if not hundreds. And, as houses go, the overall patterns in the architecture are classifiable, we have countless reference works from which to pull ideas, and any new house is probably not going to have some extraordinarily novel approach to solving a fundamental need that houses fulfill. More importantly, we’re all experienced enough with houses that we can, without expert guidance, guess that some outlandish feature of a house could be prohibitively expensive or not work as well as we’d like. For example, suppose that I wanted to put my house on a turntable so that it could face east in the morning and west at night. Without consulting anyone, I can guess that it will be extremely expensive, and may wind up causing more problems than it solves.

In contrast, few people can vividly imagine software that hasn’t been built yet. While some aspects of software are shared by most applications, most people have only experienced the user interface part and have real grasp of what happens under the hood, or what it might take to put all that stuff under the hood in the first place. So how can we help customers understand all this?

I don’t know yet. Unfortunately, I’m out of time. So, with the goal of this meeting unmet, I’ll have to adjourn and return to this tomorrow.

Cheeky Moose

Alan@TwilightSoul.com — Wed, 27 Jan 2010 15:20:22 GMT

Now, I will describe The Work.

Project Code Names

For a long time (longer than I care to admit), I’ve been toying with a project idea that, for lack of a better project name, I called “Sumatra.” (I like code names more than final product names because, although I like to imagine that I’m a great marketer, I realize that I may not actually be one.) Sumatra works as a code name for me because I have so few other projects. But now that I’ll be discussing it more openly, I’ve decided to rename it. Sumatra is now Project Cheeky Moose, thanks so some suggestions from a random project name generator.

I created a CodePlex location for this project. It’s at http://cheekymoose.codeplex.com/. I’m not very familiar with CodePlex; I hope it works well for what I’m going to try to do. Did I pick it because it’s run by Microsoft? Pretty much, yes. I don’t have, at this early stage, many requirements for what I’ll need for sharing code, documentation, and so on. I know I needed it to support SVN, and that the licensing model wasn’t onerous. CodePlex will work fine for now.

About Cheeky Moose

The very short description of Cheeky Moose (hereinafter “CM”), aside from being a vehicle to showcase technologies and techniques, is to connect real people across various networks and technologies.

Well, an iPhone does that, and I’m clearly not making an iPhone. So a little more detail is called for.

Cheeky Moose works a little like a big brother. Yep, I know that has all kinds of negative overtones, thank you very much George Orwell. That’s why this project makes for a terrible business idea. The privacy implications and potential litigation opportunities just stink. Nevertheless, creating this kind of application gives me all sorts of ways to hook into lots of different technologies, and, if it does what I think it will do, actually would be very useful in the way a real-life big brother could—or should—be.

So what does it actually do? Cheeky Moose does its best to track your actual physical self, and what you are spending your time on while connected to the Internet, and it can report to you or whomever you select what you are doing and what you have done in ways you totally control. It tries to manage your online identity and presence automatically, and reduce the number of separate identities that you need to manage independently. It helps you stay connected to other real-world people as much or as little as you desire by using information they have knowingly shared with you about their online identities. Finally, at some point, it will help you quickly find people you haven’t met who would be willing to help you solve a problem or take some action for a cause.

Is all this useful? I don’t know. As I mentioned in a previous post, this is very likely a solution in search of a problem, although I can assure you I conceived this idea out of frustration from

trying to manage my time by keeping track of where it has gone,
trying to automatically let other people know whether it’s okay to bug me right now with an instant message, text message, or phone call,
trying to let people know that I usually prefer email over phone calls for business communication, and phone calls over email for casual social communication,
trying to schedule things with people who live in different time zones or countries, respecting their preferred hours or recognized holidays,
trying to wrangle a couple of hundred different accounts on the Internet,
trying to remember who I have met on the Internet (wondering if I know them anywhere else), and
trying to find someone who could and was willing to help me with small problems or projects.

I suppose each issue could have its own solution, and indeed I will build Cheeky Moose so that each need is addressed separately, but the more I think about solutions for any one of these needs, the more I realize that the single solution that leaps to mind is a very deep and personal form of a personal assistant. But, I want this “virtual assistant” to be relatable to me more like a protector and better able to advise me than some hired flunky. I’m trusting it with a lot of very personal information! It had better live up to the trust I put in it.

That said, there will certainly be a limit on how much I actually do share using Cheeky Moose. I know that every system is potentially vulnerable, and I have no wish to unnecessarily increase the risk of having my identity stolen, or ruining my friendships, and so on.

Tomorrow: project kick off

Envisioning (cont’d)

Alan@TwilightSoul.com — Tue, 26 Jan 2010 15:13:17 GMT

The Work

Normally, for a software project, I would begin by trying to establish an understanding of what value the final solution represents. What need is it going to fulfill? Why is it needed at all? How will we know that it was worth the time it took to build it?

Indeed, I will do this exercise for this work, but I won’t be as skeptical of its estimated value as I might be normally. I will be guilty of creating a solution whether it really solves a problem or not. That is usually not a great way to design a solution.

However, this is a learning opportunity and a teaching opportunity. I hope that the solution I’ll design will have enough similarity to truly valuable solutions that the technologies and techniques I use will be transferable to real world solutions. Partly, this is so that I will have developed some skills that I can use in later projects myself, and partly it is to show you one way to use these technologies and techniques in the full context of the development of a complete solution.

So much for the disclaimer. Now, what technologies and techniques are these? The list is pretty ambitious. I may have to skip some eventually, but let’s cross that bridge when we get to it. Mostly, as I said before, these will be Microsoft-oriented, but I will venture into non-Microsoft waters occasionally.

Products

In no particular order:

Microsoft .NET Framework
Windows Communication Foundation (WCF)
Windows Workflow Foundation (WF)
Windows Presentation Foundation (WPF)
Windows Identity Foundation (WIF)
Amazon Web Services (AWS)
Google App Engine
SharePoint
BizTalk
SQL Server
Windows Azure
SQL Azure
C# programming language
F# programming language
Visual Basic programming language
JScript programming language
Java programming language
.NET Regular Expressions
XHTML
CSS
PowerShell
AJAX
ASP.NET
ASP.NET MVC Framework
Windows Forms
Silverlight
LINQ
ADO.NET Entity Framework
WCF Data Services
Code name “M” programming language
SQL Server Modeling Services (formerly code name “Oslo”)
Windows Server AppFabric (formerly code name “Dublin” and “Velocity”)
MSBuild
Managed Extensibility Framework
Windows Live Messenger
Microsoft Office SDK
Windows Mobile
NMock
Rhino Mocks

Techniques

Again, no particular order:

Test-driven design
Well-known patterns
Iterative development
Service-oriented architecture
Continuous Integration

Features and Qualities

Secure
Separation of concerns
Loose coupling
Accessible
Extensible
Interoperable
Testable
Maintainable
Manageable
Reliable
Responsive
Stable

I think that will be enough for now (only mild sarcasm intended).

Tomorrow, I’ll cover exactly what the work will do.

Envisioning

Alan@TwilightSoul.com — Mon, 25 Jan 2010 14:57:03 GMT

The Worker

Once upon a time, I fancied that I would get a certification. Specifically, the Microsoft Certified Solution Developer (MCSD) certification. (Those are now outdated.) In the process of studying, I learned about the Microsoft Solutions Framework (MSF). Oddly, I have not yet come across a developer or team utilizing it. However, I like to refer to it because I really like to have a shared understanding of common terms and ideas. Although we all know about patterns, I avoid trying to call things patterns because I think the term “pattern” is overused.

MSF describes five phases of the software development lifecycle: Envisioning, Planning, Building, Stabilizing, and Deploying. We are at the Envisioning phase right now. The goal now is to try to reach an agreement on what will be accomplished by this effort. Normally, you would do this with a team of stakeholders. As I’m the only stakeholder, I’ll just to apply decisions somewhat arbitrarily.

What I’m going to be looking for is a description of what function the solution will perform, how it is valuable (and how will I know that it is), what general strategy it will employ. In short, how will I know whether the final solution is what I want. Basically, I just need to describe in fairly specific terms what outcome I want, without describing too much how I think it should be made. I can talk about the overall cost constraints (or, in my case, the time constraints), or the political realities (that I’ll generally favor Microsoft technologies), or the goals I have in non-functional requirements.

Because I’m journaling all this and because I intend to use this project to learn new technologies as well as teach, my goals specifically do include using some technologies without knowing exactly how they are essential to the final solution. I do not recommend including any specific technologies in the Envisioning phase. Those are decisions best made in the Planning phase. For my part on this project, I won’t consider the project a failure if I wind up deciding not to use some technology I thought I would earlier.

Tomorrow, I’ll cover my vision of The Work, and lay out some of the technologies I would like to include. I’ll also, for completeness’s sake, lay out some of the methodologies and development patterns I would like to use while building The Work.

Technorati Tags: MSF,Envisioning

The Realm of the Journal

Alan@TwilightSoul.com — Mon, 25 Jan 2010 14:30:11 GMT

One of the things I’d really like to reveal with this journal is the full perspective of the software development process, all the way from the conception of the idea, to the final delivery of a product that I could feel good about handing over to someone else. (Like many artists, I have some difficulty ever calling software totally finished.)

One perspective is to consider the technologies I use. Why do I select one technology over another? How do I use it? Why do I use it and not a different method requiring a different technology altogether?

I’ll let you know right away that I will be using Microsoft technologies primarily. This is because I’ve spent many years getting to know and use these technologies, not because they always make the best choice. As much as I’m able, I’ll contrast the Microsoft technology with other technologies, and I’ll attempt to design the system in such a way that you could, if you were building a similar system, substitute a different technology.

My experience suggests that technologies evolve more rapidly than our practical design and usage. I may choose to use technology X in a part of the system now, because it seems to be the right choice, but as I build my system, the technology will reach a new version which has been extended to include new functionality, or it will be in a direction slightly different than I anticipated. I don’t want to completely redesign my system, and I don’t want to let it permanently depend on old version of the technology. So, I try to isolate the application of the technology from the dependency my system has on it.

Another perspective is to consider the development techniques I use. How do I plan my time? How do I set goals? How would I delegate work to a team or to subcontractors? How much documentation do I do? What do I document? What do I do with my documents? How long do I spend making documents versus writing code? How much testing do I do? Do I do test-driven design, or model-driven design, or something else? How much consideration do I give to the intangible, non-functional requirements like extensibility, scalability, and reusability? What patterns will I apply and reject over different aspects of creating this system?

For the moment, those two perspectives are enough. They’re pretty broad! I could say that from one perspective, I’m highlighting the work in progress, and from the other perspective I’m illuminating the worker. For brevity’s sake, I’ll begin to label these perspectives as “The Work” and “The Worker.” But keep in mind that these are merely two different perspectives of the same thing.

If metaphors help (and they do for me), I’ll give you one. Imagine that I’m setting out to build a house.When I label something as “The Work,” you could think of it like I’m describing the plans and construction of the house itself. How many stories will it have? Is it brick or wood? How large are the windows? Hardwood floors or carpets? What’s the floor plan? Who makes the sinks and faucets, and how do I put them in? How do I make it so that I don’t have to rip open ceilings and walls to add a ceiling fan with two switches in the wall where there used to be only a single light and switch?

And when I label something as “The Worker,” you could think of it like I’m describing the skills and techniques needed by the architect of the house, and the general contractor who is hired to direct the team that will build the house, and the individual concrete pourers, carpenters, electricians, plumbers, tile layers, and so on.

Summary:

Sections denoted with “The Work” are focused on the tools and technologies I apply to creating the software itself.
Sections denoted with “The Worker” are focused on the skills, techniques and methods I use in creating the software.

And so it began

Alan@TwilightSoul.com — Mon, 25 Jan 2010 13:48:08 GMT

I’m not sure where this will go, but I know how it will begin.

Inspired both by the movie Julie & Julia, and, oddly enough, Conan O’Brien’s admonition to not be cynical, I have committed finally to a project that has long been on my mind.

Every day, for two hours, I will work on a software project that has been a fancy of mine for years. I will journal all aspects of it here, with an eye towards revealing all of the things I learn, as well as sharing and teaching the things I know. I intend to write this as though it were to be the content of a book I could publish, and ultimately, I do hope that I can create a book from this.

Mostly, I want to put down what I have always found lacking in other books. A picture of good software development as seen from all perspectives. Managing the lifecycle. Choosing the right technologies. Applying appropriate patterns. And this is the most important part: Seeing mistakes and how they get detected and corrected.

I’ve read many books on software development. Not as many as some people I know, but quite a few. And rarely do I see the author correcting him/herself. I only get to see the finished product. I admit, I don’t know how useful that really would have been, but it’s always left me with the feeling that everybody else that I read knows everything and that I know next to nothing. Naturally, this has humbled me. And yet I know that I recognize good software, and I can even create good software with enough time (I’ll never win any speed awards).

So for all the future developers that, at the time they read this, believe that maybe I have something useful to impart, I want you to know that I will be showing you all the work, in full, mistakes and all. As much as I can, anyway. I don’t mean the typographical errors; hopefully, I have few of those. No, I mean the mistakes of thinking that I knew how to apply a pattern. Or that I knew how to apply a new technology. Or that the architecture I create is as effective as I hope it will be.

It will probably take me more time to create my project than it would ordinarily, as I plan to document my thoughts as I go, which will take considerably more time. I hope that the documentation will make it easier for me, as I go, to recall why I made certain decisions, to keep to the plan, and hopefully, even though I have no readers at the moment, allow for others to help me spot those mistakes as early in the project as possible.

So, what is this fantabulous project I’m going to work on that I’ve cherished for so many years? I have had, I’ll admit, the conceit that I could turn this project into a profitable venture. However, the more I consider it, the more I believe it would be doomed to failure. Not because the idea is bad or not as useful as I believe. It’s because in order for it to succeed, people will have to be very slightly different than they are, or else the people owning the business have to be much braver and much more prepared to deal with potential negative fallout than I am.

The short description of the project: The software will connect and keep connected real people to real people across all of the methods we have of being connected to people via technology.

Does it sound like a social network, like FaceBook or LinkedIn? It’s not. Sounds like Microsoft Communicator? It’s not. Sounds like Twitter? It’s not. What it does have in common with all those systems is that it can keep you connected with all of the real-world people you know and connect with over those systems. What it doesn’t have in common is that it also helps you locate people that are not in your pre-existing network but are willing to help you.

I thought, at first, that would be a great idea. At least for me it would. But there are all kinds of privacy implications of a tool like this. For one thing, it will need to be able to represent you in all of those systems you connect to. In other words, it will need your passwords. That’s a non-starter for a lot of people. A lot of people.

Furthermore, one of the main things it would do is share what you are up to with the people you trust to know what you are doing. (For those of you who tweet “I’m going to Puerto Vallarta for 2 wks,” don’t be surprised when your house gets robbed while you’re away.) While that’s great for the people you trust, there are also countless marketers who would pay dearly to know what you do, too, but not because they plan to rob you (although I guess some would argue that they actually do!). They want to know because they actually might have something to offer you that you would, if you knew about it at the right time, actually want to buy. It’s just that they never know when that will be, and they never know exactly who that is at any given moment.

And, if there ever are any plans to turn my project in to an actual business, it would have to be paid for, somehow. And whoever pays for it is going to be suspected of wanting to harvest all that incredibly valuable data for purposes which may collide with our desires. I suppose it could be funded entirely out of donations, like Wikipedia. And if that turns out to be the case, then it’s good that I fully document this system, the ultimate open-source project, and hope that it stays generally secure from attempts to hack it.

So, there you have it. Two hours a day, for as long as it takes. This could take months, or years. Depends a little on whether I’m as good as I’d like to believe I am. Mistakes will be revealed. Foolish misunderstandings of my part will be evident. And, hopefully, someone, somewhere, will get something useful from my journal. For my sake, I hope that I come across as someone who has something worthwhile to share skills that are valuable.

And so it begins, come what may.

Buying a computer

Alan@TwilightSoul.com — Sun, 25 Oct 2009 16:29:48 GMT

My cousin wrote me and a few of her friends, asking for advice on how to buy a computer.

… I'm putting some feelers out there to see what you all might recommend for what to include in on an AWESOME system. I would LOVE to get a Mac, but then we'd have to buy our programs all over again, like Photoshop and Lightroom as well as MS Office (does Mac have MS Office compatible?)

Boys, I need your help! Please shoot me off some advice about what makes a system AWESOME. We know we want a big screen and will need to include a printer in the purchase…. I want it to be awesome for editing my photography and a large memory is a plus too. Also please let me know anything about PC vs Mac. I have wanted to convert to Mac for a long while, but not sure about what it takes to work on a Mac in a PC majority world. Any advice or tidbits at all would be awesome on either PC or Mac.

This is what I wrote back:

Macs are good computers, but you tend to get less awesomeness for the money. Why? Because Apple keeps very tight control over the available software and peripherals to make sure everything works more or less the way it is supposed to. Macs are best for people who don't think of themselves as very technical (like being interested in the difference between CPU speed and bus speed). There is a way to run some or most Windows programs on Mac; I believe that includes Office. I can't really give you much advice over which Mac to pick to get if you go that way.

I favor Dell computers for PCs. After that, I think Lenovo makes good portable computers. I'm not as much a fan of Hewlett Packard; HP tends to put in their own additions on top of Microsoft Windows that, in my opinion, add only bells and whistles while making other features less stable. Dell is happy to give you a basic, working PC, and let Windows do whatever it wants.

When I select a computer, I try to get a balance between these factors:

Manufacture reliability / quality

Lower price

Higher CPU speed (with multiple cores and/or raw speed)

How much RAM (memory) is included

Higher bus speed

Available RAM expandability

It's important to get these items as good as you can because you will probably keep the computer for a while, and these are things that you can't really change over the life of the computer. You can add more RAM (see #4), but only up to the limit of #6, and usually running out of RAM becomes the main reason why older computers appear to run slower over time.

Once I have used those to narrow my selection down to a short list, I consider how much I want to spend on these things, in about this order:

Larger and/or faster hard disk

High quality, high resolution, and larger-sized monitor

Video display adapter

Printer/scanner

Optical disc player/burner

Speakers

Each of those things are very easily replaced or expanded after you have your computer. Very large hard disks are not that expensive, but you should be careful not to get a giant hard disk that is very slow. The hard disk can also make your computer appear sluggish over time, so it's important to get a moderately fast hard disk. However, like I said, you can replace hard disks, although replacing your main hard disk (the C: drive) can be a lot of work.

I tend to stay low-tech on things like keyboards and mice. My keyboard has a cord and is about 12 years old. My mouse doesn't, but it's not a blue-laser wonder mouse, either.

Don't be afraid to ask the tech guys at the stores lots of questions, and if they aren't explaining things to where you REALLY understand them, don't feel bad about asking to talk to someone else. it's YOUR decision, not theirs.

Technorati Tags: computers. mac vs pc,selecting a computer,choosing a computer,buying a computer

Error: "Control cannot be created because Visual Studio cannot find the control's type in the control's assembly"

Alan@TwilightSoul.com — Fri, 09 Oct 2009 11:06:41 GMT

I was working on a .NET 2.0 Web Application, and had attempted to add a ScriptManager control to a Master page from the Toolbox, from the ASP.NET AJAX Extensions (1.6.0something). Visual Studio 2005 refused to do it, giving me this error message in a dialog box:

Control cannot be created because Visual Studio cannot find the control's type in the control's assembly

I Binged that error message, and found something that was close: Rick Strahl had an error in his web.config file. However, I did not have an error in my web.config file.

What I did have, however, was a /<compilation> section that used the configSource attribute to use a separate .config file for that section. I find it helpful to have a separate .config file for almost every section in system.web. Unfortunately, Visual Studio doesn't care for me doing that. Once I brought all the child elements of the compilation element back into the web.config file, I could easily add the ScriptManager to my Master page. I'm not sure why -- I had already manually added the assembly to the section, so VS didn't need to add it again, nor why the error message is as cryptic as it is considering the root cause.

Late to the party: Data Access in Visual Studio 2008

Alan@TwilightSoul.com — Thu, 08 Oct 2009 05:25:37 GMT

For whatever reasons, I’m about two years behind the curve here. But, well, these are the cards I was dealt, and these are the cards I will play.

I’m beginning a series of entries about all the new data access stuff that I’m finding in Visual Studio 2008 / .NET Framework 3.5. I’ll discuss them in the context of a Visual Studio 2005 / .NET 2.0 application I wrote which analyzes historical data about public companies so that I can make decisions about whether to buy stock in those companies. At the heart of this program is a library which parses an HTML page from MSN where I have an account to retrieve historical data. Because of copyrights and terms of use agreements, I won’t be distributing or documenting that part of the program.

I have a simple data model in this Visual Studio 2005 Windows application. I’ve already converted the SQL Server 2005 database to SQL Server 2008; that went smoothly.

Simply, there are just two tables. One has a record for each company I want to look at. The other has an entry for each month where I pull down historical values. The column names are from the book that was the inspiration for doing this:

Rule #1: The Simple Strategy for Successful Investing in Only 15 Minutes a Week!

I loaded up Visual C# 2008, and then opened the old Visual Studio 2005 solution, which consisted of seven projects:

The conversion seemed to go pretty smoothly. There were some minor warnings about relative paths being different for the backup projects when they got moved to a different folder. That’s fine.

Next, I tried to compile the application. This didn’t go so well. What used to be one of my tried-and-true staples of quick-and-dirty personal projects seems to have lost a little weight. The strongly-typed DataTables in my strongly-typed DataSet object were missing a method they used to have: Select(string expression).

public EquityDataSet.CompanyHistoryRow[] GetHistoryRows()
{
return DataSet.CompanyHistory.Select("Ticker='" + Ticker + "'", "Year DESC, Month DESC") as EquityDataSet.CompanyHistoryRow[];
}

Now, I know I should have been spending the last couple of years kicking around LINQ and the ADO.NET Entity Framework and so on, but I let my day job get a little ahead of me. No more excuses, it’s time to figure out how to take my existing code and make it all new and shiny again.

One of the first things I noticed is that the direct ancestor of my strongly typed CompanyHistoryDataTable class is no longer just a System.Data.DataTable class, but is now an instance from the new generic System.Data.TypedTableBase.

It puzzles me a little, because I can see that TypedTableBase<> is a kind of DataTable. So what happened to my Select method? Why did it disappear?

It pays to read more than just the first error in the Error List pane.

The type 'System.Data.TypedTableBase`1' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Data.DataSetExtensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.

These other errors indicate that I need to add references to an assembly that I’ve never used before: System.Data.DataSetExtensions. The conversion wizard automatically added it to the Data project, but failed to add it to any of the projects that depend on the Data project. I’m sure Bing would have turned up something, but it’s easy enough to fix.

A little cleanup of SQL Connection Strings, and voila, I have a running application, but it’s still in the ADO.NET 2.0 frame of mind.

Next steps: Looking at LINQ (not posted yet)