Todd Hoff's blog

Todd Hoff's picture

Programmers: Zombies or Super Heroes?

on Jeffries wrote on :

I suppose most boys want to be Superman or Spiderman when they are little. Fact is, we'll have more impact on the world by working with people than by working alone. The sooner the young genius figures that out, the better.

Another way to understand the super hero myth is that each of us is capable of being great, but the normal pressures of tribal life encourage us to hide our capabilities so we won't stand out.

Standing out is a sure way to get hammered down. So the super hero who mythos is a reminder to us all the we are capable of reaching down, overcoming obstacles, and doing great things.

That the super hero image has become iconic speaks to how much we need to balance the message of conformity we get on a daily basis.

Personally I want Einstein thinking the big thoughts. And I want 1000s of teams working all around the world on cancer too. We can have the lone genius, we can have the Manhattan project, we can have the Justice League, we can have the team member that does great things but may not fit in 100%. On a system wide basis the power of progress is a function of the interplay of them all.

The simple rule that teams are most important is too simple.

In a group I enjoy spending the most time on that chaotic border of individual and team creativity. I hate being in a group where people are passive and wait to take direction. That's what happens when people forget they too can be superheros.

I also hate being on team where everyone thinks everyone else is an idiot. That's what happens when people are like Lex Luthor and need to control everything to feel safe.

Fortunately we don't need go to either extreme.

I think Joel has an interesting take on this subject in his High Notes article at .

Todd Hoff's picture

Is your sofware really that hard to support?

Companies are a hive of different development activities. Occassionally you hear about a project and you think hey, we could use that. You contact the developers and they say sure, no problem, but talk to our manager first. It's with the manager where you splat against a stone wall. And the wall has graffitied on it "we can't support that." Or "you can fork the code but it's yours after that."

The manager is just playing it safe. Nobody is paying them to make a general software package everyone can use. That would take more people, more time, and divert them away from the project that they really got funded for and against which they will be judged a success or failure.

But wait, the company is paying for development, is it too much to ask for people to work together so we can leverage syngeristic and other manegerial aphorisms? Apparently it is. Locally maximizing a group's outcome does not lead to global company maximization.

The manager is not being irrational. The manager wants to minimize risk and telling people to buzz off is an easy way to minimize risk.

Does it really take so much effort to internally support software? And if it does, doesn't that indicate there's a problem? The easiest way around a problem is to deny, deny, deny. But what's the big deal? What can go wrong?

Somone Has a Question

Yes, lots of questions can "waste" time locally. There are ways around this however.
* Create a community. Try an email list so everyone using a program can help with technical support. Create a wiki so common topics can be covered.
* Document all answers to questions. Everytime a question is answered try to improve the system so it doesn't need to be asked. If it is asked then people can just say RTFM and here's the link. That takes about 30 seconds.

Someone Has a Bug

Todd Hoff's picture

A Pretty Good Configuration System Even Socrates Might Like

How do you configure your system? I've worked on configuration many many times, but I've never done it right. I have come closer this time, but it's not perfect yet.

Ok, so what?

Here's what you'll get if you open up the prize hidden inside:

1. You get a highly configurable system where every stakeholder in the system gets a chance to configure the system in a rational well defined order using a powerful general mechanism. You can configure the same thing from the command, line, from the environment, from a configuration file, or over a command port at runtime.

2. You have an automatic way to provide an interactive interface to every library in an application over multiple interfaces (telnet, http, command line, etc). This means you can configure anything in an application at runtime and you can access all of an application "debug" interfaces at runtime.

3. You get a system is describable at runtime by applications in their own code. I like this better than building meta systems from configuration files. It's simple enough that programmers might just consider doing it because the benefits are many and the costs are few. The tricks are mostly in how everything fits and works together, not in the amount of programmer work that is needed.

Just what is configuration? Anything in a program that can be set or got.

Configuration is all the really sexy stuff in programming like setting log levels for different parts of an application, setting port numbers, setting host names, setting max queue sizes, max error thresholds, turning on and off features, injecting faults, getting application metrics, running tests, invoking functions, and getting values from different layers of a program so you can tell what is going on.

Todd Hoff's picture

Time Boxed Versus Feature Boxed Releases

This is an interesting discussion you can find at: It's an issue that comes up all the time inside a project and is ultimately the primary driver for project success or failure.

The key point of why time-boxes are more effective is:

To make a project successful you must change how every decision in a project is made. That's what time-boxes do. Time-boxed releases using a short time horizon force every decision made in a project. A time-box is like a black whole sucking in everything that gets close. For every decision branch a shorter implementation strategy selection is forced which results in features getting implemented in less time.

When implementing anything you always have a choice of how to implement it. You can choose a way that gets the job in done with high quality, but takes a shorter period of time. Or you can choose a way, that you may prefer, but will take a longer period of time. This pattern applies fractally to the highest architectural decision to the lowest down and dirty programming implementation detail.

Software is the result of thousands of decisions that trade time for other qualities like elegance, quality, power, generality, familiarity, and excitement.

With a short time-box you are forced to make every decision with the time-box in mind. If you make 100 decisions everyday and everyone one of them results in a shorter implementaiton time, you can see how over a succession of decisions you will implement more stories. Now multiply this same process by everyone in your project. The chances are better your project will meet its goals.

Todd Hoff's picture

What is a Software Car?

This posting ( on Joel On Software discussion group poses an interesting design question.

The first approach:
class Car
public void Wash() { /*code to wash the car*/}

Car car;

The second approach:
class Car : public IWashable
//private members. This class just acts as a business object that is manipulated by "verb" objects. This maps directly to a db entity.
//no methods. just props and constructors and a dispose pattern

class Washer
public void Wash(IWashable ptr) {/*code to wash thingie*/

IWashable myCar = new Car();
Washer w = new Washer();

From a C++ perspective Washer is nice because you don't need to change Car to add washing capabilities. This is solid pragmatic reason, though I think a more philisophical discussion was desired.

Given all the objects that would be needed to wash a car, what makes Car the right place to know how car washing works? Should a car know about how to find the nearest and cheapest car wash, for example? To answer this question you have to somehow be able to decide which objects are natural for Car to know about.

In this case, Washer is really a relation between all objects involved in washing a car. Washer is an algorithm abstraction represented by a class. This could easily be a function in a functional language, though in my mind classes and functions are pretty much the same thing.

Todd Hoff's picture

WTF: The Least Used Resource Error

There's an unexpected and often fatal type of error that happens when you add resources to a horizontally scaled architecture. When the new resource comes online all traffic can be immediately redirected to the new resource, because it has the least load, and it just folds up and dies. You are left wondering WTF (what the f*ck) and it is really hard to track to down and even harder to fix.

The idea behind a scaling out horizontally is that you can add new resources to handle load. This sounds great and it works, but it has some subtle and surprising error conditions that you may want to keep in mind.

Imagine you have a load balancing appliance using the least load metric. Now you add same some new slave MySql servers to handle the load. Your load balancer will redirect traffic to the new slaves, but the slaves are trying to sync, yet they can't sink because they are getting hammered by the new traffic. Deadlock.

Imagine you have storage network that is full to the gills with data. You add a new appliance to give yourself more storage. Now what happens? All the new data goes to the new appliance. That means all users are hitting the same appliance for their data. Your performance slows to a crawl because the appliance can't handle that. You sort of expected parallel IO among all your appliances to handle the load. Now you say WTF?

A related idea is the dark side of partitioning. You partition data to get high performance via parallelization. For example, you hash on the user name to a cluster dedicated to handle those users. Unless your system is very flexible you can't scale anymore by adding resources because you can't repartition the data. All users are handled by their cluster. If you want a different organization you would have to redistribute data across all the clusters. Most systems can't handle that and you end not being able to scale out as easily as you wished.

Todd Hoff's picture

Is programming an abstract expressive art form?

This is much shorter version of a long argument I have been having with myself about the role of software development as an abstract expressive art form. We developers are often seen as just little code robots. But software can be a lot more than that and if it was allowed to more I think we could create a lot more truly great software instead of heaping more dumptrucks full of uninspired trash in the bitbin.

There's an interesting parallel between the western musical tradition and the agile tradition.

If you would have asked Bach the purpose of music he would have said music serves to glorify god. His perspective was a carry over from the middle ages when music was primarily spiritual in purpose.

With the enlightenment a more secular view of music developed. We see, for example, in Mozart the classical emphasis on the individual which lead to clean, tuneful, and entertaining music.

The classical ideal soon evolved into the romantic ideal which valued individual self expression. In Beethoven and Liszt and in 20th century composers we see self expression as the primary driver behind their music.

Bach was undeniable creative, but his music was not intended as a vehicle of self expression. The polyphony of the fugue in many ways resembles a software project. The complex and intricate dance of the many fugue voices is held together by a strong underlying structure. Bach perfected "counterpoint", the "combination of independent melodies to form a harmonically and rhythmically vital union."

In fact, the word "baroque", Bach's musical style, originally means "irregularly shaped pearl", which I take to be in spirit with Wabi Sabi, an idea that has inspired many in the agile world.

Todd Hoff's picture

How do you create a 100th Monkey software development culture?

Someone reading my C++ coding standard recommendation for using doxygen to automatically generate documentation from source code, asked a great question:

I've often considered using doxygen, I always ask myself - is this
really useful? Would I use it if I were new to a project? Would
programmers working on the project use it?

I'll rephrase their question to more conveniently express a point I've thought a lot about:

Why do companies put so little effort into automating their own development
process to make development easier?

It's like the hair stylist whose own hair looks like someone cut it using a late night infomercial vacuum cleaner attachment. Or it's like the interior decorator whose own house looks like a monk's cell.

Software organizations rarely build software to make developing software easier. Why is that?

There are three ways changes are made in an organization:

  • Top Dog - Someone so high up in the org chart you would need a telescope to see him decided because his brother-in-law sells this very expensive Whatever system, you should use it too. Of course it will never work, even after spending all those plump and tasty consulting dollars.
  • Drunken Clown - Someone accidentally did something some way once a while back and that's just the way it works now. I think of this whenever I need to add a page number to a document I'm writing. Instead of a clear "Add Page Number" task I instead have to create a footer and then I need to insert a "special" character into the footer to see page numbers. What's up with that? I've always wanted to ask the original perp who wrote this feature just what were they were thinking? How am I ever supposed to remember page number == special character? But it doesn't matter anymore, that's just how it works and people can't even imagine it working a different way. It just seems the natural and right way now even though it makes no sense at all!
  • Todd Hoff's picture

    How about making unit testing part of the language?

    I would like to make unit tests a first class part of a language rather than bolting them on with annotations or encoding their function in class names. I don't have a strong argument against the minimalist language folks who like to keep the core language small, other than I have feeling it would make for a more powerful and useful language in the long run, even if I can't think of compelling reasons why at the moment :-)

    One feature I would like to add is the ability for one class to say it is the unit test for another class. This would allow:
    * The unit test class to have internal access to the class under test.
    * An explicit test to make sure the often volumous test code is excluded from deployment images.
    * Tools to automatically run tests and report results.
    * A unit test class to be in a different package than the class under test.

    So why did this burning issue come up?

    Because when programmers unit test their C++ code they always wonder how they should do it. Should they make classes friends so they can peek at their privates? Should they make everything public? Should the they be able to call private methods? Should they access member attributes? Should they make two builds, one with #ifdef UNIT_TEST type code and one without out? How do you deal with static methods? There are really an endless number of questions because every situation you encounter in your program needs a way to make it cleanly and easily unit testable.

    There are answers to all these questions, but the key points for me are:
    1. Maintain data hiding.
    2. A test must be able to do anything and everything necessary to verify correctness.
    3. Code under test should have no idea it is being tested.

    The engine that pulls the train here is the desire for data hiding: only expose the methods and attributes needed to fulfill the public contract of the class and the public contract should be as small and single purpose as possible.

    Todd Hoff's picture

    Reanimating Zombie Code - The Art of Source Code Necromancy

    In sites like sourceforge and in the source code repositories of corporations all over the world "live" millions upon millions of lines of zombie code. Zombie code isn't alive anymore because its original animating soul has been lost.

    That's crazy you say, source code doesn't have a soul, most people don't even think animals have a soul, and a lot of people don't even think souls exist, so how can source code have soul? Ok, you go me there.

    What I mean by source code having a soul is: As long as there are people around who understand the source code, change it, improve it, evangelize for it, and faithfully act as intermediaries between the source and users, then the source can be said to be alive.

    Just having a lot of users doesn't make source alive. We use hundreds of applications (and libraries) that nobody has a clue how they work anymore. Heck, nobody may even have the source code anymore as it was lost in the fog of organizational churn. The application just works and that's all that matters...for a while. That blisteringly fast fortran matrix multiplication package (written by what's-his-name the genius layed off three rounds ago) may be in use for years after it died. That fancy rollup report that talks to the 20 different obscure data sources may be used and admired long after it has died. That website that started with much flourish and seed cash may stay on the web virtually for the end of time, but it is still dead.

    That an application works is all that matters until something happens and it is officially pronounced dead or just floats unmourned across the river Styx. What forces are afoot that create zombies out of once living code?
    1. You might move to a different platform and nobody knows enough about the source to port it. It's just dropped.

    Todd Hoff's picture

    Gordon Ramsay On Software

    Gordon Ramsay is a world renowned chef with a surprising amount to say on software development. Well, he says it about cooking and running a restaurant, but it applies to software development too.

    You may have seen Gordon Ramsay on one of his TV shows: Hell's Kitchen or Ramsay's Kitchen Nightmares.

    Hell's Kitchen is a competition between chefs trying to win a dream job: head chef of their own high-end restaurant. On this show Ramsay is judge, jury, and executioner. And he chops off more than a few heads. Kitchen Nightmares is a show where Ramsay is called in by restaurant owners to help turn around their failing restaurants. On this show Ramsay is there to help.

    If you just watch Hell's Kitchen you will likely conclude Ramsay is one of the devil's own helpers ("ram" is the symbol of the devil and "say" means he speaks for the devil: Ramsay). Ramsay screams, yells, cusses, belittles, and throws tantrums even a 7 year old could learn from. Then he does it all over gain just for spite. In Hell's Kitchen there's no evidence at all of why Ramsay is such a respected chef. He is just a nasty man.

    Now if you watch Kitchen Nightmares you will see a slightly different side of Ramsay, he still yells and cusses a lot, but you will also see something else: this guy seriously knows what he is doing. The depth of his knowledge in all phases of the restaurant business is immediately apparent as he methodically works to fix what's broken.

    Ramsay knows how to run a profitable restaurant. That's one of his key skills. Anyone can lose money running a restaurant, the secret is knowing how to make money running a restaurant. Apparently if, you run it right, a restaurant can make a lot of money. It can also lose a lot of money.

    Todd Hoff's picture

    Copernicus is My Favorite Pattern

    In interviews it's common to ask "What's your favorite and least favorite pattern?" My usual answer for favorite pattern is "keep separate things separate." It's a bit meta which allows me to talk about a few design principles that are dear to me. My least favorite pattern is the wretched visitor pattern because it binds together different parts of an application that have no business even knowing about each other. It creates a BLOB.

    After having read "It Started with Copernicus" by Howard Margolis I am going to have a new favorite pattern: the Copernicus Pattern. I will always hate the visitor pattern, so my answer to that question is not likely to change :-)

    The premise of this book is that Copernicus' discovery of the heliocentric model of the solar system started a fire storm of scientific invention, not because of the discovery itself, but because it spread the idea that we puny humans could think and make big discoveries about the universe using nothing but our tiny brains. Copernicus gave people permission to tackle big challenges and the confidence that they could expect to meet them.

    As evidence Margolis lists the major scientific discoveries made before 1600 and the major scientific discoveries made after 1600. Copernicus published his "discovery" that the Earth revolves around the Sun in 1543. In the list of pre-1600 major discoveries there is: nothing. Zip. Nada. After 1600 the pace of scientific discovery blossoms, we discover: the distinction between electricity and magnetism; law of free fall; Galilean inertia; Earth is a magnet; theory of lenses; laws of planetary motion; various discoveries from the telescope, like sunspots; laws of hydrostatic pressure; and synchronicity of the pendulum. All these discoveries were made by Stevin, Gilbert, Kepler, and Galileo all followers of Copernicus.

    Todd Hoff's picture

    Gordon Ramsay's Lessons for Software Take Two

    Gordon Ramsay began a new season of tasty lessons for gourmet software chef's trying to run their own restaurant, err software development group. You may want to read what we learned in the first season of Gordon's tough love school of restaurantary.

    What new chicken nugget size bits of wisdom have we learned so far?

  • Always put the nights earnings in the bank.
  • Don't cook everything on the same grill. Use separate pans or flavors from last weeks fish will be in today's veggie medley.
  • Don't double book. You can't turn more than one table an hour.
  • Pride and arrogance are what stops you from learning from the provocative lessons dancing in front of your face.
  • Lazy ways of cooking taint everything you make.
  • Don't piss off the locals. If the locals aren't happy then you won't make money during the off season.
  • Create vivid experience based learning sessions. To reduce pride in a charge take them to a bullfighting ring and have them fight an angry bull with nostrils flaring for vengeance. Arrogance fades when a giant multi-horned bull tries to kill you. And once the arrogance melts some learning can occur.
  • Don't be too clever. Use local ingredients and cook cleanly and simply. Let people taste the food.
  • You know you are doing well when you aren't stressed; you are communicating; everything seems easy; dishes come back clean; customers leave happy; and you make money.
  • Eat your own dog food. Eat at your own restaurant and experience what it's like from your customer's perspective.
  • Do you like what you experienced?
  • Run your pub, not your kitchen. It's easy for a cook hideout in the kitchen. You own the business, so run the business, let the chef cook, and let your people do their jobs.
  • Don't try to be something you are not. If you are a pub then act like a pub. A pub serves well cooked, simple, traditional food. Don't be a fancy restaurant if that's not your market.
  • Todd Hoff's picture

    Why Some Photographs Look Alive and Others Look Dead

    Artists have long worked at portraying a sense of dynamism in static
    sculptures and two dimensional paintings. To this end Greeks developed
    a technique called symmetria - dynamic counterbalance between the
    relaxed and tensed body parts and between the directions in which the
    parts move.

    Look at the Egyptian sculpture and notice how static it feels. The purpose behind Egyptian art was political and religious. It was used to solidify the role of the Pharaoh as the rightful leader, the only legitimate contact between the sacred and profane worlds. So their art is impressive and enduring, but there's no sense of movement.

    Now look at the Spear-Bearer by Polyclitus. It has a life to it that you don't see in Egyptian art. Look closely and you can start to see how this animated feeling is accomplished. Look at how the left knee is bent and the right knee is straight. The left foot is back and turned out. The right foot is forward and turned the other way. The hips are gently shifted the right and the torso is slightly twisted. The right arm is straight and the left arm is up and bent. The spear angles and forms a diagonal with the right hand. The two knees form an opposite diagonal. The head is turned slightly to the right and is looking off passed us. The muscles are clearly defined leading the viewer to imagine that this man has fought and is ready for action.

    All these little "tricks" don't consciously surface, but the overall impression is one of liveness and rhythm. I've wondered how this technique developed. It's so clever, so subtle, and works so well. Where did it come from?

    Now take a look at the picture of two lionesses playing in the wild. It may look like the lionesses are fighting, but they are really just having a grand old time. When I saw this picture (see Stuart Brown: Why Play is Vital) for the first time symmetria alarm bells immediately went rang a lang a ding dang in my head.

    Compare Lionesses Playing with the Spear Bearer. Amazing similarities. Notice how alive this frozen two-dimensional snapshot of time feels. I've taken a lot of pictures of things in action and they usually feel lifeless and dead, so I know the feeling of liveness just doesn't come from the fact that they are moving. I think it feels so alive for reasons of symmetria. Notice the twist in the bodies, diagonals of the paws, parallel of the tails, diagonal of the head, twist in the heads, the correspondence between the feet, and the muscled arcs of the trunks in flight.

    Taken together all these effects make the picture feel vibrantly and gloriously alive. Maybe an ancient Greek artist witnessed a similar scene thousands of years ago and thought hey, we can do that in sculpture!

    Todd Hoff's picture

    My First Zeplin Trip!

    For a reason without reason I've always dreamed of riding in a zeppelin. No, not the Led variety. I'm not a groupy. I finally got my chance thanks to Airship Adventures. Here are a few pictures from the trip:

    Despite my lack of photography skills it was a picture perfect day. No wind and a blue sunny sky. Zeppelins are creatures of the wind. Too much wind and they can't fly. A couple on our trip were reschedulees from a wind day. We took the trip out of Moffet field. We originally signed up for the Sonoma trip but they had some issues with the landing field so we rescheduled. There's also an Oakland trip the goes out to the bay. I'm thinking I'll eventually hit them all. They also mentioned they eventually plan on a barnstorming type trip which will move along the coast. That would be a blast. Our trip went over 101 to San Mateo and back again. Not the fall colors of wine country but it was still gorgeous.

    If you are expecting something the size of the Hindenburg you'll be disappointed. These beasts are about 1/3rd the size and carry 12 passengers instead of 100. On the inside aren't the luxurious appointments of a gilded age, but functional seating meant for tourists on a short excursion. It's not claustrophobic on the inside (and I'm claustrophobic). All around you are windows so you can see from any angle. A few windows open so you can take pictures without window glare. The whole back end is a window with a window seat so you can look, sit, and contemplate. More than enough room and comfort for our 1 hour tour. And there is a tiny teeny bathroom if the need should arise (complementary picture included).

    Saying zeppelins are creatures of the wind gets directly to heart of what makes zeppelin riding different than a plane or helicopter ride. Zeppelins do not stay still. They are always moving. This makes getting on and off a zeppelin more of an adventure than expected. To get on the zeppelin is moving towards you and you have to scurry up the entry stairs on the fly. Same with getting off. It's quite a lot of fun and adds a bit of spice. The support people have all this worked out so you don't have to worry about anything going wrong. It's no problem.

    Seeing the world from a 1000 feet is a completely different experience. We are either high up in the air or on the ground. Trawling slowly close to the ground but still high above allows you to see patterns you may have never seen before. Some of my favorite pictures are of the sinuous river ways that lign the bay in contrast to the compulsively square shapes humans inflict everywhere. We humans love our boxes, right angles, and straight lines. Where is Antonio Gaudi when you need him? I Iike the contrast here:

    Someone went wild and made a circular shape:
    How crazy is that? It stood out amongst the Roman inspired order of everything around it.

    Another noticeable pattern is you can tell where people have money. It doesn't take a genius to know when you've hit Atherton:

    Though most houses are boring, company campuses have a little creativity:

    And there were a lot of buildings where I wonder what is that?

    There's not much green space for people to play in. We cram every spare inch with a house or something. It's hard to exercise when everything is covered in pavement. It would be nice to plan that out a little better so malls weren't the only place to get away.

    I also really had no idea how much wet lands we had:

    Spectacular to look at from above. Endless gorgeous patterns. And that's what I'll take away from this wonderful trip.