Custom Software Story


When I’m not batting my eyelashes or working on classic cars, I manage a tiny custom business software company in which I’m the lead developer. It’s fun. We start new systems out as small investments and then the company pays us to grow the software to the extent needed and to the extent worth it. Typically, it’s very much worth it, and some of our clients have been with us for 15+ years.

They keep spending money on the software because it makes or saves them more than they spend — which explains why I’m able to pay the rent every month.

Here’s an interesting story from tonight: It covers compounding complexity in software.

The above screen shot (or part thereof) shows a subtle bug that took my 2nd-in-command programmer and I several hours to pinpoint tonight, in a piece of custom software that we have grown over 15+ years to fit whatever needs the client’s complex multi-state business had. Were all the program code printed out it’d be several reams of paper, in small font.
The client company is very well-managed. It figures its profit margins very exactly so that it makes money while also being competitive. The final price that a customer is charged is a composite consisting of multiple factors including a carefully calculated admin fee to cover paperwork issues, which might be minor, average or significant — and the company adjusts the fee accordingly.
Compounding complexity comes into play when the real world piles on ever-more complex requirements, and the business software (if it’s flexible and custom-made) keeps pace, but the complexities affect each other, so things get more complex very quickly.
Here’s an example of the compounding complexities that underlie tonight’s bug (and why it was so hard to find):
The simple client’s story is: we track every order, and we track what we charge per order … but then it gets more complex:
  • We ship orders in one of four ways: airfreight, interstate, intrastate or locally.
  • Local orders have four separate sub-types that differ significantly
  • We break down what we charge, including an admin (paperwork) fee
  • There’s a standard amount that we charge for the admin fee
  • We sometimes want to override the admin fee to a lower amount
  • These must be limited to one of a few finite choices
  • Only certain employees are allowed to override the admin fee
  • To other employees, the override amount must display but be non-editable
  • Beyond a certain point in time, the order is locked (frozen) and the admin fee cannot be edited any more.
This bug (in a local order) showed a symptom only when there was a need to override the admin fee. A qualified employee did so, and then a non-qualified employee opened that same screen before the order was locked. This (and only this) caused an unintentional recalculate-and-save event using flawed math that overlooked the override amount.
Finding this bug included logging in an out with various access privileges, poring over vast amounts of code … but we finally found it, yay!
The bug has been fixed, and a new version has been delivered and distributed. All in a day’s work – or half a day, actually. 🙂
Yes, custom software is cool but it’s complex to make and complex to maintain, if it models the business well. As one of my clients remarked: “I never realized how complicated my business is … until I had to explain it to you.”



Rock-Star Irony


Panorama of Sanremo from the harbour, made available by Dr. Gert Blazejewski – Own workAttribution-ShareAlike 3.0 Unported

I was recently listening to a recording in which William Garrett of Room Tone Radio interviewed Susanna Hoffs of the Bangles. In the interview, she mentioned how, even after they had had two hit songs in the Top Ten, they didn’t feel like rock stars. It took a trip to Italy, to play at the San Remo festival, to experience flash bulbs going off, a fuss being made, and the Bangles finally feeling a little famous.

However,as she’s quoted on the KPCC website, that time-frame was “… when we finally had a number one song … with ‘Walk Like an Egyptian.’ We still had no money. We hadn’t seen any money from it … which is probably why there was a disconnect … We’re all living in, like, terrible apartments. And I was living in a garage in Santa Monica. And we were driving beat-up cars, but we had a number one song.”

Their second big hit was in 1987, and their third (and last) major hit was in 1989, so (following the logic) she was probably describing the late-1987 and 1988 time-frame.

This particular musician is intellectually intriguing to me, and I see several parallels between her life and mine. At the time, she and I were both living in LA in the area generally surrounding Santa Monica. We’d both grown up in similar households, both grew up loving and playing 60s music, played the guitar, had an A-student attitude, etc. The list of similarities is longer than belongs in this article, not least because this article actually highlights some diametric opposites that I find most ironic.

As to the image of software developers, it has of late veered toward an under-appreciated, unspectacular, and socially inhibited person. This was to a large extent the consequence of the popular “Dilbert” cartoon strip that celebrated its prime in the late 1990s, and originated in the San Francisco Bay area. Specifically, at the time, the creator, Scott Adams, worked at Pacific Bell, a heavily regulated public utility.

His uninspiring manager and the resulting miserable department culture motivated Scott to draw a daily cartoon on his cubicle whiteboard as a form of silent protest. His co-workers loved his sense of humor and candor. The popularity of the cartoon gradually grew from these humble beginnings.

With yet another ironic story-line twist: in the same time-frame, I also worked in the same area, the same profession and the same company: a subsidiary named Pacific Bell Information Services, being the voice-mail division.

My own manager certainly inspired similar dismay. I recall at least one occasion where the irrationality of his decision-making had so poisoned the department culture that it became too much for me, and I gave in to the need to leave the building, sit down somewhere in private and burst into tears.

Suffice it to say that I can relate to what sentiment might have inspired the bitter humor behind the comic strip. Had Scott Adams been female, perhaps a few good crying sessions might so have diffused his tensions that they might never have boiled over and inspired the cartoon.

The Dilbert image was the exact opposite of what things in the software developer career path were like for me, ten years before, and four hundred miles further south: LA in late 1987 or 1988.


IBM PC – Attribution: Bundesarchiv, B 145 Bild-F077948-0006 / Engelbert Reineke / CC-BY-SA 3.0; picture cropped by Tanya Charbury

The IBM PC had arrived just a few short years before, and was taking the business world by storm. A few professionals knew how to create custom business software that enabled skyrocketing business productivity and profits, far beyond what could be attained using standardized, packaged software solutions purchased off-the-shelf.

This small group of specialists had a unique sub-culture, completely in contrast with mainstream information technology tradition. We were proud, fiercely independent, irreverent, flamboyant beyond offensive, and disrespectful of tradition. We had only one gold standard: merit. If someone was spectacularly productive, that’s all that mattered. We didn’t care about national original, gender, race or age. We didn’t care about degrees, seniority, paperwork or anything else, unless it enabled spectacular productivity. We worked long hours, got up late and ate whatever was handy. We often worked from home and collaborated loosely around the company office, which was located just south of Santa Monica.

I recall one crunch-time long weekend when I was teamed with another developer. We probably worked sixteen-plus hours a day for several days straight, surviving on whatever happened to be in the office refrigerator. Food was simply fuel for the machine. During that particular time frame, near-beer and Girl Scout cookies were the only items stocked, so we survived on that, on adrenaline and on whatever was available in nearby restaurants within ultra-quick walking distance. And yes, we met the deadline. We always did. This was just one example of many such events.

Porsche_911We were paid vast sums of money. Our boss was a math genius, and he owned a huge mansion on top of a hill near the beach. He owned a huge, new, luxurious Mercedes-Benz car plus a bright red Porsche 911 Carrera with black leather interior. He also owned an older Chevrolet Camaro, just for the stake of nostalgia.

The rest of the team was no less unusual. Even by our high standards, our chief geek was a programming genius. He was maybe in his mid 20s but looked sixteen. He owned a large house in Palos Verdes, a new Maserati Biturbo and several other impressive vehicles, including a racing Porsche Turbo. For the latter car, he had ordered as large and powerful an engine as Porsche offered at the time, and then had the engine flown in from the Porsche factory in Germany.


Courtesy of Bull-Doser – Own work.

He understood social nuances precisely and then pointedly chose to ignore them. For example, I overheard a conversation where he called up the middle-aged, status-conscious president of a phone company that was our most important client.

Our programming genius was trying to isolate a problem in the memory structure of the computer that the president was personally using at his desk at the company office. During that time, the pinnacle of geekiness involved running a Microsoft-provided memory analysis program (aptly named “Debug”) whose display is in hexadecimal, and which is utterly incomprehensible to anyone not deeply involved with troubleshooting obscure issues inside the memory model of the IBM PC architecture. Our resident genius conducted the conversation as follows: “Bob, go into ‘Debug’” – no salutation, no small talk, no inquiring whether this might be a convenient time.

This was before the days of Microsoft Windows, and only one program could viably be running on the computer at any given time, given the architecture of MS-DOS, the operating system at the time. By implication, Bob had to save and exit from whatever he was doing, with zero advance notice.


At the computer command line, Bob dutifully typed “Debug” and hit the “enter” key, plus whatever other keystrokes he was told. As prompted, he read aloud the various obscure codes he saw. When our genius had obtained the desired information, he simply said “ok” or something to that effect, and hung up — no expression of gratitude, no goodbye, and no help as to how the client was supposed to cleanly exit this bewildering software, which had no menu nor anything helpful as a guide. Unless the company president knew to press the letter “Q” and then “enter,” he would have been stuck, and would have had to restart his machine.

This is rudeness at an expert level, compared to which a rock band trashing a hotel room is almost polite.

I was even younger than our genius. As to my looks: even after my mid-20s, I was told I looked fifteen. I started humbly and worked harder than anyone else, because I considered myself the least competent and most junior member of the team. In spite of these presumed deficiencies, I ended up being highly productive. Money poured in. I moved to a nice condo in an elegant complex, complete with tropical plants, sauna, lovely pool, security systems, and its own gym.

My junk mail seemed to presume that I was in my mid 60s and financially successful. Offers to attend time-share presentations enticed me with expensive loot such as high-end cameras, if I would only attend. I did occasionally do so, out of sheer curiosity. Even had I been three decades older, I would probably still have been the youngest person at these presentations.


This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license. Courtesy of: Olli1800

I bought a new car, plus also bought a rare silver BMW two-door six-cylinder low-production coupe, plus a black limited-edition MGB convertible.

Even the official socialist on our team drove a flashy Italian convertible sports car. He was a software speed demon; in one example, he began with someone’s phone call data processing program that took 30+ hours to run; by the time he was done speed-optimizing it, the execution time was close to 30 minutes.

The other members of the team were no less spectacular or brilliant. At the time, a popular TV show was “The A-Team.” We personified it.


Gladstone’s Restaurant in Malibu. Photo provided courtesy of Commonist under CC BY-SA 3.0

Christmas parties were wildly ostentatious, such as when the boss treated every employee, plus a guest, to an elegant dinner at Gladstone’s 4 Fish, in Malibu.

I’ve never met any actual royalty, but our boss radiated a regal style, as best I could tell. He had chosen a lovely bride who was also a nice person. She and I soon became friends. She looked like a trophy wife (good) but was treated like one too (not so good).

As for my own love life, I decided to import the pretty brunette whom I had romanced and serenaded when she was sixteen, perhaps eight years prior. From half way around the globe, I sent her red roses and methodically began to woo her. Two months later, she was in California, living with me as my girlfriend, lovelier and nicer than she had been when last we saw each other. We traveled all over the western United States including more than a dozen trips to Las Vegas in one year.

I recall one scene where my girlfriend was driving behind me in the black MGB, with the top down on a lovely sunny LA day. We stopped at a red light. I hopped out of whatever I happened to be driving, ran back to her car and bent down to give her a quick but passionate kiss. I was about to sprint back to my own vehicle so as to not hold up traffic, when there was a huge commotion; the driver of another car had observed this and had very recently moved to California, and he evidently considered this scene to be so quintessentially LA that he was applauding and cheering loudly.

Life was good, though not in a quiet or humble way.

In the 1988 time frame, if attitude, wealth and life-style were any indication, I was living the dream: quasi-rock-star-like, though professionally my fingers were not strumming a guitar but working a computer keyboard.

Coming back to our main theme: as she subsequently made clear in a detailed interview, Susanna Hoffs of the Bangles had the heart and soul of a punk rocker — but publicly (away from the stage) she looked like a well-behaved, nice girl from the west side of LA.

Taking her at her word as to her humble finances, where she lived (a garage) and what she drove at the time (a crappy car), it’s so ironic to me that the real rock star, Susanna Hoffs, didn’t live the rock-star lifestyle whereas in many ways, the tall blonde computer-geek girl did.

From there, the story becomes starkly parallel. In 1989, she and I each experienced, in our respective contexts, great professional success, though serious problems were already looming on the horizon.

Only a couple of years later, these problems had arrived, and had destroyed much of what we’d each worked for. In the 1990-1992 time-frame, she and I each experienced, in our respective contexts, a period of great difficulty, including professional doldrums and an emotionally bewildering time.

Fast-forward to today, almost thirty years later. She is once again a successful rock star and I am once again a successful computer geek girl — and life happily goes on.