blog dds

Software Tracks

A generous car reviewer might praise a vehicle’s handling by writing that it turns as if it’s running on railroad tracks. Indeed, tracks offer guidance and support. When you run on tracks you can carry more weight, you can run faster, and you can’t get lost. That’s why engineers, from early childhood to old age, get hooked on trains. Can we get our software to run on tracks?

Continue reading "Software Tracks"

Basic Etiquette of Technical Communication

Parents spend years trying to teach their children to be polite, and some of us had to learn at school how to properly address an archbishop. Yet, it seems that advice on courteousness and politeness in technical communication is in short supply; most of us learn these skills through what is euphemistically called “on the job training.” With enough bruises on my back to demonstrate the amount and variety of my experience in this area (though not my skill), here are some of the things I’ve learned.

Continue reading "Basic Etiquette of Technical Communication"

Job Security

My colleague, who works for a major equipment vendor, was discussing how his employer was planning to lay off hundreds of developers over the coming months. “But I’m safe,” he said, “as I’m one of the two people in our group who really understand the code.” It seems that writing code that nobody else can comprehend can be a significant job security booster. Here’s some advice.

Continue reading "Job Security"

Drawing Tools

1 Word = 1 Millipicture

— /usr/games/fortune

It’s no accident that in all engineering branches, our colleagues often communicate using drawings and diagrams. Given many artifacts’ scale and complexity, a drawing is often the best way to describe them. Uniquely, in software development we can easily derive pictures from code, and sometimes even code from pictures.

Continue reading "Drawing Tools"

Start With the Most Difficult Part

There’s not a lot you can change in the process of constructing a building. You must lay the foundation before you erect the upper floors, and you can’t paint without having the walls in place. In software, we’re blessed with more freedom.

Continue reading "Start With the Most Difficult Part"

Brian Kernighan on 30 Years of Software Tools

As part of the IEEE Software 25th anniversary, Brian Kernighan graciously agreed to write a Tools of the Trade column. His article, titled Sometimes the Old Ways are Best, is now freely available online through the Computing Now web site.

Continue reading "Brian Kernighan on 30 Years of Software Tools"

The Way We Program

If the code and the comments disagree, then both are probably wrong.

— Norm Schryer

I can still remember the first time I laid my eyes on production-quality source code. This was in the early 1980s and the code was the BIOS listing of the original IBM PC. The 5940 lines of code spanned 80 neatly typeset pages in a three-ring slip-covered binder. Two things made a lasting impression on me. The first was the elation of being able to read, understand and learn from the code that made a real machine tick. This may have sparked my current practical and research interests in open source software. The second, was the way the code was commented. The BIOS was written in 8086 assembly language, and almost every line had a comment on its right hand side. By poring over the code and its comments I learned 8086 assembly, programming style, and understood the PC’s hardware architecture.

Continue reading "The Way We Program"

Software Builders

The tools and processes we use to transform our system’s source code into an application we can deploy or ship were always important, but nowadays they can mean the difference between success and failure. The reasons are simple: larger code bodies, teams that are bigger, more fluid, and wider distributed, richer interactions with other code, and sophisticated tool chains. All these mean that a slapdash software build process will be an endless drain on productivity and an embarrassing source of bugs, while a high-quality one will give us developers more time and traction to build better software.

Continue reading "Software Builders"

Using and Abusing XML

Words are like leaves; and where they most abound, Much fruit of sense beneath is rarely found.

— Alexander Pope

I was recently gathering GPS coordinates and cell identification data, researching the algorithms hiding behind Google’s “My Location” facility. While working on this task, I witnessed the great interoperability benefits we get from XML. With a simple 140-line script, I converted the data I gathered into a de facto standard, the XML-based GPS-exchange format called GPX. Then, using a GPS-format converter, I converted my data into Google Earth’s XML data format A few mouse clicks later, I had my journeys and associated cell tower switchovers beautifully superimposed on satellite pictures and maps.

Continue reading "Using and Abusing XML"

Rational Metaprogramming

Metaprogramming, using programs to manipulate other programs, is as old as programming. From self-modifying machine code in early computers to expressions involving partially applied functions in modern functional-programming languages, metaprogramming is an essential part of an advanced programmer’s arsenal.

Continue reading "Rational Metaprogramming"

On Paper

A box of crayons and a big sheet of paper provides a more expressive medium for kids than computerized paint programs.

— Clifford Stoll

This column came to life as I was trying to devise an algorithm for analyzing initializers for C arrays and structures. At the time I was using the CScout refactoring browser to look for possible differences between closed and open source code. I had already processed the Linux, FreeBSD, and Windows research kernel source codel and only the OpenSolaris kernel remained. Unlikethe other three code bases, Sun’s code didn’t appear to use any exotic compiler extensions, so CScout uncomplainingly devoured one file after the next. Then, after aspproximately six hours of processing and 80 percent along the way, it reported a syntax error.

Continue reading "On Paper"

Abstraction and Variation

“Master, a friend told me today that I should never use the editor’s copy-paste functions when programming,” said the young apprentice. “I thought the whole point of programming tools was to make our lives easier,” he continued.

The Master stroked his long grey beard and pressed the busy button on his phone. This was going to be one of those long, important discussions.

Continue reading "Abstraction and Variation"

The Tools we Use

It is impossible to sharpen a pencil with a blunt ax. It is equally vain to try to do it with ten blunt axes instead.

— Edsger W. Dijkstra

What’s the state of the art in the tools we use to build software? To answer this question I let over a period of a month a powerful server build from source code about seven thousand open-source packages. The packages I built form a subset of the FreeBSD ports collection, comprising a wide spectrum of application domains: from desktop utilities and biology applications to databases and development tools. The collection is representative of modern software, because, unlike say a random sample of sourceforge.net projects, these are programs that developers have found useful enough to spend effort to port to FreeBSD. The build process involves fetching each application’s source code bundle from the internet, patching it for FreeBSD, and compiling the source code into executable programs or libraries. Over the monthly period I also setup the operating system to write an accounting record for each one of the commands it executed. I then tallied up the CPU times of the 144 million records corresponding to the work in order to get a picture of how our software builds exploit the power of modern GHz processors.

Continue reading "The Tools we Use"

Silver Bullets and Other Mysteries

It seemed like a good idea at the time.

—Ken Thomson, on naming the Unix system call to create a file "creat"

When conference participants interrupt a speaker with applause, you know the speaker has struck a chord. This happened when Alan Davis, past editor in chief of IEEE Software, gave a talk on improving the requirements engineering process at the NASSCOM (Indian National Association of Software and Services Companies) Quality Summit in Bangalore in September 2006. He was explaining why a marketing team will often agree with developers on additional features and a compressed delivery schedule that both sides know to be unrealistic. The truth is that this places the two parties in a Machiavellian win-win situation. When the product's delivery is inevitably delayed, the developers will claim that they said from the beginning that they couldn't meet the schedule but that marketing insisted on it. The marketing people also end up with a convenient scapegoat. If the product launch is a flop, they can say they missed a critical marketing time window owing to the product's delay. Where else are we playing such games?

Continue reading "Silver Bullets and Other Mysteries"

I Spy

Knowledge is power.

—Sir Francis Bacon

The ultimate source of truth regarding a program is its execution. When a program runs everything comes to light: correctness, CPU and memory utilization, even interactions with buggy libraries, operating systems, and hardware. Yet, this source of truth is also fleeting, rushing into oblivion at the tune of billions of instructions per second. Worse, capturing that truth can be a tricky, tortuous, or downright treacherous affair.

Continue reading "I Spy"

Cracking Software Reuse

[Newton] said, "If I have seen further than others, it is because I've stood on the shoulders of giants." These days we stand on each other's feet!

— Richard Hamming

Sometimes we encounter ideas that inspire us for life. For me, this was a Unix command pipeline I came across in the '80s:

Continue reading "Cracking Software Reuse"

Open Source and Professional Advancement

Doing really first-class work, and knowing it, is as good as wine, women (or men) and song put together.

— Richard Hamming

I recently participated in an online discussion regarding the advantages of the various certification programs. Some voiced skepticism regarding how well one can judge a person's knowledge through answers to narrowly framed multiple choice questions. My personal view is that the way a certification's skills are examined is artificial to the point of uselessness. In practice I often find solutions to problems by looking for answers on the web. Knowing where and how to search for an answer is becoming the most crucial problem-solving skill, yet typical certification exams still test rote learning. Other discussants suggested that certification was a way to enter into a job market where employers increasingly asked for experience in a specific technology. My reaction to that argument was that open source software development efforts offer us professionals a new and very valuable way to obtain significant experience in a wide range of areas. In this column I'll describe how we can advance professionally by contributing to open source projects.

Continue reading "Open Source and Professional Advancement"

Choosing a Programming Language

A language that doesn't have everything is actually easier to program in than some that do.

— Dennis M. Ritchie

Computer languages fascinate me. Like a living person, each one has its own history, personality, interests, and quirks. Once you've learned one, you can use it again after years of neglect, and it's like reconnecting with an old friend: you can continue discussions from the point they were broken off years before. For a task I recently faced I adopted a language I hadn't used for 15 years, and felt enlightened.

Continue reading "Choosing a Programming Language"

Debuggers and Logging Frameworks

As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered.

— Maurice Wilkes discovers debugging, 1949

The testing, diagnostic, and repair equipment of many professions is horrendously expensive. Think of logic analyzers, CAT scanners, and dry docks. For us the cost of debuggers and logging frameworks is minimal; some of them are even free. All we need to become productive, is to invest some time and effort to learn how to use these tools in the most efficient and effective way.

Continue reading "Debuggers and Logging Frameworks"

Bug Busters

Although only a few may originate a policy, we are all able to judge it.

— Pericles of Athens

Popular folklore has our profession's use of the word bug originating from a real insect found in an early electromechanical computer. Indeed, on September 9th of 1947 the Harvard Mark II operators did find a moth obstructing a relay's contacts. They removed it and dutifully taped it in the machine's logbook. However, engineers were using the term "bug" many decades before that incident. For example, in a 1878 letter Edison used the term referring to the faults and difficulties he was facing while moving from an invention's intuition to a commercialisable product.

Continue reading "Bug Busters"

Project Asset Portability

It's said that real computer scientists don't program in assembler; they don't write in anything less portable than a number two pencil. Joking aside, at the end of the 1970s, the number of nonstandard languages and APIs left most programs tied to a very specific and narrow combination of software and hardware. Entire organizations were locked in for life to a specific vendor, unable to freely choose the hardware and software where their code and data would reside. Portability and vendor independence appeared to be a faraway, elusive goal.

Continue reading "Project Asset Portability"

Working with Unix Tools

A successful [software] tool is one that was used to do something undreamed of by its author.

— Stephen C. Johnson

Line-oriented textual data streams are the lowest useful common denominator for a lot of data that passes through our hands. Such streams can be used to represent program source code, web server log data, version control history, file lists, symbol tables, archive contents, error messages, profiling data, and so on. For many routine, everyday tasks, we might be tempted to process the data using a Swiss army knife scripting language, like Perl, Python, or Ruby. However, to do that we often need to write a small, self-contained program and save it into a file. By that point we've lost interest in the task, and end-up doing the work manually, if at all. Often, a more effective approach is to combine programs of the Unix toolchest into a short and sweet pipeline that we can run from our shell's command prompt. With the modern shell command-line editing facilities we can build our command bit by bit, until it molds into exactly the form that suits us. Nowadays, the original Unix tools are available on many different systems, like GNU/Linux, Mac OS X, and Microsoft Windows, so there's no reason why you shouldn't add this approach to your arsenal.

Continue reading "Working with Unix Tools"

Version Control Talk Demystified

One indication of the importance an endeavor has in our lives is the vocabulary associated with it. If developers employ a tool or a method, inevitably they will come up with words to describe their corresponding work in an accurate and concise way. I recently heard a colleague describe version control systems (also formally known as configuration management tools) as boring. I hope that this dictionary will dispel this myth by documenting a rich technical and social vocabulary. If you don’t work with a VCS I believe this list will give you plenty of reasons to look at what these systems can do for you and your projects. On the other hand, if you already use a VCS I hope you will find ideas on how to use it more productively and how to improve your configuration management process. And, no matter to which group you belong to, I am sure you’ll find here some new words worth knowing.

Continue reading "Version Control Talk Demystified"

Version Control Systems

A source code control system [is] a giant UNDO key—a project wide time machine.

— A. Hunt and D. Thomas

Sane programmers don't write production code without the help of an editor and an interpreter or a compiler, yet I've seen many software projects limping along without using a version control system. We can explain this contrast if we think in terms of the increased startup costs and the delayed gratification associated with the adoption of a VCS. We humans typically discount the future, and therefore implementing version control in a project appears to be a fight against the human nature. It is true that you can't beat the productivity boost that compilers and editors have provided us, but four decades after punched card programming in assembly language has gone out of fashion we must look elsewhere to reap our next gains in efficiency. And if you or your project is not using a VCS, adopting one may well be the single most important improvement you can undertake.

Continue reading "Version Control Systems"

Tool Writing: A Forgotten Art?

Merely adding features does not make it easier for users to do things—it just makes the manual thicker. The right solution in the right place is always more effective than haphazard hacking.

— Brian W. Kernighan and Rob Pike

In 1994 Chidamber and Kemerer defined a set of six simple metrics for object-oriented programs. Although the number of object-oriented metrics swelled to above 300 in the years that followed, I had a case where I preferred to use the original classic metric set for clarity, consistency, and simplicity. Surprisingly, none of the six open-source tools I found and tried to use fitted the bill. Most tools calculated only a subset of the six metrics, some required tweaking to make them compile, others had very specific dependencies on other projects (for example Eclipse), while others were horrendously inefficient. Although none of the tools I surveyed managed to calculate correctly the six classic Chidamber and Kemerer metrics in a straightforward way, most of them included numerous bells and whistles, such as graphical interfaces, XML output, and bindings to tools like ant and Eclipse.

Continue reading "Tool Writing: A Forgotten Art?"

Java Makes Scripting Languages Irrelevant?

Simplicity does not precede complexity, but follows it.

— Alan J. Perlis

In computing we often solve a complex problem by adding another level of indirection. As an example, on Unix file systems an index node, or inode, data structure allows files to be allocated concurrently and sparsely, and yet still provide an efficient random access capability. When we want to customize large and complex systems or express fluid and rapidly changing requirements a common tool we employ is to add a scripting layer on top of the corresponding system. An early instance of this approach was employed in Dan Murphy's TECO editor developed on the DEC PDP-1 computer in 1962–63: its command language also doubled as an arcane (to put it politely) macro language.

Continue reading "Java Makes Scripting Languages Irrelevant?"

Dear Editor

Machines should work. People should think.

— Richard Hamming

Dear Editor,
I know that you are nowadays often taken for granted, and that many programmers consider you a relic of an older age. Yet, programmers continue to spend an inordinate amount of time with you, and often listen to your advice. As you have no doubt observed I am often mistreated; in this letter I have written my most common grievances hoping you can convince them programmers to behave better toward me in the future.

Continue reading "Dear Editor"

The Tools at Hand

The tools we use have a profound (and devious!) influence on our thinking habits, and, therefore, on our thinking abilities.

— Edsger W. Dijkstra

With a shovel excavator a single operator can effortlessly move 720 tons of earth with a single movement; a VLSI fabrication plant allows a designer to create elaborate sub-micron structures. Without tools the thousands employed in a car factory are nothing, with tools they can assemble a car in 18 effort hours. Sometimes, tools can even subsume the importance of their operators. The violinist Ivry Gitlis, considered one of the most talented musicians of his generation, said of his Stradivarius: "I have a violin that was born in 1713. I don't consider it my violin. Rather, I am its violinist; I am passing though its life." Tools are clearly an important and defining element of any profession and activity: tools help us move boulders and atoms, tools help us reach the Moon and our soul.

Continue reading "The Tools at Hand"