In an open online edX course on Unix tools I was running over the spring with more than a thousand registered learners, I got asked for ideas on how shell scripts can be useful. This is an intriguing question, because the course focuses mainly on performing one-off tasks in the areas of software development, data engineering, and system administration, rather than automation through shell scripts. In response, I posted how shell scripting improves my personal productivity. Here's my take on how shell scripts are employed in diverse software development tasks. I plan to post further installments on system administration and data analytics.
To produce the examples listed here, I searched the entire FreeBSD source code tree for files with a
.sh (shell script) suffix. In total I found 1305 scripts (by running the Unix
find /usr/src/ -name \*.sh | wc -l pipeline). In summary, I found that shell scripts are used for testing, as program wrappers and adaptors, for stand-alone program implementations, as code generators, for build orchestration and automation, for release management, and for crafting development tools. My experience is that other large software systems use shell scripts in similar ways.
Shell scripts are used to automate integration testing, by running the program under specific configurations and scenarios and verifying that the program has the expected behavior. They are also be used for regression testing, by running a program and comparing its output against results of previous runs. The FreeBSD /tests suite contains 261 tests of both kinds. Many other programs have test scripts bundled with their source code; see for example the pkill command tests directory. As an example of regression testing consider the OpenSSH regression tests. In total, another 355 test shell scripts reside in individual programs'
tests directories and 227 come bundled with the separate NetBSD test suite.
In some cases, either by design or through evolution, a program can be easily written by having a shell script call another one. Consider the following examples.
Some programs are easy to construct by gluing together other existing Unix commands in a shell script. Here are some examples.
Many scripts are used for creating code from higher level descriptions. Consider the following cases.
Although shell commands can be readily specified in most build systems, such as make, complex build actions are better documented and tested by putting them in a separate shell script file. For example, yacc uses such a script to orchestrate the build process, while libsodium uses separate shell scripts to initiate the make-based build process on different systems, ranging from iOS to Android. More widely, the meta2deps shell script looks for "meta" files and extracts information needed to deduce build and src dependencies, depend-cleanup cleans dependencies after some difficult-to-track source code changes, and tinder builds parts of the source code tree on all supported architectures.
Several scripts are used for putting together the system's release, including the distribution packages, the release notes, the distributed file manifest, and the architecture-specific ISO images.
Shell scripts are also used for constructing ad hoc tools for bespoke development tasks. For instance, the kernxref tool creates a cross reference of a kernel's symbols, while the kerncruft tool finds unused kernel source code files.
If you'd like to learn more about how you can apply Unix command-line tools in software development (and other) tasks you can register for my free open edX course through this link.Read and post comments.
Last modified: Thursday, August 27, 2020 7:26 pm
Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.