blog dds: 2020-08-27 — Shell scripting for software developers
source link: https://www.spinellis.gr/blog/20200827/index.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
In an open online edX course on Unix tools I was running over the spring with more than a thousand registered learners, I got asked for ideas on how shell scripts can be useful. This is an intriguing question, because the course focuses mainly on performing one-off tasks in the areas of software development, data engineering, and system administration, rather than automation through shell scripts. In response, I posted how shell scripting improves my personal productivity. Here's my take on how shell scripts are employed in diverse software development tasks. I plan to post further installments on system administration and data analytics.
To produce the examples listed here, I searched the entire FreeBSD source code tree for files with a .sh (shell script) suffix. In total I found 1305 scripts (by running the Unix find /usr/src/ -name \*.sh | wc -l pipeline). In summary, I found that shell scripts are used for testing, as program wrappers and adaptors, for stand-alone program implementations, as code generators, for build orchestration and automation, for release management, and for crafting development tools. My experience is that other large software systems use shell scripts in similar ways.
Testing
Shell scripts are used to automate integration testing, by running the program under specific configurations and scenarios and verifying that the program has the expected behavior. They are also be used for regression testing, by running a program and comparing its output against results of previous runs. The FreeBSD /tests suite contains 261 tests of both kinds. Many other programs have test scripts bundled with their source code; see for example the pkill command tests directory. As an example of regression testing consider the OpenSSH regression tests. In total, another 355 test shell scripts reside in individual programs' tests directories and 227 come bundled with the separate NetBSD test suite.
Program wrappers and adaptors
In some cases, either by design or through evolution, a program can be easily written by having a shell script call another one. Consider the following examples.
- The program ranlib, which adds a symbol table to an archive (library) file, used to be a stand-alone program. It is now implemented as a one-line shell script, which calls the ar program to do the actual work.
- The lp POSIX 1003.2-compliant print spooler interface is implemented as a wrapper around the traditional BSD lpr command.
Program implementations
Some programs are easy to construct by gluing together other existing Unix commands in a shell script. Here are some examples.
- The adduser and rmuser user management commands.
- The manctl program, which compresses or decompresses the system's manual pages.
- The service system service management command (the equivalent of the systemd systemctl command).
- The ypinit command, which builds and installs network information service (NIS or yellow pages) databases.
- The freebsd-update, which fetches and installs the system's binary updates.
Code generation
Many scripts are used for creating code from higher level descriptions. Consider the following cases.
- Trivially, the flex mkskel script, reads a text file containing the boilerplate of the generated lexical analyzer, and converts it into a C array that can be compiled into the lexical analyzer generator executable program.
- The curses library MKparametrized script generates C source for an array specifying whether terminal capability (termcap) strings should undergo parameter and padding translation.
- As a more sophisticated case, the kernel's makesyscalls shell script, reads a declarative specification of the supported system calls, and generates the C code that implements the interface to call the lower-level implementation routines.
Build orchestration and automation
Although shell commands can be readily specified in most build systems, such as make, complex build actions are better documented and tested by putting them in a separate shell script file. For example, yacc uses such a script to orchestrate the build process, while libsodium uses separate shell scripts to initiate the make-based build process on different systems, ranging from iOS to Android. More widely, the meta2deps shell script looks for "meta" files and extracts information needed to deduce build and src dependencies, depend-cleanup cleans dependencies after some difficult-to-track source code changes, and tinder builds parts of the source code tree on all supported architectures.
Release management
Several scripts are used for putting together the system's release, including the distribution packages, the release notes, the distributed file manifest, and the architecture-specific ISO images.
Development tools
Shell scripts are also used for constructing ad hoc tools for bespoke development tasks. For instance, the kernxref tool creates a cross reference of a kernel's symbols, while the kerncruft tool finds unused kernel source code files.
If you'd like to learn more about how you can apply Unix command-line tools in software development (and other) tasks you can register for my free open edX course through this link.
Read and post comments, or share through
Recommend
-
40
-
10
DDS是一套通信协议和API标准，它提供了以数据为中心的连接服务。Fast-RTPS是DDS的开源实现，借助它可以方便的开发出高效，可靠的分布式系统。本文是对DDS以及Fast RTPS的介绍文章。
-
24
The release of the Raspberry Pi 400 personal computer reminded me of a wildly popular home computer that was launched in a similar computer-in-a-keyboard f...
-
7
2020-03-23 — Shell scripting for personal productivityIn an edX course on Unix tools I am running these weeks, I got
-
7
2020-08-13 — IBAN length and DESI scoreLooking at the formats of European country international bank account number (IBAN) codes, I noticed that the IBAN length didn't se...
-
13
2020-04-06 — A chat protocol for remote lecturesDue to the current coronavirus lock-down we're holding all our lectures remotely; currently via Microsoft Teams meetings. Students can see my presentation and me talking, and I can hear th...
-
5
DDS的全称是直接数字合成, 这是一种方便地产生自定义任意频率/任意波形信号的方法. 一般说, DDS需要有查找表、相位累加器和DAC三个组成部分. 把需要的正弦波或方波或三角波之类波形数据写进查找表, 相位累加器不断累加固定的步长, 截取高...
-
13
Eclipse Cyclone DDS Eclipse Cyclone DDS is a very performant and robust open-source DDS implementation. Cyclone DDS is developed completely in the open as an Eclipse IoT project (see
-
6
Using Catch2 v3 in CMake with DDS and PMM Nov 13, 2021 When the maintainers of Catch2 announced that the next major version (v3) would be migrating away from its long-standing header-only approach...
-
2
2020-08-27 — Shell scripting for software developers In an open online edX course on Unix tools I was running over the s...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK