Oil Shell: Ambitions for a Unix Shell

blog |oilshell.org

Ambitions for a Unix Shell

2020-01-22

As discussed in theJanuary blog roadmap, I want to concretely describe a reducedOil language, and see if we can get it "done" in 2020.

So, to give context to upcoming posts about the language, let's review the project's goals from different perspectives:

1. What languages and tools does Oil aim to "replace"?
   • bash, awk, make, Python 2, ... ?
2. What problems does a shell help you with?
   • testing, benchmarking, deployment, ...
3. What application areas is shell good for?
   • building distros and packages, automating data science, ...

This post discusses just the first question.

Three Levels

I visualize the goals as a "ladder" to climb:

1. Replace my use ofbash . This the bare minimum . If it doesn't happen, then I'll feel like the project has failed.

   On the other hand, there are "backup plans" mentioned in the roadmap. In the worst case, I hope the source code, tests, and knowledge contained therein will be useful to another project.

2. Replace the world's usage ofbash . Replacing my usage isn't enough, because other projects I want to work on are blocked by gross shell scripts :-) For example, many embedded systems are now Linux systems.

   Another way to think about it: a programming language is a kind of intersubjective truth: its meaning depends on agreement in the minds of other people . It's not enough that a program can be executed by the computer; it has to be understandable and modifiable by people.

   But I realize that, in the best case, this will take decades . That's OK, but I want to get something concrete done in 2020.

1. Replace my use of Python. I've written and maintain dozens of Python 2 programs, and Oil could provide an alternative upgrade path for them, as well as for similar programs with few dependencies.

   I may not have stated this goal explicitly, but I've had it in mind since the beginning. It's natural because Oil currently reuses parts of theCPython 2.7 interpreter.

   But I'm explicitly cutting this out of the project.

   I decided to cut it after making progress on it! I hacked on py_object_main.c , which compiles against a 57K line slice of the CPython interpreter.

   That showed me that it's very feasible, but will take months, not weeks. Implementing the equivalent of that 57K lines from scratch would take more than a year.

The Biggest Cut

So the biggest change is that Oil will be based on strings (and arrays of strings) like shell, rather than having structured data types like Python. It may have someawk-like numbers, but not immediately.

In particular, I no longer think Oil will reuse any of theCPython interpreter. Translating Oil withmycpp made reusing it less appealing. The reasons for this are complicated, but feel free to leave a comment if you're curious about it.

This is drastic, but upcoming posts will show that the language's power isn't reduced for the primary use cases.

Importantly, I didn't start writing a shell because I want to write all my code in shell.

I started writing a shell because I program regularly in Python, C and C++, JavaScript, R, and many DSLs. And I need a way to glue these programs together!

Next

I have several drafts, and I'll try to arrange them in a coherent order:

• Use Cases for a Shell : Addressing questions #2 and #3 from the beginning of this post.
• Recent Progress in Oil . Performance optimization, documentation,Oil 0.7.0, and more.
• Engineering Tasks for Oil 1.0 .
• Features Cut from Oil . Typed data is the biggest one, but there are more!

Appendix: Comments That Add Color

• Python is an amazing achievement (lobste.rs). Looking at the source for Python 0.9 from 1991 gave me new respect for the project's achievements. We now take them for granted, but even decades later, Oil can't retread much of this ground. Shell and Python are overlapping but distinct.

• Oil vs. nushell and other alternative shells (reddit). Oil should be usable as the default system shell on Unix machines. It adds features and fixes design and usability problems in shell, but it's also compatible. This increases the complexity of Oil, but reduces system complexity .

• An analogy to LLVM (lobste.rs). C++ compilers also reside at the foundation of an operating system.Clang and LLVM are amazing projects that are 15+ years old, but most distros still use GCC.

  This analogy shows me that in the best case, Oil will be adopted over decades. The world of software is big, and there's simply a lot of inertia and entanglement in the open source world.

  Relevant links:

  • Building The Mainline Linux x86_64 Kernel With LLVM Clang (September 2019, Phoronix). With the upcoming LLVM Clang 9.0 compiler release there is an amazing achievement more than a decade in the making... The mainline Clang compiler can finally build the mainline Linux x86_64 kernel.
  • EuroLLVM 2019: Switching a Linux distribution's main toolchains to LLVM/Clang (YouTube Video). OpenMandriva is the first general-purpose Linux distribution that has switched its primary toolchain to Clang - this talk will give an overview of what we did, what problems we've faced, and where we're still having problems (usually worked around by using gcc for some packages).

So Oil will certainly be a "layer on top" at first, but I hope that eventually small systems can be built with a kernel, a compiler, and Oil.