Oil: Success With the Interactive Shell

blog |oilshell.org

Success With the Interactive Shell

2019-02-05

I'm now usingOSH as my interactive shell ! Three recent milestones enabled this:

1. Successfully running thousands of lines ofbash completion scripts found in the wild. I wrotea blog post in October that explains why this task is both necessary and hard!
2. Running shell programs that modify the interactive prompt , like git-prompt.sh and Python's virtualenv .
3. Implementing basic history expansion. For example, !$ expands to the last word in the last command.

This was a long slog through many features and bugs. There's still more to do, but I'm confident that this "dogfooding" process will make the shell usable by others. It immediately revealed several issues, many of which I've already addressed.

This is the biggest milestone since last January, when I wrote that OSH could run large shell scripts that build Linux distros .

The Next Few Posts

This work will take more than one post to explain. Here's an outline:

(1) The Interactive Shell Needs a Principled Parser

Poking atbash's interactive features revealed several bugs which have the same underlying cause. I'll demonstrate them, and then:

• Explain how OSH avoids these bugs with its principled parser.
• Show that OSH has 8 times less source code thanbash. The reuse of the parser is a major reason this is possible. Remember that OSH can run thousands of lines of diverse distro scripts and interactive programs, so the comparison is now pretty fair.

(2) Interactive Shell Screencasts

I'll show OSH running the programs above, and a sketch for a better interactive UI.

(3) Release of OSH 0.6.pre14

As part of the dogfooding process, I used OSH to release itself . Recall that each/release/$VERSION/ page — with its many tests, metrics, and benchmarks — involves thousands of lines of shell. In particular, devtools/release.sh invokes many other shell scripts.

Version 0.6.pre13 is linked below, but the dogfooding process has already produced more changes, which will go in 0.6.pre14 .

Caveat: the biggest open problem is that OSH is still too slow . Although this doesn't matter for many batch programs, it does matter for interactive scripts.

The code inbash-completion consists largely of messy and expensive string processing, and it's already too slow underbash. But it can be more than an order of magnitude slower under OSH.

I have some ideas to fix this, but they aren't guaranteed to succeed. OSH is still experimental in that respect.

I'll also discuss what I want to happen for the 0.6.0 release, which has been brewing for a long time.

More

I want to get back to coding after those three posts, but I also want to write:

(4) Project Roadmap . This big milestone, as well as feedback from theBayLISA talk, clarified what I should work on next. I'm really motivated to work on the interpreter performance problem.

(5) Blog Roadmap . There are even more things to publish, like:

• The next post in theDev Log series.
• In January, I re-discovered a pervasive and old shell security issue , which is yet another reason that shell needs a principled parser.

OSH 0.6.pre13

In the meantime, please tryOSH 0.6.pre13:

• https://www.oilshell.org/download/oil-0.6.pre13.tar.xz ( gzip version )

I'm looking for experienced shell users to try OSH on real shell scripts — both interactive and batch — and report bugs !

This is important because changes to theOSH language will be more painful later. OSH can now run many real programs, so its architecture is becoming stable.

To build and run it, follow the instructions inINSTALL.txt.

Feel free to ask me questions in the comments . Or join oilshell.zulipchat.com and ask questions!