Archive for September, 2014

How to patch bash

2014-09-26

3rd in what is now becoming a series on ShellShock. 1st one: 10 tips for turning bash scripts into portable POSIX scripts; 2nd one: why big is bad.

ShellShock is a remote code exploit in /bin/bash (when used in conjunction with other system components). It relies on the way bash exports functions to its environment. If you run the command «env - bash -c 'foo () { hello ; } ; export -f foo ; env'» you can see how this works ordinarily:


PWD=/home/drj/bash-4.2/debian/patches
SHLVL=1
foo=() { hello
}
_=/usr/bin/env

The function foo when exported to the environment turns into an environment variable called foo that starts with «() {». When a new bash process starts it scans its environment to see if there are any functions to import and it is that sequence of characters, «() {», that triggers the import. It imports a function by executing its definition which causes the function to be defined.

The ShellShock bug is that there is a problem in the way it parses the function out of the environment which causes it to execute any code after the function definition to be also executed. That’s bad.

So the problem is with parsing the function definition out of the environment.

Fast forward 22 years after this code was written. That’s present day. S Chazelas discovers that this behaviour can lead to a serious exploit. A patch is issued.

Is this patch a 77 line monster that adds new functionality to the parser?

Why, yes. It is.

Clearly function parsing in bash is already a delicate area, that’s why there’s a bug in it. That should be a warning. The fix is not to go poking about inside the parser.

The fix, as suggested by I Jackson, is to “Disable exporting shell functions because they are mad“. It’s a 4 line patch (well, morally 4 lines) that just entirely removes the notion of importing functions from the environment:

--- a/variables.c
+++ b/variables.c
@@ -347,6 +347,7 @@ initialize_shell_variables (env, privmode)
 
       temp_var = (SHELL_VAR *)NULL;
 
+#if 0 /* Disable exporting shell functions because they are mad. */
       /* If exported function, define it now.  Don't import functions from
         the environment in privileged mode. */
       if (privmode == 0 && read_but_dont_execute == 0 && STREQN ("() {", string, 4))
@@ -380,6 +381,9 @@ initialize_shell_variables (env, privmode)
              report_error (_("error importing function definition for `%s'"), name);
            }
        }
+#else
+      if (0) ; /* needed for syntax */
+#endif
 #if defined (ARRAY_VARS)
 #  if ARRAY_EXPORT
       /* Array variables may not yet be exported. */

In a situation where you want to safely patch a critical piece of infrastructure you do the simplest thing that will work. Later on you can relax with a martini and recraft that parser. That’s why Jackson’s patch is the right one.

Shortly after the embargo was lifted on the ShellShock vulnerability and the world got to know about it and see the patch, someone discovered a bug in the patch and so we have another CVN issued and another patch that pokes about with the parser.

That simply wouldn’t have happened if we’d cut off the head of the monster and burned it with fire. /bin/bash is too big. It’s not possibly to reliably patch it. (and just in case you thinking making patches is easy I had to update this article to point to Jackson’s corrected patch)

Do you really think this is the last patch?

Advertisements

Why big is bad

2014-09-26

If you know me, you know that I don’t like using /bin/bash for scripting. It’s not that hard to write scripts that are portable, and my earlier “10 tips” article might help.

Why don’t I like /bin/bash? There are many reasons, but it’s mostly about size.


drj$ ls -lL $(which sh bash)
-rwxr-xr-x 1 root root 959120 Sep 22 21:39 /bin/bash
-rwxr-xr-x 1 root root 109768 Mar 29 2012 /bin/sh

/bin/bash is nearly 10 times the size of /bin/sh (which in this case, is dash). It’s bigger because it’s loaded with features that you probably don’t need. An interactive editor (two in fact). That’s great for interactive use, but it’s just a burden for non-interactive scripts. Arrays. Arrays are really super useful and fundamental to many algorithms. In a real programming language. If you need arrays, it’s time for your script to grow up and become a program, in Python, Lua, Go, or somesuch.

Ditto job control.
Ditto Extended Regular Expression matching.
Ditto mapfile.
Ditto a random number generator.
Ditto a TCP/IP stack.

You might think that these things can’t harm you if you don’t use them. That’s not true. We have a little bit of harm just by being bigger. When one thing is 10 times bigger than it needs to be, no one will notice. When everything is 10 times bigger than it needs to be then it’s wasteful, and extremely difficult to fix. These features take up namespace. Got a shell script called source or complete? Can’t use it, those are builtins in bash. They slow things down. Normally I wouldn’t mention speed, but 8 years ago Ubuntu switched from bash to dash for the standard /bin/sh and the speed increase was enough to affect boot time. Probably part of the reason that bash is slower is simply that it’s bigger. There are more things it has to do or check even though you’re not making use of those features.

If you’re unlucky a feature you’ve never heard of and don’t use will interact with another feature or a part of your system and surprise you. If you’re really unlucky it will be a remote code exploit so easy to use you can tweet exploits, which is what ShellShock is. Did you know you can have functions in bash? Did you know you can export them to the environment? Did you know that the export feature works by executing the definition of the function? Did you know that it’s buggy and can execute more than bash expected? Did you know that with CGI you can set environment variables to arbitrary strings?

There are lots of little pieces to reason about when considering the ShellShock bug because bash is big. And that’s after we know about the bug. What about all those features of you don’t use and don’t even know about? Have you read and understood the bash man page? Well, those features you’ve never heard of are probably about as secure as the feature that exports functions to the environment, a feature that few people know about, and fewer people use (and in my opinion, no one should use).

The most important thing about security is attitude. It’s okay to have the attitude that a shell should have lots of useful interactive features; it’s arguable that a shell should have a rich programming environment that includes arrays and hash tables.

It’s not okay to argue that this piece of bloatware should be installed as the standard system shell.

10 tips for turning bash scripts into portable POSIX scripts

2014-09-26

In the light of ShellShock you might be wondering whether you really need bash at all. A lot of things that are bash-specific have portable alternatives that are generally only a little bit less convenient. Here’s a few tips:

1. Avoid «[[»

bash-specific:

if [[ $1 = yes ]]

Portable:

if [ "$1" = "yes" ]

Due to problematic shell tokenisation rules, Korn introduced the «[[» syntax into ksh in 1988 and bash copied it. But it’s never made it into the POSIX specs, so you should stick with the traditional single square bracket. Ahttps://drj11.wordpress.com/2014/09/26/10-tips-for-turning-bash-scripts-into-portable-posix-scripts/s long as you double quote all the things, you’ll be fine.

doublequote

2. Avoid «==» for testing for equality

bash-specific:

if [ "$1" == "yes" ]

Portable:

if [ "$1" = "yes" ]

The double equals operator, «==», is a bit too easy to use accidentally for old-school C programmers. It’s not in the POSIX spec, and the portable operator is single equals, «=», which works in all shells.

Technically when using «==» the thing on the right is a pattern. If you see something like this: «[[ $- == *i* ]]» then see tip 8 below.

3. Avoid «cd -»

bash-specific:

cd -

ksh and bash:

cd ~-

You tend to only see «cd -» used interactively or in weird things like install scripts. It means cd back to the previous place.

Often you can use a subshell instead:

... do some stuff in the original working directory
( # subshell
cd newplace
... do some stuff in the newplace
) # popping out of the subshell
... do some more stuff in the original working directory

But if you can’t use a subshell then you can always store the current directory in a variable:

old=$(pwd)

then do «cd “$old”» when you need to go there. If you must cling to the «cd -» style then at least consider replacing it with «cd ~-» which works in ksh as well as bash and is blessed as an allowed extention by POSIX.

4. Avoid «&>»

bash-specific:

ls &> /dev/null

Portable:

ls > /dev/null 2&>1

You can afford to take the time to do the extra typing. Is there some reason why you have to type this script in as quickly as possible?

5. Avoid «|&»

bash-specific:

ls xt yt |& tee log

Portable:

ls xt yt 2>&1 | tee log

This is a variant on using «&>» for redirection. It’s a pipeline that pipes both stdout and stderr through the pipe. The portable version is only a little bit more typing.

6. Avoid «function»

bash-specific:

function foo { ... }

Portable:

foo () { ... }

Don’t forget, you can’t export these to the environment. *snigger*

7. Avoid «((»

bash-specific:

((x = x + 1))

Portable:

x=$((x + 1))

The «((» syntax was another thing introduced by Korn and copied into bash.

8. Avoid using «==» for pattern matching

bash-specific:

if [[ $- == *i* ]]; then ... ; fi

Portable:

case $- in (*i*) ... ;; esac

9. Avoid «$’something’»

bash-specific:

nl=$'\n'

Portable:

nl='
'

You may not know that you can just include newlines in strings. Yes, it looks ugly, but it’s totally portable.

If you’re trying to get even more bizarre characters like ISO 646 ESC into a string you may need to investigate printf:

Esc=$(printf '\33')

or you can just type the ESC character right into the middle of your script (you might find Ctrl-V helpful in this case). Word of caution if using printf: while octal escapes are portable POSIX syntax, \377, hex escapes are not.

10. «$PWD» is okay

A previous version of this article said to avoid «$PWD» because I had been avoiding it since the dark ages (there was a time when some shells didn’t implement it and some did).

Conclusion

Most of these are fairly simple replacements. The simpler tokenisation that Korn introduced for «[[» and «((» is welcome, but it comes at the price of portability. I suspect that most of the bash-specific features are introduced into scripts unwittingly. If more people knew about portable shell programming, we might see more portable shell scripts.

I’m sure there are more, and I welcome suggestions in the comments or on Twitter.

Thanks to Gareth Rees who suggested minor changes to #3 and #7 and eliminated #10.