shell booleans are commands!

2014-10-03

How should we represent boolean flags in shell? A common approach, possibly inspired by C, is to set the variable to either 0 or 1.

Then you see code like this:


if [ $debug = 1 ]; then
...

or this example from zgrep:


if test $have_pat -eq 0; then
...

there is nothing special about 0 and 1, they are just two strings for repreresenting “the flag is set” and “the flag is unset”.

Test for strings is surprisingly awkward in shell. In Python you can go if debug: .... It would be nice if we could do something similar in shell:


if $debug ; then
...
fi

Well we can. In a shell if statement, if thing, the thing is just a command. If we arrange that debug is either true or false, then if $debug will run either the command true or the command false.


debug=true # sets flag
debug=false # unsets flag

I wish I could remember who I learnt this trick off because I think it’s super cool, and not enough shell programmers know about it. true and false are pretty much self explanatory as boolean values, and no extra code is needed because they already exist as shell commands.

You can also use this with &&:


$debug && stuff

Sometimes shell scripts have a convention where a variable is either unset (to mean false) or set to anything (to mean true). You can convert from this convention to the true/false convention with 2 lines of code:


# if foo is set to anything, set it to "true"
foo=${foo+true}
# if foo is the empty string, set it to "false"
foo=${foo:-false}


bash functions: it is mad!

2014-10-02

bash can export functions to the environment. A consequence of this is that bash can import functions from the environment. This leaves us #shellshocked. #shellshock aside, why would anyone want to do this? As I Jackson says: “it is mad”.

Exporting bash functions allows a bash script, or me typing away at a terminal in bash, to affect the behaviour of basically any other bash program I run. Or a bash program that is run by any other program.

For example, let’s say I write a program in C that prints out the help for the zgrep program:

#include <stdlib.h>

int main(void)
{
    return system("zgrep --help");
}

This is obviously just a toy example, but it’s not unusual for Unix programs to call other programs to do something. Here it is in action:


drj$ ./zgrephelp
Usage: /bin/zgrep [OPTION]... [-e] PATTERN [FILE]...
Look for instances of PATTERN in the input FILEs, using their
uncompressed contents if they are compressed.

OPTIONs are the same as for 'grep'.

Report bugs to <bug-gzip@gnu.org>.

Now, let’s say I define a function called test in my interactive bash session:


test () { bob ; }

This is unwise (test is the name of a well know Unix utility), but so far only harmful to myself. If I try and use test in my interactive session, things go a bit weird:


drj$ test -e /etc/passwd
The program 'bob' is currently not installed. You can install it by typing:
sudo apt-get install python-sponge

but at least I can use bash in other processes and it works fine:


drj$ bash -c 'test -e /etc/passwd' ; echo $?
0

What happens if I export the function test to the environment?


drj$ export -f test
drj$ ./zgrephelp
/bin/zgrep: bob: command not found
/bin/zgrep: bob: command not found
/bin/zgrep: bob: command not found
/bin/zgrep: bob: command not found
/bin/zgrep: bob: command not found
gzip: /bin/zgrep: bob: command not found
--help.gz: No such file or directory
/bin/zgrep: bob: command not found
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
/bin/zgrep: bob: command not found
/bin/zgrep: bob: command not found
/bin/zgrep: bob: command not found

zgrephelp stops working. Remember, zgrephelp is written in C! Of course, zgrephelp runs the program zgrep which is written in… bash! (on my Ubuntu system).

Exporting a function can affect the behaviour of any bash script that you run, including bash scripts that are run on your behalf by other programs, even if you never knew about them, and never knew they were bash scripts. Did you know /bin/zcat is a bash script? (on Ubuntu)

How is this ever useful? Can you ever safely export a function? No, not really. Let’s say you export a function called X. Package Y might install a binary called X and a bash script Z that calls X. Now you you’ve broken Z. So you can’t export a function if it has the same name as a binary installed by any package that you ever might install (including packages that are you never use directly but are installed merely to compile some package that you do want to use).

Let’s flip this around, and consider the import side.

When a bash script starts, before it’s read a single line of your script it will import functions from the environment. These are just environment variables of the form BASH_FUNC_something()=() { function defintion here ; }. You don’t have to create those by exporting a function, you can just create an environment variable of the right form:


drj$ env 'BASH_FUNC_foo()=() { baabaa ; }' bash -c foo
bash: baabaa: command not found

Imagine you are writing a bash script and you are a conscientious programmer (this requires a large amount of my imagination). bash will import potentially arbitrary functions, with arbitrary names, from the environment. Can you prevent these functions being defined?

It would appear not.

Any bash script should carefully unset -f prog for each prog that it might call (including builtins like cd; yes you can define a function called cd).

Except of course, that you can’t do this if unset has been defined as a function.

Why is exporting functions ever useful?


How to patch bash

2014-09-26

3rd in what is now becoming a series on ShellShock. 1st one: 10 tips for turning bash scripts into portable POSIX scripts; 2nd one: why big is bad.

ShellShock is a remote code exploit in /bin/bash (when used in conjunction with other system components). It relies on the way bash exports functions to its environment. If you run the command «env - bash -c 'foo () { hello ; } ; export -f foo ; env'» you can see how this works ordinarily:


PWD=/home/drj/bash-4.2/debian/patches
SHLVL=1
foo=() { hello
}
_=/usr/bin/env

The function foo when exported to the environment turns into an environment variable called foo that starts with «() {». When a new bash process starts it scans its environment to see if there are any functions to import and it is that sequence of characters, «() {», that triggers the import. It imports a function by executing its definition which causes the function to be defined.

The ShellShock bug is that there is a problem in the way it parses the function out of the environment which causes it to execute any code after the function definition to be also executed. That’s bad.

So the problem is with parsing the function definition out of the environment.

Fast forward 22 years after this code was written. That’s present day. S Chazelas discovers that this behaviour can lead to a serious exploit. A patch is issued.

Is this patch a 77 line monster that adds new functionality to the parser?

Why, yes. It is.

Clearly function parsing in bash is already a delicate area, that’s why there’s a bug in it. That should be a warning. The fix is not to go poking about inside the parser.

The fix, as suggested by I Jackson, is to “Disable exporting shell functions because they are mad“. It’s a 4 line patch (well, morally 4 lines) that just entirely removes the notion of importing functions from the environment:

--- a/variables.c
+++ b/variables.c
@@ -347,6 +347,7 @@ initialize_shell_variables (env, privmode)
 
       temp_var = (SHELL_VAR *)NULL;
 
+#if 0 /* Disable exporting shell functions because they are mad. */
       /* If exported function, define it now.  Don't import functions from
         the environment in privileged mode. */
       if (privmode == 0 && read_but_dont_execute == 0 && STREQN ("() {", string, 4))
@@ -380,6 +381,9 @@ initialize_shell_variables (env, privmode)
              report_error (_("error importing function definition for `%s'"), name);
            }
        }
+#else
+      if (0) ; /* needed for syntax */
+#endif
 #if defined (ARRAY_VARS)
 #  if ARRAY_EXPORT
       /* Array variables may not yet be exported. */

In a situation where you want to safely patch a critical piece of infrastructure you do the simplest thing that will work. Later on you can relax with a martini and recraft that parser. That’s why Jackson’s patch is the right one.

Shortly after the embargo was lifted on the ShellShock vulnerability and the world got to know about it and see the patch, someone discovered a bug in the patch and so we have another CVN issued and another patch that pokes about with the parser.

That simply wouldn’t have happened if we’d cut off the head of the monster and burned it with fire. /bin/bash is too big. It’s not possibly to reliably patch it. (and just in case you thinking making patches is easy I had to update this article to point to Jackson’s corrected patch)

Do you really think this is the last patch?


Why big is bad

2014-09-26

If you know me, you know that I don’t like using /bin/bash for scripting. It’s not that hard to write scripts that are portable, and my earlier “10 tips” article might help.

Why don’t I like /bin/bash? There are many reasons, but it’s mostly about size.


drj$ ls -lL $(which sh bash)
-rwxr-xr-x 1 root root 959120 Sep 22 21:39 /bin/bash
-rwxr-xr-x 1 root root 109768 Mar 29 2012 /bin/sh

/bin/bash is nearly 10 times the size of /bin/sh (which in this case, is dash). It’s bigger because it’s loaded with features that you probably don’t need. An interactive editor (two in fact). That’s great for interactive use, but it’s just a burden for non-interactive scripts. Arrays. Arrays are really super useful and fundamental to many algorithms. In a real programming language. If you need arrays, it’s time for your script to grow up and become a program, in Python, Lua, Go, or somesuch.

Ditto job control.
Ditto Extended Regular Expression matching.
Ditto mapfile.
Ditto a random number generator.
Ditto a TCP/IP stack.

You might think that these things can’t harm you if you don’t use them. That’s not true. We have a little bit of harm just by being bigger. When one thing is 10 times bigger than it needs to be, no one will notice. When everything is 10 times bigger than it needs to be then it’s wasteful, and extremely difficult to fix. These features take up namespace. Got a shell script called source or complete? Can’t use it, those are builtins in bash. They slow things down. Normally I wouldn’t mention speed, but 8 years ago Ubuntu switched from bash to dash for the standard /bin/sh and the speed increase was enough to affect boot time. Probably part of the reason that bash is slower is simply that it’s bigger. There are more things it has to do or check even though you’re not making use of those features.

If you’re unlucky a feature you’ve never heard of and don’t use will interact with another feature or a part of your system and surprise you. If you’re really unlucky it will be a remote code exploit so easy to use you can tweet exploits, which is what ShellShock is. Did you know you can have functions in bash? Did you know you can export them to the environment? Did you know that the export feature works by executing the definition of the function? Did you know that it’s buggy and can execute more than bash expected? Did you know that with CGI you can set environment variables to arbitrary strings?

There are lots of little pieces to reason about when considering the ShellShock bug because bash is big. And that’s after we know about the bug. What about all those features of you don’t use and don’t even know about? Have you read and understood the bash man page? Well, those features you’ve never heard of are probably about as secure as the feature that exports functions to the environment, a feature that few people know about, and fewer people use (and in my opinion, no one should use).

The most important thing about security is attitude. It’s okay to have the attitude that a shell should have lots of useful interactive features; it’s arguable that a shell should have a rich programming environment that includes arrays and hash tables.

It’s not okay to argue that this piece of bloatware should be installed as the standard system shell.


10 tips for turning bash scripts into portable POSIX scripts

2014-09-26

In the light of ShellShock you might be wondering whether you really need bash at all. A lot of things that are bash-specific have portable alternatives that are generally only a little bit less convenient. Here’s a few tips:

1. Avoid «[[»

bash-specific:

if [[ $1 = yes ]]

Portable:

if [ "$1" = "yes" ]

Due to problematic shell tokenisation rules, Korn introduced the «[[» syntax into ksh in 1988 and bash copied it. But it’s never made it into the POSIX specs, so you should stick with the traditional single square bracket. As long as you double quote all the things, you’ll be fine.

doublequote

2. Avoid «==» for testing for equality

bash-specific:

if [ "$1" == "yes" ]

Portable:

if [ "$1" = "yes" ]

The double equals operator, «==», is a bit too easy to use accidentally for old-school C programmers. It’s not in the POSIX spec, and the portable operator is single equals, «=», which works in all shells.

Technically when using «==» the thing on the right is a pattern. If you see something like this: «[[ $- == *i* ]]» then see tip 7 below.

3. Avoid «cd -»

bash-specific:

cd -

ksh and bash:

cd ~-

You tend to only see «cd -» used interactively or in weird things like install scripts. It means cd back to the previous place.

Often you can use a subshell instead:

... do some stuff in the original working directory
( # subshell
cd newplace
... do some stuff in the newplace
) # popping out of the subshell
... do some more stuff in the original working directory

But if you can’t use a subshell then you can always store the current directory in a variable:

old=$(pwd)

then do «cd “$old”» when you need to go there. If you must cling to the «cd -» style then at least consider replacing it with «cd ~-» which works in ksh as well as bash and is blessed as an allowed extention by POSIX.

4. Avoid «&>»

bash-specific:

ls &> /dev/null

Portable:

ls > /dev/null 2&>1

You can afford to take the time to do the extra typing. Is there some reason why you have to type this script in as quickly as possible?

5. Avoid «|&»

bash-specific:

ls xt yt |& tee log

Portable:

ls xt yt 2>&1 | tee log

This is a variant on using «&>» for redirection. It’s a pipeline that pipes both stdout and stderr through the pipe. The portable version is only a little bit more typing.

6. Avoid «function»

bash-specific:

function foo { ... }

Portable:

foo () { ... }

Don’t forget, you can’t export these to the environment. *snigger*

7. Avoid «((»

bash-specific:

((x = x + 1))

Portable:

x=$((x + 1))

The «((» syntax was another thing introduced by Korn and copied into bash.

8. Avoid using «==» for pattern matching

bash-specific:

if [[ $- == *i* ]]; then ... ; fi

Portable:

case $- in (*i*) ... ;; esac

9. Avoid «$’something’»

bash-specific:

nl=$'\n'

Portable:

nl='
'

You may not know that you can just include newlines in strings. Yes, it looks ugly, but it’s totally portable.

If you’re trying to get even more bizarre characters like ISO 646 ESC into a string you may need to investigate printf:

Esc=$(printf '\33')

or you can just type the ESC character right into the middle of your script (you might find Ctrl-V helpful in this case). Word of caution if using printf: while octal escapes are portable POSIX syntax, \377, hex escapes are not.

10. «$PWD» is okay

A previous version of this article said to avoid «$PWD» because I had been avoiding it since the dark ages (there was a time when some shells didn’t implement it and some did).

Conclusion

Most of these are fairly simple replacements. The simpler tokenisation that Korn introduced for «[[» and «((» is welcome, but it comes at the price of portability. I suspect that most of the bash-specific features are introduced into scripts unwittingly. If more people knew about portable shell programming, we might see more portable shell scripts.

I’m sure there are more, and I welcome suggestions in the comments or on Twitter.

Thanks to Gareth Rees who suggested minor changes to #3 and #7 and eliminated #10.


Why I gave up beef

2014-08-15

or “peer reviewed article changed behaviour”, or “this one crazy trick will reduce your land use by 66%!”.

Land use of animal calories

I had known for a while that eating meat was resource intensive, and beef was particularly bad (mostly from MacKay’s Sustainable Energy Without the Hot Air), and over the last couple of years I have been trying to reduce my meat intake. Sometimes I claimed I was “mostly vegetarian” (with some success, a friend had known me for a few weeks before realising, as I chewed my bacon one lunchtime, that I wasn’t vegetarian). Recently a friend gave up beef, and as I said at the time it was probably a better environmental commitment than my “mostly vegetarian”. But it wasn’t until I saw the numbers crunched in Eshel et al 2014 that I decided to eliminate beef.

It is no longer reasonable to entertain doubt as to the environmental impact of beef.

The graph I show above is for land use per megacalorie. I had to redraw it from Eshel et al 2014 figure 2 because… Well, why don’t you see:

esheletal2014figure2

In order to fit each graph into its tiny rectangle the long bars have been truncated and the extra long bit that has been removed has been replaced with a tiny number giving the coordinate that should have been plotted. For beef land use it’s 147 m²·yr (compared to poultry which is 4). So where the graph should show a spike that’s 40 times bigger, instead it shows one that’s 4 or 5 times. And this is their headline figure. The whole point of the article is to show how much more resource intensive beef is. Are the PNAS page charges really so high that they have to cram all the graphs into one corner?

I’ve just shown the land use figure, but Eshel et al 2014 have analyses for water, greenhouse gasses, and reactive nitrogen. Tiny little arrows give numbers for potato, wheat, and rice (which are on the whole a lot smaller, except the rice’s use of water). You can explore the Supplementary Information too, including the spreadsheet they used.

Obviously peer review is not perfect (it is merely evidence that a couple of reviewers ran out of reasons to delay its publication), and there are caveats. This studies only US beef. What about Europe? What about ostriches? What about food miles? But I think you would be foolish to think that these other matters would affect the central conclusion: eating beef uses a lot of resources.


Nick Clegg on DRIP

2014-07-15

I used WriteToThem to write to Nick Clegg, my MP; asking him how he intended to vote on the Data Retention and Investigatory Powers Bill. He didn’t say. But he did make this reply:

Thank you for your email regarding the Data Retention and Investigation Powers Bill.

The bill ensures that critical capabilities to fight crime and protect the public are maintained; it clarifies existing law without extending current powers. Interception and access to communications data are critical to the ability of our law enforcement and intelligence agencies to fight crime and protect the public. Furthermore, the bill makes clear that anyone providing a communications service to customers in the UK, regardless of where that service is provided from, should comply with lawful requests made under the Regulation of Investigatory Powers Act 2000. It also replaces the current Regulations under which domestic companies can be required to retain certain types of communications data for up to 12 months, so this may later be acquired by law enforcement and used in evidence. In addition, the bill will introduce further safeguards for the use of investigatory powers, building on our already stringent regime, to respond to criticisms raised by the European Court of Justice. If we fail to act immediately, there would be a critical gap in our investigative capabilities.

As a Liberal Democrat I believe that successive governments have neglected civil liberties as they claimed to pursue greater security. There is a real risk that we will suddenly be deprived of the legitimate means by which we keep people safe. Liberty and security must go hand in hand; we cannot enjoy our freedom if we are unable to keep ourselves safe. I therefore believe there is an urgent challenge facing us. As the Prime Minister has set out, we face the very stark prospect that powers we have taken for granted in the past, will no longer be available to us in the coming weeks.

Communications data and lawful intercept are now amongst the most useful tools available to us to prevent violence and bloodshed on Britain’s streets. Vital to the work of the police and the agencies is lawful intercept, which allows them to look, under warrant, at specific communications between individuals, who may be planning plots, and without retained communications data, we wouldn’t, for example, be able to piece together the web of relationships between members of the organised crime gangs responsible for trafficking drugs and vulnerable children and adults into our country.

It must be crucially noted that this bill has nothing to do with the so-called ‘Snooper’s Charter’. That was a Home Office proposal to store every website you’ve ever visited for a whole year. I blocked that last year and I’ve blocked every further attempt to bring it back. Moreover, this bill is about maintaining existing capabilities, not creating new powers. The legislation will restore the data retention powers we had before, but amend the law to do it in a more proportionate way, and it will help companies which currently provide assistance with UK intercept warrants to continue to do so.

Most importantly, I have only agreed to this emergency legislation because we can use it to kick-start a proper debate about freedom and security in the internet age. In the post-Snowdon age, people are, rightly demanding to know more about what the state does on our behalf. There are fundamental questions to be asked about the scope of existing powers; about whether the Regulation of Investigatory Powers Act has kept pace with technology; about how nation states grapple with a global internet; and about how we continue to protect security, privacy and liberties while safeguarding our security. We cannot answer these big questions on the hoof, and that is why I’ve insisted that the legislation only extends the existing powers for a temporary period. We’ve inserted a termination clause in the bill that means the legislation falls at the end of 2016, so the next government is forced to look again at these big issues.

Moreover, we will introduce new checks and balances to protect privacy and civil liberties for the future. We will establish, for the first time, a Privacy and Civil Liberties Oversight Board on the American model to ensure that Civil Liberties are properly considered in the formulation of government policy on counter-terrorism. We will radically cut the number of public bodies who have the right to approach phone and internet companies for your data. I don’t believe, for example, that local councils should unilaterally be able to access your communications data. From now on, councils will need to justify their requests first to a central body, and then a magistrate, and will not be able to approach phone and internet companies directly.

We will take practical steps to improve the accountability and transparency of our laws, by publishing regular transparency reports, listing new details about exactly how many warrants are issued, by whom, and for what purposes. The public will know more about how and why surveillance powers are administered on their behalf than ever before.

We have been working on this bill on a cross-party basis to achieve a balanced package which restores necessary powers while at the same time taking big new steps to defend our civil liberties.

I hope that this helps to answer your concerns. Please do not hesitate to get in touch again on this matter.

Yours sincerely,

Nick Clegg MP


Piping into shell may be harmful

2014-03-19

Consider

curl https://thing | sh

It has become fashionable to see this kind of shell statement as a quick way of installing various bits of software (nvm, docker, salt).

This is a bad idea.

Imagine that due to bad luck the remote server crashes halfway through sending the text of the shell script, and a line that reads

rm -fr /usr/local/go

gets truncated and now reads:

rm -fr /

You’re hosed.

curl may write a partial file to the pipe and sh has no way of knowing.

Can we defend against this?

Initially I was pessimistic. How can we consider all possible truncations of a shell program? But then I realised, after a couple of false turns, that it’s possible to construct a shell script that is syntactically valid only when the entire script has been transmitted; truncating this special shell script at any point will result in a syntactically invalid script which shell will refuse to execute.

Moreover this useful property of being syntactically valid only when complete is not just a property of a small specially selected set of shell scripts, it’s entirely general. It turns out to be possible to transform any syntactically valid shell script into one that has the useful property:

{
...
any old script here
...
}

We can bracket the entire script with the grouping keyword { and }. Now if the script gets cutoff somewhere in the middle, the { at the beginning will be missing its matching } and be syntactically invalid. Shell won’t execute the partial script.

As long as the script in the middle is syntactically valid, then the bracketed script will be syntactically valid too.

Let’s call these curly brackets that protect the script from truncation, Jones truncation armour.

Clearly Jones truncation armour should be applied to all scripts that may be piped directly into shell.

Can it be applied as an aftermarket add-on? Yes it can!

{ echo { && curl https://thing && echo } ; } | sh

Maybe this is even better. It means that the consumer of the script doesn’t have to rely on the provider to add the Jones truncation armour. But it also doesn’t matter if the script is already armoured. It still works.


Coding is like Cooking

2014-03-14

Coding is like cooking.

Well, not really. But a bit. This is not an article about how recipes are like programs, it’s about the role that cooking has in our personal lives and in society.

I can cook. A bit. Well enough that I can cook for my household, and friends that might drop by. I don’t always eat frozen pizza. Day to day cooking I can mostly do without a written recipe (spag bol, salmon and broccoli, that kind of thing), but when we entertain I’ll generally use a recipe; we own a few too many cookbooks and I can find recipes online. Perhaps one or two dishes I can make well enough that they’re actually good. So I can cook, but not well enough that anyone would pay me to do it. And as for being a chef there are probably other skills that professional cooks have that are part of the job that are simply not on my radar. Planning a menu, choosing suppliers, managing a kitchen.

I’m not suggesting that because I can cook a bit that I’ll be a cook. But conversely just because I can’t be a chef, that doesn’t mean that cooking is pointless. I don’t cook because it’s useful to the economy or because it helps me get a better job. I don’t do it as a hobby. I cook because it’s useful to me personally. It’s a sort of basic life skill.

I imagine most people are like me in this regard. They can cook, something. The amount of cooking that people do might vary. Some people will do it as a hobby, cooking things for their friends every week. Some people will do it professionally and cook for hundreds or thousands of people.

I would like programming to be like this.

I think most people can program. A bit. Not to a professional level, not to a standard where they would be comfortable getting paid to do it. Some people might like programming a bit more and do it as a hobby. Again, doesn’t mean they would be paid to do it.

You don’t have to be a programmer to program. Just like you don’t have to be a cook to cook.

Tinkering

I have friends who I think of as cooks. Some of them do it as a hobby, some professionally. One thing I’ve noticed is that they enjoy cooking, and they like to tinker. They’ll try cooking something just for fun on a Saturday morning. The professionals will try some new technique or ingredient or idea because it will expand their power. I don’t do that. Not with cooking anyway. But I do with programming. I’ll write a browser-based Logo implementation to learn a bit about SVG and JavaScript. Or I’ll learn a new language because it seems fun and might stretch me intellectually.

Are there problems with this analogy? Yes there are:

  • Cooking is useful in itself. The result is usually a meal; you can eat it. This is not so clearly true of programming. There are useful programs to be written, but by and large the kinds of programs that non-professional coders write are not all that useful (yet; see below).
  • Terminology. Lots of people (professionals and hobbyists alike) seem to think that when you say “teach people to program” you mean “train people to be professional programmers”. We don’t. Or at least, I don’t. No more than “teaching people to cook” means “teaching people to be cooks”.
  • Access

    The access route to cooking is pretty straightforward. Most homes and (all?) schools have a hob or a cooker or a microwave. You can cook something with that. I don’t have to buy a stainless steel counter and professional range to cook. You shouldn’t need “pro-level” tools to program. You shouldn’t have to buy a specialist computer and learn how to use an editor like vi, emacs, or TextMate.

    So I think we have to make the access route to programming simpler.

    Tools like ScraperWiki’s Code in your Browser tool help (full disclosure: I work for the company that made this), as may things like CoffeePlay (note I said things like CoffeePlay, which is a whimsical tool created over a weekend, not a serious product). You can start programming using just your browser. There is a browser based version of MIT’s Scratch programming environment. The Raspberry Pi helps.

    At the moment, I think these things show possible futures. They are hints at how we can make coding more accessible. But I think it is possible to make the route to programming simpler. A child can take the first steps in cooking by mixing flour and butter, and putting scones in the oven. We need to make programming just as easy and accessible.

    We need to make it easier to do more things with code. What I mean here is more hackable things in the spirit of the maker community, hack spaces, and so on. It’s kind of neat that I can log into my Kobo ereader and modify the software, it’s a shame I can’t do that with my Kindle so easily. I really love my label printer, but I wish I could hack the firmware on it.

    I don’t think everyone should be a cook, but I think everyone should cook.


    Traditional approach to ATtiny programming

    2014-03-07

    I recently bought an Adafruit Gemma. It’s a little programming board that is slightly bigger than a 10p coin and it costs about GBP 7.

    It uses an ATtiny85 micro and is Arduino compatible, so the way you’re encouraged to program is to use the Arduino GUI tools and all that good stuff.

    By “traditional approach” I mean grumpy old man approach. I don’t like GUIs much. I can’t use vi and the rendering of the font in the editor is terrible. And the syntax highlighting burns my eyes.

    So you can just use the command line tools, right? Right. On Ubuntu you can apt-get install gcc-avr and then use the avr-gcc compiler to compile your C code. You’ll need avr-objcopy (from the binutils-avr package) to convert your .elf file into an Intel .hex file, and you’ll need avrdude (from the avrdude package) to flash the device. The gory details are captured in this Makefile, and I got those gory details by switching the Arduino GUI into its verbose mode and watching it compile my project.

    My first demo project is also an exercise in avoiding the Arduino libraries. Mostly because when I was working out how to use the command line tools, I didn’t want the hassle of dealing with multiple files and libraries and things.

    So this is also an example of how to program the ATtiny85 (and more or less any AVR type micro) without using heavyweight libraries. The Gemma has a built-in red LED on PB1. This was definitely one of the things that attracted me to the Gemma. I can program it do something without needing to plug any extra hardware in. Specifically, I can flash the LED.

    Flashing an LED is a matter of using a GPIO pin and driving it high (on) and low (off). The assembler programmer would do that with the SBI (Set BIt) and CBI (Clear BIt) instructions. So I’m thinking “Can we have reasonable looking C code generate The Right Instruction?”.

    The C code to set a bit is generally of the form *p |= b where p is a pointer to some memory location and b is a number with a single bit set (a power of 2 in other words). Similarly the C code to clear a bit is *p &= ~b. As long as we give the compiler enough information, it should be able to compile the code *p |= b into an SBI instruction.

    And so it can. Through some fairly tedious but also fairly ordinary C macros, I can write BIT_SET(PORTB, 1) to set pin 1 or PORTB (PB1, the pin with the LED attached), and it gets converted into roughly: *(volatile uint8_t *)0x38 |= 2; which is basically saying modify memory location 0x38 by setting its bit 1. In a little oddity of the AVR architecture the SBI and CBI instructions operate on IO addresses which are at memory locations 0x20 onwards. The upshot of this is that memory location 0x38 is modified with a SBI 0x18 instruction (this mystified me for about 2 hours last night, and I realised what was wrong just as I was drifting off to sleep).

    Because in the code *(volatile uint8_t *)0x38 |= 2; both the location, 0×38, and the value, 2, are constant, the compiler has everything it needs to generate the right SBI instruction. And it does!

    drj$ avr-objdump -d main.elf 2>&1 | sed -n '/<main>:/,$p' | sed 9q
    00000040 <main>:
      40:	1f 93       	push	r17
      42:	b9 9a       	sbi	0x17, 1	; 23
      44:	c1 9a       	sbi	0x18, 1	; 24
      46:	88 ec       	ldi	r24, 0xC8	; 200
      48:	90 e0       	ldi	r25, 0x00	; 0
      4a:	f2 df       	rcall	.-28     	; 0x30 <delay>
      4c:	c1 98       	cbi	0x18, 1	; 24
      4e:	88 ec       	ldi	r24, 0xC8	; 200
    

    You can see at the beginning of the disassembly of main the SBI 0x17, 1 instruction which is as a result of the macro BIT_SET(DDRB, 1) (setting the pin to be a digital output). And you can see SBI 0x18, 1 to drive the pin high and light the LED and CBI 0x18, 1 to drive the pin low. The compiler has even subtracted 0x20 from the addresses.

    avr-gcc -Os FTW!


    Follow

    Get every new post delivered to your Inbox.