Mark Needham

Thoughts on Software Development

Archive for the ‘bash’ tag

Shell: Create a comma separated string

without comments

I recently needed to generate a string with comma separated values, based on iterating a range of numbers.

e.g. we should get the following output where n = 3


I only had the shell available to me so I couldn’t shell out into Python or Ruby for example. That means it’s bash scripting time!

If we want to iterate a range of numbers and print them out on the screen we can write the following code:

for i in $(seq 0 $(($n > 0? $n-1: 0))); do 
  echo "foo-$i"

Combining them into a string is a bit more tricky, but luckily I found a great blog post by Andreas Haupt which shows what to do. Andreas is solving a more complicated problem than me but these are the bits of code that we need from the post.

for i in $(seq 0 $(($n > 0? $n-1: 0))); do 
echo $combined

This won’t work if you set n<0 but that’s ok for me! I’ll let Andreas explain how it works:

  • ${combined:+,} will return either a comma (if combined exists and is set) or nothing at all.
  • In the first invocation of the loop combined is not yet set and nothing is put out.
  • In the next rounds combined is set and a comma will be put out.

We can see how it in action by printing out the value of $combined after each iteration of the loop:

for i in $(seq 0 $(($n > 0 ? $n-1: 0))); do 
  echo $combined

Looks good to me!

Written by Mark Needham

June 23rd, 2017 at 12:26 pm

Posted in Shell Scripting

Tagged with , ,

Finding ways to use bash command line history shortcuts

with 2 comments

A couple of months ago I wrote about a bunch of command line history shortcuts that Phil had taught me and after recently coming across Peteris Krumins’ bash history cheat sheet I thought it’d be interesting to find some real ways to use them.

A few weeks ago I wrote about a UTF-8 byte order mark (BOM) that I wanted to remove from a file I was working on and I realised this evening that there were some other files with the same problem.

The initial command read like this:

awk '{if(NR==1)sub(/^\xef\xbb\xbf/,"");print}' data/Taxonomy/Products.csv  > data/Taxonomy/Products.csv.bak

The version of the file without the BOM is data/Taxonomy/Products.csv.bak but I wanted it to be data/Taxonomy/Products.csv so I needed to mv it to that location.

By making use of history expansion we can write this as follows:

mv !$ !!:2

!$ represents the last argument which is data/Taxonomy/Products.csv.bak and !!:2 gets the 2nd argument passed to the last command which in this case is data/Taxonomy/Products.csv.

As you’re typing it will expand to the following:

mv data/Taxonomy/Products.csv.bak data/Taxonomy/Products.csv

One of the things that we do quite frequently is look at the nginx configurations and logs of our different applications which involved doing the following:

$ tail -f /var/log/nginx/site-1-access.log
$ tail -f /var/log/nginx/site-2-access.log


$ vi /etc/nginx/sites-enabled/site-1-really-long-name-cause-we-can
$ vi /etc/nginx/sites-enabled/site-2-really-long-name-cause-we-can

Everything except for the file name is the same but typing the up arrow to get the previous command and then manually deleting the file name can end up taking longer than just writing out the whole command again if the site name is long.

Ctrl-w deletes the whole path so that doesn’t help us either.

An alternative is the use the ‘h’ modifier which “Removes a trailing pathname component, leaving the head.”

In this case we could do the following:

$ vi /etc/nginx/sites-enabled/site-1-really-long-name-cause-we-can
$ vi !$:h/site-2-really-long-name-cause-we-can

We still have to type out the whole file name and we don’t get any auto complete help which is a bit annoying.

I realised that on my zsh if I type a space after a history expansion command it expands what I’ve typed to the full paths of everything, which is due to the following key binding:

.oh-my-zsh $ grep -rn "magic-space" *
lib/key-bindings.zsh:20:bindkey ' ' magic-space    # also do history expansion on space

We can do the same thing in bash by running the following command:

bind Space:magic-space

Then if I wanted to open that second nginx file I could do the following:

$ vi !$:h # then type a space which will expand it to:
$ vi /etc/nginx/sites-enabled/ # I can then type backspace, then type 'site-2' and tab and open the file

It’s not completely smooth because of the backspace but I think it’s marginally quicker than the other options.

Another one which I mentioned in the first post is the ^original^replacement which will run the previous command but replace the first instance of ‘original’ with ‘replacement’.

With this one it often seems faster to type the up arrow and change what you want manually or retype the command but when doing a grep of a specific folder I think this is faster.


$ grep -rn "magic-space" ~/.oh-my-zsh/lib
/Users/mneedham/.oh-my-zsh/lib/key-bindings.zsh:20:bindkey ' ' magic-space    # also do history expansion on space

Let’s say I was intrigued about bindkey and wanted to find all the instances of that.

One way to do that would be to type up and then manually go back along the line using Meta-B until I get to ‘bind-key’ when I can delete that with a few Ctrl-W‘s but in this case the search/replace approach is quicker:

$ ^magic-space^bindkey
grep -rn "bindkey" ~/.oh-my-zsh/lib
/Users/mneedham/.oh-my-zsh/lib/completion.zsh:24:bindkey -M menuselect '^o' accept-and-infer-next-history
/Users/mneedham/.oh-my-zsh/lib/completion.zsh:71:  bindkey "^I" expand-or-complete-with-dots

I’m still looking for other ways to re-use bash history more effectively so let me know any other cool tricks in the comments.

Written by Mark Needham

September 19th, 2012 at 7:00 am

Posted in Shell Scripting

Tagged with

Bash: Reusing previous commands

with 2 comments

A lot of the time when I’m using the bash shell I want to re-use commands that I’ve previously entered and I’ve recently learnt some neat ways to do this from my colleagues Tom and Kief.

If we want to list the history of all the commands we’ve entered in a shell session then the following command does the trick:

> history
  761  sudo port search pdfinfo
  762  to_ipad andersen-phd-thesis.pdf 
  763  vi ~/.bash_profile
  764  source ~/.bash_profile
  765  to_ipad andersen-phd-thesis.pdf 
  766  to_ipad spotify-p2p10.pdf 
  767  mkdir LinearAlgebra

If we want to execute any of those commands again then we can do that by entering ![numberOfCommand. For example, to execute the last command on that list we’d do this:

> !767
mkdir LinearAlgebra
mkdir: LinearAlgebra: File exists

We can also search the history and execute the last command that matches the search by doing the following:

> !mk
mkdir LinearAlgebra
mkdir: LinearAlgebra: File exists

A safer way to do this would be to suffix that with :p so the command gets printed to stdout rather than executed:

> !mk:p
mkdir LinearAlgebra

A fairly common use case that I’ve come across is to search for a file and then once you’ve found it open it in a text editor.

We can do this by using the !! command which repeats the previously executed command:

> find . -iname "someFile.txt"
> vi `!!`

We can achieve the same thing by wrapping ‘!!’ inside ‘$()’ as well:

> find . -iname "someFile.txt"
> vi $(!!)

Sam Rowe has a cool post where he goes into this stuff in even more detail.

I’m sure there are more tricks that I haven’t learnt yet so please let me know if you know some!

Written by Mark Needham

October 13th, 2011 at 7:46 pm

Posted in Shell Scripting

Tagged with ,