Mark Needham

Thoughts on Software Development

Archive for the ‘shell’ tag

Shell: Create a comma separated string

without comments

I recently needed to generate a string with comma separated values, based on iterating a range of numbers.

e.g. we should get the following output where n = 3


I only had the shell available to me so I couldn’t shell out into Python or Ruby for example. That means it’s bash scripting time!

If we want to iterate a range of numbers and print them out on the screen we can write the following code:

for i in $(seq 0 $(($n > 0? $n-1: 0))); do 
  echo "foo-$i"

Combining them into a string is a bit more tricky, but luckily I found a great blog post by Andreas Haupt which shows what to do. Andreas is solving a more complicated problem than me but these are the bits of code that we need from the post.

for i in $(seq 0 $(($n > 0? $n-1: 0))); do 
echo $combined

This won’t work if you set n<0 but that’s ok for me! I’ll let Andreas explain how it works:

  • ${combined:+,} will return either a comma (if combined exists and is set) or nothing at all.
  • In the first invocation of the loop combined is not yet set and nothing is put out.
  • In the next rounds combined is set and a comma will be put out.

We can see how it in action by printing out the value of $combined after each iteration of the loop:

for i in $(seq 0 $(($n > 0 ? $n-1: 0))); do 
  echo $combined

Looks good to me!

Written by Mark Needham

June 23rd, 2017 at 12:26 pm

Posted in Shell Scripting

Tagged with , ,

cURL and the case of the carriage return

without comments

We were doing some work this week where we needed to make a couple of calls to an API via a shell script and in the first call we wanted to capture one of the lines of the HTTP response headers and use that as in input to the second call.

The way we were doing this was something like the following:

# We were actually grabbing a different header but for the sake 
# of this post we'll say it was 'Set-Cookie'
AUTH_HEADER=`curl -I | grep Set-Cookie`

When we echoed $AUTH_HEADER it looked exactly as we’d expect…

$ ./blah.txt 2>/dev/null
Set-Cookie: NID=63=gwfYa4fhbdqYyEdySrFn1AYybExgjQbQUKPdC5sZ5orRznGY-bt3gTwlc0XaPXv
expires=Sun, 17-Mar-2013 08:28:25 GMT; path=/;; HttpOnly

…but when we passed that value into the next cURL command it was returning a 401 response code which suggested that we hadn’t even sent the header at all.

We changed the code so that we manually assigned AUTH_HEADER with the correct value and then everything worked fine which suggested there was something weird in the value we were getting back from cURL.

We were constructing the arguments to our next cURL command like so:

# We were actually grabbing a different header but for the sake 
# of this post we'll say it was 'Set-Cookie'
AUTH_HEADER=`curl -I | grep Set-Cookie`
echo $ARGS

When we ran that we noticed that $ARGS was displaying some quite strange behaviour where the text after $AUTH_HEADER was overriding the value of $AUTH_HEADER:

$ ./blah.txt 2>/dev/null
Set-Cookie: NID=63=rma3ah7oBhyirDUqFPODHfaTK9XOqs0CPapYVgTM6vHyCgDTcXs2P_mVDI_hnsap
expires=Sun, 17-Mar-2013 08:37:47 GMT; path=/;; HttpOnly
Set-Cookie: NID=63=rma3ah7oBhyirDUqFPODHfaTK9XOqs0CPapYVgTM6vHyCgDTcXs2P_mVDI_hnsap
expires=Sun, 17-Mar-2013 08:37:47 GMT; path=/; dom RANDOM TEXT SO RANDOMpOnly

Paul was wondering by at the time so we asked him if he could think of anything that could be leading to what we were seeing. He suggested there was probably a carriage return lurking at the end of the line.

Nick showed us how we could prove that was the case using xxd:

xxd <<< $AUTH_HEADER

When we ran the script again we could see the carriage return character (0d) at the end of the line:

$ ./blah.txt 2>/dev/null
Set-Cookie: NID=63=QDECY69302tLN0CSMyug-TzzczxzGNWs70i8huV60qM7BFv18F63dNSz4trqzHXvzbKXNLb
expires=Sun, 17-Mar-2013 08:55:37 GMT; path=/;; HttpOnly
0000160: 2f3b 2064 6f6d 6169 6e3d 2e67 6f6f 676c  /; domain=.googl
0000170: 652e 636f 2e75 6b3b 2048 7474 704f 6e6c; HttpOnl
0000180: 790d 0d0a                                y..

Nick then showed us how to get rid of it using tr like so:

AUTH_HEADER=`curl -I | grep Set-Cookie` | tr -d '\r'`

Written by Mark Needham

September 15th, 2012 at 9:06 am

Posted in Shell Scripting

Tagged with

Bash: Piping data into a command using heredocs

with 2 comments

I’ve been playing around with some data modelled in neo4j recently and one thing I wanted to do is run an adhoc query in the neo4j-shell and grab the results and do some text manipulation on them.

For example I wrote a query which outputted the following to the screen and I wanted to sum together all the values in the 3rd column:

| ["1","2","3"]         | "3"                             | 1234567    |   
| ["4","5","6"]         | "6"                             | 8910112    |

Initially I was pasting the output into a text file and then running the following sequence of commands to work it out:

$ cat blah2.txt| cut -d"|" -f 4  | awk '{s+=$0} END {print s}'  

One way to avoid having to create blah2.txt would be to echo the output into standard out like so:

$ echo "| ["1","2","3"]         | "3"                             | 1234567    |   
| ["4","5","6"]         | "6"                             | 8910112    | " | cut -d"|" -f 4  | awk '{s+=$0} END {print s}'   

But it gets a bit confusing as the number of lines of results increases and you have to keep copy/pasting the cut and awk parts of the chain around which was annoying.

One of the things I read on the bus this week was a blog post going through a bunch of bash one liners and half way through it covers piping data into commands using heredocs which I’d completely forgotten about!

A simple example could be to send a simple message to cat which will output the message to standard out:

$ cat <<EOL
heredoc> hello i am mark
heredoc> EOL
hello i am mark

That works if we want to pipe data into a single command but I didn’t know how we’d be able to pipe the output of that command to another command.

In fact it’s actually reasonably simple:

$ cat <<EOL | cut -d"|" -f 4  | awk '{s+=$0} END {print s}' 
pipe pipe heredoc> | ["1","2","3"]         | "3"                             | 1234567    |   
pipe pipe heredoc> | ["4","5","6"]         | "6"                             | 8910112    | 
pipe pipe heredoc> EOL

And now I have no need to create random text files all over my machine!

Written by Mark Needham

September 15th, 2012 at 7:54 am

Posted in Shell Scripting

Tagged with

Unix: Caught out by shell significant characters

without comments

One of the applications that Phil and I were deploying today needed a MySQL server and part of our puppet code to provision that node type runs a command to setup the privileges for a database user.

The unevaluated puppet code reads like this:

/usr/bin/mysql -h ${host} -uroot ${rootpassarg} -e "grant all on ${name}.* to ${user}@'${remote_host}' identified by '$password'; flush privileges;"

In the application we were deploying that expanded into something like this:

/usr/bin/mysql -h localhost -uroot root_pw -e "grant all on db_name.* to db_user@'%' identified by 'awe$ome+password'; flush privileges;"

Unfortunately when we ran puppet it was executing without any problems but when we tried to connect to MySQL using ‘db_user’ with a password of ‘awe$ome+password’ we kept being denied access.

We tried changing the password to ‘bob’ to see what would happen, expecting that to fail as well, but were actually able to login so we figured there was something wrong with the password.

Phil suggested echoing the command to see what it was being evaluated to in the shell and once we did that we realised that the password was actually being set to ‘awe+password’ because the $ome bit was being evaluated as a shell variable.

This happens because shell variables are evaluated if they are enclosed in “” which is the case here as our whole grant statement is enclosed in “”. If variables are enclosed in ” then they won’t be evaluated:

$ mark="foo"; echo "$mark"
$ mark="foo"; echo '$mark'

In this case we can therefore switch the ” and “” around to solve the problem:

/usr/bin/mysql -h localhost -uroot root_pw -e 'grant all on db_name.* to db_user@"%" identified by awe$ome+password"; flush privileges;'

Written by Mark Needham

September 13th, 2012 at 12:17 am

Posted in Shell Scripting

Tagged with

While waiting for VMs to provision…

without comments

Phil and I spent part of the day provisioning new virtual machines for some applications that we need to deploy which involves running a provisioning script and then opening another terminal and repeatedly trying to ssh into the box until it succeeds.

Eventually we got bored of doing that so we figured out a nice little one liner to use instead:

while :; do ssh; done

The ‘:’ is a bash noop and is defined like so:

null command [colon].

This is the shell equivalent of a “NOP” (no op, a do-nothing operation). It may be considered a synonym for the shell builtin true. The “:” command is itself a Bash builtin, and its exit status is true (0).

In this case it helps us to create an infinite loop which exits once an ssh session is established, meaning that the machine has its ssh daemon running and is ready to roll.

Since we’re using a puppet client/server setup we also want to run something on the puppet master to make sure that the client’s certificate has been signed.

Here we can use the ‘watch‘ command to help us out:

watch "puppet cert list -a | grep new-client-new"

So we’ll see an empty screen until the client has sent a certificate request that’s been picked up by the puppet master and then we’ll see it come up.

As usual if you know any cooler ways to do the same things let me know in the comments!

Written by Mark Needham

September 12th, 2012 at 10:53 pm

Posted in Shell Scripting

Tagged with

Bash Shell: Reusing parts of previous commands

with 3 comments

I’ve paired a few times with my colleague Phil Potter over the last couple of weeks and since he’s a bit of a ninja with bash shortcuts/commands I wanted to record some of the things he’s shown me so I won’t forget them!

Let’s say we’re in the ‘/tmp’ directory and want to create a folder a few levels down but forget to pass the ‘-p’ option to ‘mkdir’:

$ mkdir blah/de/blah
mkdir: cannot create directory `blah/de/blah': No such file or directory

One way of fixing that would be to press the up arrow and navigate along the previous command and put in the ‘-p’ flag but it’s a bit fiddly so instead we can do the following:

$ ^mkdir^mkdir -p
mkdir -p blah/de/blah

The ‘^’ allows us to replace any parts of the previous command and then run it again, so we could actually make that more concise if we wanted to:

$ ^r^r -p
mkdir -p blah/de/blah

Reasonably frequently after we’ve created a folder like this we’ll want to create a file inside it. ‘!$’ comes in handy here as it allows us to refer to the last argument passed to the last command:

$ touch !$/blah.xml
touch blah/de/blah/blah.xml

If we decide to remove that file and want to check it’s been deleted we can run the following:

$ touch blah/de/blah/blah.xml
$ rm blah/de/blah/blah.xml
$ ls -alh !$:h
ls -alh blah/de/blah
total 8.0K
drwxr-xr-x 2 mneedham mneedham 4.0K 2012-07-05 16:26 .
drwxr-xr-x 3 mneedham mneedham 4.0K 2012-07-05 16:16 ..

The ‘:h’ modifier removes the file name and leaves the rest of the file path alone.

We can expand the value of ‘!$’ or any other command by typing ‘Esc’ followed by ‘^’ (Shift 6):

mneedham@ubuntu:/tmp$ mkdir -p blah/de/blah
mneedham@ubuntu:/tmp$ touch !$

Esc + ^

$ mkdir -p blah/de/blah
$ touch blah/de/blah

If we get carried away with modifiers we could also fix that first ‘mkdir’ command by making use of the ‘substitution’ modifier:

$ mkdir blah/de/blah
mkdir: cannot create directory `blah/de/blah': No such file or directory
$ !!:s/r/r -p
mkdir -p blah/de/blah

There’s not really any reason I can think of why you’d want to use that when you can use the initial ‘^’ approach though!

I came across this blog post which explains how to do this type of thing and much more in the bash shell – worth a read!

Written by Mark Needham

July 5th, 2012 at 11:42 pm

Posted in Shell Scripting

Tagged with

Unix: Summing the total time from a log file

with 2 comments

As I mentioned in my last post we’ve been doing some profiling of a data ingestion job and as a result have been putting some logging into our code to try and work out where we need to work on.

We end up with a log file peppered with different statements which looks a bit like the following:

18:50:08.086 [akka:event-driven:dispatcher:global-5] DEBUG - Imported document. /Users/mneedham/foo.xml in: 1298
18:50:09.064 [akka:event-driven:dispatcher:global-1] DEBUG - Imported document. /Users/mneedham/foo2.xml in: 798
18:50:09.712 [akka:event-driven:dispatcher:global-4] DEBUG - Imported document. /Users/mneedham/foo3.xml in: 298
18:50:10.336 [akka:event-driven:dispatcher:global-3] DEBUG - Imported document. /Users/mneedham/foo4.xml in: 898
18:50:10.982 [akka:event-driven:dispatcher:global-1] DEBUG - Imported document. /Users/mneedham/foo5.xml in: 12298

I can never quite tell which column I need to get so end up doing some exploration with awk like this to find out:

$ cat foo.log | awk ' { print $9 }'

Once we’ve worked out the column then we can add them together like this:

$ cat foo.log | awk ' { total+=$9 } END { print total }'

I think that’s much better than trying to determine the total run time in the application and printing it out to the log file.

We can also calculate other stats if we record a log entry for each record:

$ cat foo.log | awk ' { total+=$9; number+=1 } END { print total/number }'
$ cat foo.log | awk 'min=="" || $9 < min {min=$9; minline=$0}; END{ print min}' 

Written by Mark Needham

July 27th, 2011 at 11:02 pm

Posted in Shell Scripting

Tagged with ,

Browsing around the Unix shell more easily

with 14 comments

Following on from my post about getting the pwd to display on the bash prompt all the time I have learnt a couple of other tricks to make the shell experience more productive.

Aliases are the first new concept I came across and several members of my current team and I now have these setup.

We are primarily using them to provide a shortcut command to get to various locations in the file system. For example I have the following ‘work’ alias in my ~/.bash_profile file:

alias work='cd ~/path/to/my/current/project'

I can then go to the bash prompt and type ‘work’ and it navigates straight there. You can put as many different aliases as you want in there, just don’t forget to execute the following command after adding new ones to get them reflected in the current shell:

. ~/.bash_profile

A very simple idea but one that helps save so many keystrokes for me every day.

Another couple of cool commands I recently discovered are pushd and popd

They help provide a stack to store directories on, which I have found particularly useful when browsing between distant directories.

For example suppose I am in the directory ‘/Users/mneedham/Desktop/Blog/’ but I want to go to ‘/Users/mneedham/Projects/Ruby/path/to/some/code’ to take a look at some code.

Before changing to that directory I can execute:

pushd .

This will push the current directory (‘/Users/mneedham/Desktop/Blog/’) onto the stack. Then once I’m done I just need to run:


I’m back to ‘/Users/mneedham/Desktop/Blog/’ with a lot less typing.

Running the following command shows a list of the directories currently on the stack:


I love navigating with the shell so if you’ve get any other useful tips please share them!

Written by Mark Needham

October 15th, 2008 at 10:31 pm

Posted in Shell Scripting

Tagged with

Calling shell script from ruby script

with one comment

Damana and I previously posted about our experiences with different Ruby LDAP solutions.

Having settled on Ruby-LDAP (although having read Ola and Steven’s comments we will now look at ruby-net-ldap) we then needed to put together the setup, installation and teardown into a ruby script file.

A quick bit of Googling revealed that we could use the Kernel.exec method to do this.

For example, you could put the following in a ruby script file and it would execute and show you the current directory listing:

exec "ls"

The problem with using Kernel.exec, which we became aware of after reading Jay’s post, is that we lose control of the current process – i.e. the script will exit after running ‘exec’ and won’t process any other commands that follow it in the file.

Luckily for us there is another method called Kernel.system which allows us to execute a command in a sub shell, and therefore continue processing other commands that follow it.

We were able to use this method for making calls to the make script to install Ruby-LDAP:

@extconf = "ruby extconf.rb"
system @extconf
system "make"
system "make install"

There is one more option we can use if we need to collect the results called %x[…]. We didn’t need to collect the results so we have gone with ‘Kernel.system’ for the time being.

Jay covers the options in more detail on his post for those that need more information than I have presented.

Written by Mark Needham

October 6th, 2008 at 8:12 pm

Posted in Ruby

Tagged with , ,

Show pwd all the time

with 3 comments

Finally back in the world of the shell last week I was constantly typing ‘pwd’ to work out where exactly I was in the file system until my colleague pointed out that you can adjust your settings to get this to show up automatically for you on the left hand side of the prompt.

To do this you need to create or edit your .bash_profile file by entering the following command:

vi ~/.bash_profile

Then add the following line to this file:

export PS1='\u@\H \w\$ '

You should now see something like the following on your command prompt:

mneedham@Macintosh-5.local /users/mneedham/Erlang/playbox$

Another colleague pointed out that the information on the left side is completely configurable. The following entry from the manual pages of bash (Type ‘man bash’ then search for ‘PROMPTING’) show how to do this:

       When executing interactively, bash displays the primary prompt PS1 when it is ready to read a command, and the secondary prompt PS2 when it needs more input to complete a command.  Bash allows these prompt
       strings to be customized by inserting a number of backslash-escaped special characters that are decoded as follows:
              \a     an ASCII bell character (07)
              \d     the date in "Weekday Month Date" format (e.g., "Tue May 26")
                     the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation.  The braces are required
              \e     an ASCII escape character (033)
              \h     the hostname up to the first `.'
              \H     the hostname
              \j     the number of jobs currently managed by the shell
              \l     the basename of the shell's terminal device name
              \n     newline
              \r     carriage return
              \s     the name of the shell, the basename of $0 (the portion following the final slash)
              \t     the current time in 24-hour HH:MM:SS format
              \T     the current time in 12-hour HH:MM:SS format
              \@     the current time in 12-hour am/pm format
              \A     the current time in 24-hour HH:MM format
              \u     the username of the current user
              \v     the version of bash (e.g., 2.00)
              \V     the release of bash, version + patchelvel (e.g., 2.00.0)
              \w     the current working directory
              \W     the basename of the current working directory
              \!     the history number of this command
              \#     the command number of this command
              \$     if the effective UID is 0, a #, otherwise a $
              \nnn   the character corresponding to the octal number nnn
              \\     a backslash
              \[     begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
              \]     end a sequence of non-printing characters

This page has more information on some of the other files that come in useful when shell scripting.

Written by Mark Needham

September 28th, 2008 at 10:50 pm

Posted in Shell Scripting

Tagged with ,