Mark Needham

A couple of years ago I used to write a blog post reflecting on what I’d worked on in the preceding year and what I’d learned and having read 2011 reviews by a couple of other people I thought I’d have a go.

Am I actually learning anything?

A thought I had many times in 2011 was ‘am I actually learning anything?‘ as, although I was working with languages that I hadn’t used professionally before, the applications that we I worked on were very similar to ones that I’ve worked on previously.

Often I’d work on something and know exactly how it should be designed and where we could go wrong since I’d done the same thing several times before and the challenge of not knowing what to do had disappeared somewhat.

Now and then…

I certainly failed to learn one thing a day as I suggested in a blog post a couple of years ago although eventually I managed to learn a bit about node.js and clojure by building some toy applications with my colleague Uday.

We decided to rewrite part of our Scala application in clojure in our own time to see what it’d look like which provided us with an interesting insight into what it’d be like to build a system for the second time when you know exactly what to do.

I also completed ml-class which was fun as it was the type of programming that I’ve never done before. Obviously I’m still a novice at the whole machine learning thing but it’s given me an idea of the sorts of things you can do.

Learning is doing

From February until April I was in Bangalore working as a trainer/coach for one of the ThoughtWorks University batches where we tried as much as possible to reduce the amount of ‘teaching’ done.

Sumeet has previously written about the new style of ThoughtWorks University which is more focused on people working on a real project than sitting in workshops and we tried to take this even further.

Previous groups had spent about 2 weeks doing workshop style sessions and then 4 weeks working on a project but we got it to the point where we spent just over a week in workshops and the rest working on the project.

In general I think it worked reasonably well and the skill level of the group seemed reasonably high by the end. We were lucky that there were only 13 people in the group – it would be interesting to see how our approach would scale.

I’ve also noticed this last year that when I’m learning something new it’s not enough to just do toy exercises anymore, I actually have to build something to retain interest.

During the Christmas holidays I decided to try and build a Flipboard style application for my Android phone so I can (yet again) capture the links that people post on twitter.

Actually having a real problem to solve has made me much more engaged than following a tutorial or hello world demo would have done.

Remembering the value of blogging

My rate of posting on here has decreased a lot over the last year which I think is partly down to the fact that I’ve written about a lot of the stuff I see on projects before but also because I started filtering what I thought was interesting enough to write about.

In hindsight the latter approach doesn’t necessarily make sense – the most read posts on this blog are the ones which I thought were the most pointless when I wrote them.

I got stuck in the mindsight that I wasn’t actually learning anything by writing blog posts, which has been proved wrong multiple times both in terms of what I learn in writing the post and from what I learn from people’s comments.

Expressing opinions in big groups/public

I spent 10 months in late 2010/early 2011 working in India and one of the most interesting things I remember observing was that people seemed very reluctant to express their opinion in big groups.

I thought that was something specific to India but on coming back to the UK I’ve noticed the same thing here as well which means we need to adjust our approach in retrospectives if we want everyone to participate.

I also learnt that expressing strong opinions in public in isn’t necessarily the most effective way of making change happen. I probably should have learnt this already but it became increasingly evident how ineffective this approach was in 2011.

Going at my own pace

A couple of years ago I was advised by a couple of colleagues that the way to get to the ‘next level’ was to become more knowledgeable about the overall architectural design of systems but at the time I wasn’t that interested in that.

It’s only more recently that I’ve found it interesting to read about different architectures on High Scalability or Systems We Make.

Another interesting way for me to learn in this area is to try and understand the architectures used in other ThoughtWorks projects that I didn’t work on and see how they compare to the ones I’ve worked on.

I generally can’t force myself to be interested in something if I’m not but once I am interested then I want to learn every detail about it so it’s better to wait until I become interested naturally.

The next thing which I’m sure I’ll eventually become interested in is tech leading a team which several of my peers (in terms of years of experience) are doing now or have been doing for a year or two. Right now though I want to focus on coding!

Overall…

I’m not sure 2011 was a year where I learned as much as I did in previous years – the learning did seem to taper off a bit which in a way is inevitable unless you completely change your role/the types of things you’re building.

In 2012 I plan to keep learning about Android development and I’m going to be doing algo-class to try and get better at another aspect of programming which I’m not very good at right now.

I want to be able to get the tweets from my timeline into my app which means I need to authorise the app with Twitter using OAuth.

The last time I tried to authenticate using OAuth a couple of years ago was a bit of a failure but luckily this time Honza Pokorny has written a blog post explaining what to do.

I had to adjust the code a little bit from what’s written on his post so I thought I’d document what I’ve done.

We’re using the signpost library for which we need to download the following two jars and put them into the app’s ‘libs’ directory.

wget http://oauth-signpost.googlecode.com/files/signpost-commonshttp4-1.2.1.1.jar
wget http://oauth-signpost.googlecode.com/files/signpost-core-1.2.1.1.jar

I created a button that had to be clicked to fire the first step of OAuth authentication with twitter. The code looks like this:

public class MyActivity extends Activity {
  private String CALLBACKURL = "app://twitter";
  private String consumerKey = "TwitterConsumerKey";
  private String consumerSecret = "TwitterConsumerSecret";
 
  private OAuthProvider httpOauthprovider = new DefaultOAuthProvider("https://api.twitter.com/oauth/request_token", "https://api.twitter.com/oauth/access_token", "https://api.twitter.com/oauth/authorize");
  private CommonsHttpOAuthConsumer httpOauthConsumer = new CommonsHttpOAuthConsumer(consumerKey, consumerSecret);
 
  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
 
    ImageButton oauth = (ImageButton) findViewById(R.id.oauth_button);
    oauth.setOnClickListener(new View.OnClickListener() {
      public void onClick(View v) {
        try {
          String authUrl = httpOauthprovider.retrieveRequestToken(httpOauthConsumer, CALLBACKURL);
          Intent intent = new Intent(Intent.ACTION_VIEW, Uri.parse(authUrl));
          v.getContext().startActivity(intent);
        } catch (Exception e) {
          Log.w("oauth fail", e);
          Toast.makeText(v.getContext(), e.getMessage(), Toast.LENGTH_LONG).show();
        }
      }
    });
  }
}

The CALLBACK URL is called by Twitter when the user authorises the application (and therefore the request token). Usually it would be a HTTP URL but in this case we need to define a special URL which gets handled by our application.

The consumerKey and consumerSecret are values assigned by Twitter for the application.

The definition of the callback URL in the manifest file looks like this:

<activity android:name="MyActivity" android:label="@string/app_name" android:launchMode="singleInstance">
  ...
  <intent-filter>
    <action android:name="android.intent.action.VIEW"/>
    <category android:name="android.intent.category.DEFAULT"/>
    <category android:name="android.intent.category.BROWSABLE"/>
    <data android:scheme="app" android:host="twitter" />
  </intent-filter>
</activity>

We can change app://twitter to be anything we want but it needs to match what’s defined in the data element in our manifest file.

I found that I needed to define Callback URL in my application’s settings on Twitter otherwise I ended up getting this error:

oauth.signpost.exception.OAuthCommunicationException: Communication with the service provider failed: https://api.twitter.com/oauth/request_token
        at oauth.signpost.AbstractOAuthProvider.retrieveToken(AbstractOAuthProvider.java:214)
        at oauth.signpost.AbstractOAuthProvider.retrieveRequestToken(AbstractOAuthProvider.java:69)
...
 Caused by: java.io.FileNotFoundException: https://api.twitter.com/oauth/request_token
        at org.apache.harmony.luni.internal.net.www.protocol.http.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:521)
        at org.apache.harmony.luni.internal.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:258)
        at oauth.signpost.basic.HttpURLConnectionResponseAdapter.getContent(HttpURLConnectionResponseAdapter.java:18)
        at oauth.signpost.AbstractOAuthProvider.handleUnexpectedResponse(AbstractOAuthProvider.java:228)
        at oauth.signpost.AbstractOAuthProvider.retrieveToken(AbstractOAuthProvider.java:189)

I ended up just setting my callback URL on Twitter to the URL of this blog. It doesn’t seem to matter what you put the URL as since it’s going to be overridden by our callback URL anyway but it does need to be set.

The callback then gets handled by the following code in MyActivity…

public class MyActivity extends Activity {
  ...
 
  @Override
  protected void onNewIntent(Intent intent) {
    super.onNewIntent(intent);
 
    Log.w("redirect-to-app", "going to save the key and secret");
 
    Uri uri = intent.getData();
    if (uri != null && uri.toString().startsWith(CALLBACKURL)) {
 
        String verifier = uri.getQueryParameter(oauth.signpost.OAuth.OAUTH_VERIFIER);
 
        try {
            // this will populate token and token_secret in consumer
 
            httpOauthprovider.retrieveAccessToken(httpOauthConsumer, verifier);
            String userKey = httpOauthConsumer.getToken();
            String userSecret = httpOauthConsumer.getTokenSecret();
 
            // Save user_key and user_secret in user preferences and return
            SharedPreferences settings = getBaseContext().getSharedPreferences("your_app_prefs", 0);
            SharedPreferences.Editor editor = settings.edit();
            editor.putString("user_key", userKey);
            editor.putString("user_secret", userSecret);
            editor.commit();
 
        } catch (Exception e) {
 
        }
    } else {
        // Do something if the callback comes from elsewhere
    }
  }
}

…which makes another call to Twitter to get the user’s key and secret (the access token) which it then stored in shared preferences so we can use it in future without having to re-authenticate with Twitter.

We can then query Twitter like so:

HttpGet get = new HttpGet("http://api.twitter.com/1/statuses/home_timeline.json");
HttpParams params = new BasicHttpParams();
HttpProtocolParams.setUseExpectContinue(params, false);
get.setParams(params);
 
try {
  SharedPreferences settings = getContext().getSharedPreferences("your_app_prefs", 0);
  String userKey = settings.getString("user_key", "");
  String userSecret = settings.getString("user_secret", "");
 
  httpOauthConsumer.setTokenWithSecret(userKey, userSecret);
  httpOauthConsumer.sign(get);
 
  DefaultHttpClient client = new DefaultHttpClient();
  String response = client.execute(get, new BasicResponseHandler());
  JSONArray array = new JSONArray(response);
} catch (Exception e) { 
  // handle this somehow
}

Here we retrieve the user’s key and secret which we saved on the previous step and then set them on our OAuth consumer which we use to sign our request to Twitter.

There is a nice explanation of how OAuth works about half way down this StackOverFlow post, Eran Hammer-Lahav has a pretty good “beginner’s guide to OAuth” on his blog and the OAuth spec is surprisingly readable as well.

In the Android application that I’ve been playing around with I wrote a service which consumes the Twitter streaming API which I trigger from the app’s main activity like so:

public class MyActivity extends Activity {
    ...
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
 
        Intent intent = new Intent(this, TweetService.class);
        startService(intent);
        ...
    }
}

Where TweetService is defined roughly like this:

public class TweetService extends IntentService {
    @Override
    protected void onHandleIntent(Intent intent) {
      // Twitter streaming API stuff goes here
    }
}

Unfortunately when I tried to deploy the app the service wasn’t starting and I got this message in the log:

01-01 03:10:31.758: WARN/ActivityManager(106): Unable to start service Intent { cmp=com.example/.TweetService }: not found

What I hadn’t realised is that the service needs to be specified in the AndroidManifest.xml file but not inside the activity definition:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example" android:versionCode="1" android:versionName="1.0">
    <application android:label="@string/app_name">
        <activity android:name="MyActivity" android:label="@string/app_name" android:launchMode="singleInstance">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
        <service android:name="TweetService"></service>
    </application>
    <uses-permission android:name="android.permission.INTERNET"/>
</manifest>

After adding the service definition it works fine.

I have a bit of Java code for working out the final destination of a URL assuming that there might be one redirect which looks like this:

private String resolveUrl(String url) {
  try {
    HttpURLConnection con = (HttpURLConnection) (new URL(url).openConnection());
    con.setInstanceFollowRedirects(false);
    con.connect();
    int responseCode = con.getResponseCode();
 
    if (String.valueOf(responseCode).startsWith("3")) {
      return con.getHeaderField("Location");
    }
  } catch (IOException e) {
    return url;
  }
 
  return url;
}

I need to cast to HttpURLConnection on the first line so that I can make the call to setInstanceFollowRedirects which isn’t available on URLConnection.

I wanted to write some similar code in Clojure and my first thought was that I needed to work out how to do the cast, which I didn’t know how to do.

I then remembered that Clojure is actually dynamically typed so there isn’t any need – as long as the object has the method that we want to call on it everything will be fine.

In this case we end up with the following code:

(defn resolve-url [url]
  (let [con (.. (new URL url) openConnection)]
    (doall
     (.setInstanceFollowRedirects con false)
     (.connect con))
    (if (.startsWith (str (.getResponseCode con)) "3")
      (.getHeaderField con "Location")
      url)))

Which can be simplified to this:

(defn resolve-url [url]
  (let [con (doto (.. (new URL url) openConnection)
                  (.setInstanceFollowRedirects false)
                  (.connect))]
  (if (.startsWith (str (.getResponseCode con)) "3")
    (.getHeaderField con "Location")
    url)))

While I’ve been learning how to write an android application there’s been plenty of opportunities for me to go off shaving yaks, it’s pretty much Yakville Central.

Typically I’d end up spending hours trying to work out some obscure thing which I didn’t really need to know so I wanted to try and avoid that this time.

I started keeping a track of the ‘yak stack’ which I was currently following and mentally noting exactly where I was up to.

An example of a yak stack I kept while trying to authorise a user of the app with Twitter using OAuth is shown in the photo on the right hand side.

It ended up looking like this:

• Get the home timeline working
  • OAuth blowing up
    • Not actually capturing redirect back to app
      • Launch mode in Android manifest

Once I realise I’m heading down the stack I’ve been giving myself one pomodoro to try and dig myself one level up.

If I still haven’t managed to solve the problem I might keep going for one more pomodoro or just find another way around the problem.

I’m sure I’ll come across problems where I need to spend more than an hour trying to solve it but for now it’s working ok as a rule of thumb.

It’s definitely fun chasing yaks but I get to the end of the day and haven’t really achieved anything which isn’t fun.

A few weeks ago Chris Matts wrote an interesting blog post ‘the language of risk‘ in which he describes an approach he used to explain the processes his team uses to an auditor.

Why did the auditor like what I said?

Because I explained everything we did in terms of risk. When they asked for a “process”, I explained the risk the process was meant to address. I then explained how our different process addressed the risk more effectively.

This seems like a pretty cool idea to me and it got me thinking of the different ‘processes’ we’ve used in teams I’ve worked on and what risks they might be addressing:

• Pair Programming
  • Becoming dependent on one person with respect to knowledge of part of the code base.
  • Having someone new working on an area of the code that they don’t know well and making a mistake.
• Retrospective
  • Making the same mistakes repeatedly/working in a way that (indirectly) wastes money.
• Story Kick Off
  • Building the wrong thing
  • Solving the business problem in an inefficient way
  • Building something which is very difficult to test
• Stand Up
  • Someone getting stuck on something which someone else in the group might be able to help with.
  • People going down rabbit holes and getting stuck on things that don’t really matter
• Show Case
  • Building the wrong thing for too long
• Automated testing
  • The application regresses as new functionality is added
  • Humans make mistakes when manually going through scenarios

That’s just a first attempt at this, I’m sure others could come up with something better!

In coming up with the list I’ve been working from a process which I’ve seen used and trying to work out what risk that might be addressing.

Chris seems to look at risks/processes the other way around to i.e. we think about what risks we need to address and then work out whether we need a process to address it and if so which one.

Taking that approach would help to explain why some teams don’t necessarily need a lot of process – the risks might be catered for in different ways or maybe they just don’t exist in specific contexts.

For example a lot of risks around communication go away if the product owner and the team are sitting in the same physical location and can easily just turn and talk to each other if they have any questions.

Even with this new way of looking at risks/process I still think it’s useful to keep checking whether or not a process is still necessary because as our team/product changes the risks we face probably do as well.

One thing I wanted to do in the little application I’m working on was send data to other apps on my phone using the ‘share via’ dialog which I’ve seen used on the Twitter app.

In this case I wanted to send a link and its title to twitter and came across a StackOverflow post which explained how to do so.

To keep it simple I added a button to the view and then shared the data via the on click event on that button:

Button button = createButton();
button.setOnClickListener(new View.OnClickListener() {
  public void onClick(View v) {
    Intent shareIntent = new Intent(android.content.Intent.ACTION_SEND);
    shareIntent.setType("text/plain");
    shareIntent.putExtra(android.content.Intent.EXTRA_TEXT, "Stupid Scanner tricks - http://weblogs.java.net/blog/pat/archive/2004/10/stupid_scanner_1.html");
    v.getContext().startActivity(Intent.createChooser(shareIntent, "Share via"));
  }
});

In this case if I choose the Twitter app from the drop down list that appears it will create a new tweet with the value of ‘android.content.Intent.EXTRA_TEXT’ as its body.

If we needed to pass a subject to the other app then we could set that using ‘android.content.Intent.EXTRA_SUBJECT’ but in this case it’s unnecessary.

From what I understand so far only apps which can handle the ‘text/plain’ format will show up in the drop down list but that seems to be pretty much every app on my phone.

This works reasonably well but I wanted to see if it was possible to share directly with the Twitter app rather than having to choose from a selection of apps.

Via a combination of blog posts and StackOverflow questions I came across the following solution for posting directly to the Twitter app:

Button button = createButton();
button.setOnClickListener(new View.OnClickListener() {
  public void onClick(View v) {
    Intent shareIntent = new Intent(android.content.Intent.ACTION_SEND);
    shareIntent.setType("text/plain");
    shareIntent.putExtra(android.content.Intent.EXTRA_SUBJECT, "Some text");
    shareIntent.putExtra(android.content.Intent.EXTRA_TEXT, "Stupid Scanner tricks - http://weblogs.java.net/blog/pat/archive/2004/10/stupid_scanner_1.html");
 
    final PackageManager pm = v.getContext().getPackageManager();
    final List<ResolveInfo> activityList = pm.queryIntentActivities(shareIntent, 0);
    for (final ResolveInfo app : activityList) {
      if ("com.twitter.android.PostActivity".equals(app.activityInfo.name)) {
        final ActivityInfo activity = app.activityInfo;
        final ComponentName name = new ComponentName(activity.applicationInfo.packageName, activity.name);
        shareIntent.addCategory(Intent.CATEGORY_LAUNCHER);
        shareIntent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK | Intent.FLAG_ACTIVITY_RESET_TASK_IF_NEEDED);
        shareIntent.setComponent(name);
        v.getContext().startActivity(shareIntent);
        break;
      }
    }
  }
}):;

It does depend on the Twitter app being installed on the phone to work but since the app is just for me I think it’s ok for the moment.

In the last week or so before Christmas I got the chance to spend some time pairing with my colleague Alex Harin while trying to understand how an existing application which we were investigating was written.

We knew from watching a demo of the application that the user was able to send some processing off to be done in the background and that they would be emailed once that had happened.

Our starting point was therefore to work backwards from the labels on the UI and finding which code got executed when the user submitted the task.

My initial approach was to find the entry point and then follow the method calls line by line, slowly building my knowledge of how the application actually worked.

Alex used a much quicker approach whereby he thought about how the code would be designed and then looked for the bit of code which proved his belief.

In this case we knew that the application had a 2 tier architecture and that it didn’t use any middleware which meant that it would more than likely be using the database as a queue to store the tasks to be processed.

Another colleague and I were then able to make use of this approach when looking at another piece of code.

It was being used to do pricing and we knew that there were different algorithms depending on which country you were in, which meant that we needed to look for anything country related to answer our questions.

Effectively we’re looking at the code with a hypothesis in hand and then trying to see whether or not what we see proves or disproves that hypotheses.

I need to practice a bit more but it seems to let you navigate code much more quickly and makes you much less likely to dive down a rabbit hole.

In my continued attempts to learn how to write an Android application I came across a problem with character encoding when trying to load some text into a WebView.

I was initially trying to write the text to the WebView like this:

WebView webview = new WebView(collection.getContext());
webview.loadData(textWithQuotesIn, "text/html", "UTF-8");

But ended up with the output in the picture on the left hand side. I tried playing around with the encoding and debugged the application all the way through until it hit the WebView but there didn’t seem to be any problem with the text.

I eventually came across a post on StackOverflow where mice suggested using one of the other methods available for writing to a WebView.

I changed my code to read like this:

WebView webview = new WebView(collection.getContext());
webview.loadDataWithBaseURL(url, textWithQuotesIn, "text/html", "UTF-8", url);

And now the single quotes are rendering correctly as can be seen on the image on the right.

I had a quick look at the Android source code to see if there was any obvious reason why one of the methods would work and the other wouldn’t but I couldn’t see anything.

Perhaps I’m doing something wrong with my call to ‘loadData’ and that’s why it’s not rendering the character set correctly. If that’s the case please let me know.

I’ve been playing around a little bit with goose – a HTML content/article extractor – originally in Java but later in clojure where I needed to work out how to include goose and all its dependencies via Leiningen.

goose isn’t included in a Maven repository so I needed to create a local repository, something which I’ve got stuck on in the past.

Luckily Paul Gross has written a cool blog post explaining how his team got past this problem.

Following the instructions from Paul’s post this is how I got goose playing nicely with clojure:

Inside my clojure project:

/Users/mneedham/github/android/text-extraction $ mkdir maven_repository

I then ran the following command from where I had goose checked out on my machine:

mvn install:install-file -Dfile=target/goose-2.1.6.jar -DartifactId=goose -Dversion=2.1.6 -DgroupId=goose -Dpackaging=jar -DlocalRepositoryPath=/Users/mneedham/github/android/text-extraction/maven_repository -DpomFile=pom.xml

I added the repository and goose dependency to my project.clj file which now looks like this:

(defproject textextraction "0.1.0"
  :description "Extract text from urls"
  :dependencies [[org.clojure/clojure "1.2.0"],
		 [org.clojure/clojure-contrib "1.2.0"],
		 [ring/ring-jetty-adapter "0.3.11"],
         [compojure "0.6.4"]
         [goose "2.1.6"]]
  :dev-dependencies [[swank-clojure "1.2.1"]]
  :repositories {"local" ~(str (.toURI (java.io.File. "maven_repository")))}
  :main textextraction.main)

I then run:

/Users/mneedham/github/android/text-extraction $ lein run

And goose and all its dependencies are included in the ‘lib’ directory.