So now you’re a manager

This happened to me a few years ago. I joined a new company as technical lead and within a few days, was ‘promoted’ to the role of manager, while still being the technical lead…. Over the next 3 years I built the team from 6 to nearly 30. Eventually I found myself trying to both remain technical, and be effective in middle management, and had to make a difficult choice.

These days my ’empire’ is a much more manageable 4 and I’ve regained my technical focus. 🙂
A lot of very good technical people have been turned into very bad managers over the years. My goal was (and is) to be at least tolerable. Here are some sources and a couple of observations I found useful along the way.

Get training

It sounds so blindingly obvious, and yet I’ve met many managers who don’t believe there’s anything to learn. If you’re in a small company and there’s no formal training, then you’ll have to do it yourself.
I found the following sources very useful, even if your company offers training.

Behind Closed Doors. Secrets of great management, by Johanna Rothman and Esther Derby

Short and pragmatic. A great starting point. Johanna and Esther also both have great blogs as well.

Manager Tools podcasts

These are more biased towards corporate America, but I’ve found most of their content is perfectly applicable to any company. There is a lot of focus and specific guidance on the basics of good management; One-on-Ones, Feedback and Coaching; as well as many other topics. At this point they’ve done well over 500 casts, so they have collected 30 or so core casts here.

In an agile environment, your responsibilities may not be what you think

In a traditional development team, the manager tends to be the entry point to the team, while in an Agile (Scrum) team, work items flow directly from the product owner to the developers. This means that in an Agile team:
  • The role of line manager is less focused on the work items and more on team well-being, coaching and mentoring, as well as the usual process work that comes with being a manager (expenses, hiring, resource allocation, reports, etc).
  • You can scale a little further in an agile environment since some of the responsibilities of the role are taken by others (primarily the Product Owner and Scrum Master)

Be aware of your role power

You’re not a normal member of the team any more; your words and opinions have extra weight. If you find yourself as both a technical contributor and line manager you’ll need to work hard to clearly separate your roles. It’s very easy for the team to defer to your greatness, even when their ideas are better. In my case I explicitly raised it as a topic with the team.

It’s a journey

Management is a discipline just as much as software development is. It takes years to be an expert programmer in even a single language – don’t expect to be an expert manager overnight.
Expect to make mistakes; lots of them. Unlike software you can’t just fire up the IDE to correct your mistakes. Dealing with peoples feelings and ego’s is a whole new skill set.
Ultimately though, what made it worth while for me was seeing an organisation come to life. That can be very rewarding.
.
Advertisements

Learning TDD

A friend described recently what happened when his some of his team started learning TDD.

They were doing the Roman Numerals kata, where you write a class to convert arabic numerals into their roman equivalents. So for example; an input of ‘7’ gives an output of ‘VII’.
The code to support the first test was easy.

public String arabicToRoman(int input) {
    return "1";
}

So was the second.

public String arabicToRoman(int input) {
    if (input == 1) {
        return "I";
    } else {
        return "II";
    }
}

However by the time they got to the third there was a problem. The code turned into a switch statement. Everyone knew that as the numbers got larger, a better algorithm would be needed, but didn’t know when make the change.

What was missing?

One of the key steps in TDD is to refactor to remove duplication after you get the test working. That’s key, but isn’t always fully appreciated.

  • Duplication refers not just to code blocks but of logic as well. That seems to be the case here
  • The open-closed principle may also help when deciding when to refactor. In this case the switch statement must be modified every time a number is added.

Refactored logic might look like this. An aggressive refactorer might even have done this at the second step:

public String arabicToRoman(int arabic) {
    String roman = "";
    for (;arabic > 0; arabic--) {
        roman = roman.concat("I");
    }
    return roman;
}

Of course the algorithm will continue to evolve as we add support for more numbers. And the tests need to be refactored as we go, just as aggressively as the production code.

So what?

To some, this might seem obvious and maybe simplistic. But its easy to miss subtleties when trying out the mechanical aspects of a new practice such as TDD – I’m sure I did something similar. Exploring a few blind alleys is a natural part of learning.

.

Technical debt and legacy code

Every job I’ve ever had has involved dealing with some amount of technical debt. The amount has varied, but its always there, even in modern codebases.

So what is technical debt?

There are loads of definitions – I think of it as code that is difficult or dangerous to change or extend. Technical debt is often associated with legacy code, which Michael Feathers defines as code without automated tests, though i’ve seen legacy code that was straightforward to change, and code with automated tests that wasn’t.

There’s a surprising amount of code with tests thats difficult to change or extend. Often dependencies have not been isolated well, so tests have extensive scaffolding. Another reason is overly complex or poor designs – if I find i’m changing every class in a subsystem for a simple addition, there is often a design problem.

The issue for me is not that technical debt exists (we’ve all written some – or at least I have), but how to address it. I think its as much a people problem as a technical one and there are many reasons for it

Managers discourage changes to existing code

  • Some managers don’t understand why existing code should change to accommodate new features. They ask why couldn’t you get it right first time? Ironically the more successful a product, the more likely it is to change and grow, often in unexpected ways.
  • Managers who have been developers in the past, know the dangers of making changes to code that has no tests, and shy away from making changes for that reason.
  • Others have been burnt by teams that got bogged down performing epic refactoring’s or rewrites
  • Finally, there is the ever present pressure of the roadmap, which managers are typically far more exposed to than any individual developer. Under that pressure, its tempting for even the best manager to encourage shortcuts.
I’m maybe being unfair to managers here. Much of the above could be applied to product owners, team leads, or anyone with else with a say in what’s being built.

We want to work on new code

Preferably in the hottest language using a cool new framework. It combines with appeal of the new with a great bullet point for the resume. And of course, the newer frameworks and languages can be more productive than older ones. Many (perhaps most) developers would rather attempt a rewrite than refactor legacy code. That can be the right thing to do; often it isn’t.
  • Old code still needs maintaining while the rewrite is happening
  • In agile teams where backlogs can change rapidly its easy for large rewrites to stall or be abandoned as priorities change.
  • If you are dealing with a codebase that has a lot of duplication, it may be better to reduce the duplication first, rather than introduce yet another mechanism for doing something
  • If the code is reasonably modern and has decent test coverage, refactoring is often by far the safer route – though some developers still argue for a rewrite

We don’t recognise that there is a problem

Ignorance or apathy? I’m not sure.

We’re scared to touch it

After being burnt a couple of times by introducing subtle bugs its easy to see why. Which leads to…

We don’t have the skills

There are many developers with good skills these days in TDD/BDD (or at least writing tests concurrently with the code). Fewer know the techniques for dealing with legacy code. A rewrite often feels easier.

What to do about it?

Change attitudes

I like Robert Martins boy scout rule: ‘Leave the campground cleaner than you found it’, which encourages a continual improvement mindset. Improve the code base in a minor way every time you make a change, even if its just clarifying a name or removing an unused variable.
Changing managerial attitudes can be a little trickier. Modern code is meant to be malleable, and not everyone gets that. Incremental refactoring rather than big epics can help.

Skills and Training

There are some great books, primarily ‘Working Effectively with Legacy Code’, which give practical techniques and insights.
A simple example:
Imagine we have a class that we cannot easily put under test – maybe it references many other classes or a socket or database connection, but we need to add or change some behaviour. Here are a few techniques to consider:
  • If the method in question does not reference any member variables we can make it static and thereby write tests for it without having to instantiate the whole class. Ugly but effective.
  • We can subclass and in the derived class override selected methods to effectively null out dependencies
  • We can add setter methods to override dependencies.
  • We can make private methods public to get access to them (!)
  • We can link to mock libraries
  • When adding behaviour we could create a small object with the new behaviour using TDD and then just call it from the legacy class. In the short term this can be pretty ugly – maybe the new class has only a single method; but over time we could move behaviour as appropriate from the legacy class to the new one.

These techniques are highly incremental and can make the code feel worse in the short term. Maybe thats why they are used as much as they could be.

Another fun exercise is to try out the excellent ‘Gilded Rose’ kata, which presents a horribly messed up method and asks you to refactor it.

Practice

I learn a lot by doing dry runs. Check out the code and try out a few refactorings. Then throw that code away and try it for real. Don’t be surprised if your real refactoring works out differently – the point of the dry run is to gain confidence and context.

Personally I find improving a legacy codebase can be a rewarding activity in itself. Good luck with yours!

Favourite books – Agile and Lean

I first discovered agile though a copy of Kent Beck’s Extreme Programming explained. I spent the next few years experimenting with bits of it, but the concept as a whole was considered a bit too, em, extreme for use in real organisations. Fast forward a few years and the world has changed. Agile is mainstream, though the number of companies that do it well is more limited (I will post about that sometime). I’ve helped introduce it to a few companies now and don’t think I could go back.

Here’s a few of my favourites, there are literally hundreds of agile books out there, but this lot should cover the basics. The following books focus more on the the process side than the technical practices. That’s a subject for another post.

Anything by Henrik Kniberg

Seriously, these are by far the best books i’ve found for introducing people to agile and lean. They are short, easy to read and full of real world examples. Best of all he doesn’t have a huge chip on his shoulder over which is best, unlike the originators of the various methodologies. Read these before going to the primary sources.

As well as this lot, he’s done some nice presentations on team organisation and agile metrics. Well worth a look on his blog.

Succeeding with Agile (Cohn)

A great book for taking you beyond the primary sources and putting agile into practice in the real world. Useful chapters on transitioning to agile and how traditional roles map (or don’t) to agile ones.

Scrumban (Ladas)

What could be better than Scrum or Kanban? Both. The author has a bias to Kanban and his tone annoys me, but this provides an interesting insight in how to gradually transition from one to the other.

Agile Retrospectives (Derby, Larsen)

A handy book describing many of the exercises and games you can use to generate insights and solutions during retrospectives and prevent them becoming stale, which is a real danger if you run them regularly.

Primary Sources

These are the original books for Scrum, XP and Kanban. They are all opinionated, somewhat dismissive of other approaches, revolutionary in their time, and perhaps overly prescriptive. The books on XP and Scrum in particular suffered from this. I think there was good reason for this at the time as its too easy to pick a couple of practices and say ‘we’re now doing Scrum’, and thereby miss the whole point of the exercise. As a team gains experience with agile they they will naturally find their own way. The mindset matters more than the specific practices.

There are also other lesser known agile methodologies, such as Crystal and DSDM, should you wish to look deeper. Given the sheer number of books on agile, you could be reading for years…

Ebooks, an evolving landscape

I read a lot. Amazon made a fortune out of me over the years as the most convenient place to buy technical books; but eventually I hit a tipping point. Physical books, particularly technically ones, take up a lot of space, and I literally ran out of room. Luckily technology has finally caught up, and I’ve bought into the ebook (r)evolution over the last year or so.

I decided to stick to the open ePub format, which meant leaving Amazon behind as they are the only major vendor not to support ePub. Unfortunately Amazon makes things very easy, if you decide to go elsewhere you may have to put in a little more effort.

I started with iBooks on my iPad, a great application. I bought a couple of books in the iBookStore. And all was well… until I tried to transfer them to a Kobo. Both use ePub, but Apple’s DRM means I can only view the books on the iPad or iPhone. No-one has yet reliably cracked Apples DRM, so I cut my losses, and don’t buy there anymore.

These days I buy directly from publishers where I can. The upside is access to beta versions pf some books and occasional chances to pick up eBook versions of physical books I bought long ago, sometimes at a huge discount. The downside is I now have at least 6 separate accounts to manage. You may find it worth your while to buy direct from the publisher even if you use Amazon, as they all support Amazons Mobi format as well as ePub.

For general books I tend to go to Kobo or Diesel. Another two accounts to manage… 😦

For managing books I use Calibre. It’s free, open source, works with just about all e-readers, and allows converting between most formats.

Most of the technical publishers books are either DRM free (OReilly) or watermarked (PragProg, Informit) which is great. Most publishers using DRM tend to use Adobe which is easy to crack if you are so inclined. You will probably have to install Adobe Editions so the procedure is download to Adobe Editions, and either use that as your library manager or fish the books out of there, remove the DRM, and pull them into Calibre.

An unexpected side effect is that I now buy even more books than before. However doing the above does require effort. No one else has yet replicated the magical mix of good hardware, ease of use and sheer breadth of content that Amazon has; though plenty are trying.

http://calibre-ebook.com

Favourite books – better programming

I didn’t do software engineering at college, so for a long time I worried about what I didn’t know, which translated into a continuing quest for knowledge. This and a few followup posts will list a few of my favourite books and other sources, with some brief explanations of why I think they are important. Your opinions may vary. 🙂

So here goes…

Programming

These are books about the technicalities of programming; about being better programmers. We can get away with minimal designs and documentation, and many products have shipped with little of either, but there is always code, which has to be maintained and extended. In fact the more successful a product is, the more its likely to change.
All of the following cover (to greater or lesser extents) style, structure, debugging, documentation and low level design. The principles apply across many languages and environments, though the examples typically use the most popular language of the day.

The practice of Programming (Kernighan and Pike)

Brian Kernighan is famous as the co author of ‘The C programming language’, another of my favourite books, and his clear writing style is put to good use here. The tag line to this book is: Simplicity, Clarity, Generality. Written in 1999, its examples are mostly in C, C++ and Java, though there is a C and Unix bias, not surprising from the people so heavily involved with both. And it’s short, only 288 pages.

Code Complete (McConnell)

A heavyweight tome. Very comprehensive and a little verbose. McConnell backs up his advice with plenty of citations from industry studies. This one tops many peoples favourite technical book lists. It’s best to pick up the second edition, which has more of an OO focus than the first. I’m hoping for a third edition one day incorporating more agile and lean practises.

Clean code (Martin)

A book for the agile era, so an emphasis on things like unit testing and code as documentation. The cartoon about the only true measure of code quality being ‘WTFs per minute’ is a classic, and sets the tone. Bob Martin is very opinionated, so the book is a bit of a polemic, but the content (especially the early chapters) is excellent.
This book is required reading for all developers in our company.
There are mny books that focus on a single practice (such as TDD) or best use of a given language, but surprisingly few books like the three above, that focus on the nuts and bolts of general programming. Hopefully there will soon be one incorporating functional programming elements…

NoSQL introduction

I’ve been researching data persistence recently, and wrote a few notes to try and organise my thoughts. Here’s a summary that might be useful, along with a list of useful sources at the end. I’ve missed out a huge amount (CAP theorem, map-reduce, etc), so this just scratches the surface.

Firstly, what is NoSQL?

NoSQL as a term was coined for a conference of next generation databases in 2009. However it’s not a particularly useful name. My favourite definition comes from the Fowler/Sadalage book ‘NoSQL Distilled’ where they define it as:
“an ill defined set of mostly open-source databases, mostly developed in the early 21st century, and mostly not using SQL”. Hmm…

Some typical characteristics of NoSQL databases are:

  • Most of them run well on clusters (except for graph databases) and are designed for very high scale
  • They don’t use SQL, though some are getting close
  • They don’t have a schema, so you can add new fields without having to define schema changes. This doesn’t mean that there is no schema. There is always a schema, it’s just now in code instead of the database. So engineering best practice around schema migrations still applies…
  • They do have transactions but not in the SQL sense. For example a single write to a document database is a transaction which could persist one or many entities, depending upon the design of the document.

There are four main types of NoSQL database:

Key Value Store

Examples: Riak
I think of these as distributed Dictionaries or HashMaps. You store and retrieve values by key. The values are opaque to the database and joins across multiple values are typically not allowed. As a result they are very fast.
A classic use case is storing session data. They are not recommended when you have relationships between values.

Document Database

Examples: MongoDB, CouchDB
A shorthand way to think of these is a bit like key value stores except that the values are not opaque. This opens up a world of possibilities. You can store complex structures and query them by something other than the primary key. So for example if you store addresses, keyed by ID, you could also query them by town or zip/post code.
Joins across documents are either not allowed or expensive. However a document can be arbitrarily complex, containing many nested elements, for example, a customer and their recent orders could be stored as a single document. By careful grouping of related information into the same document, lookups can be fast lookups in typical business use cases, as all related information can be returned by a single query.
Document databases are being used for object persistence. Both MongoDB and CouchDB store documents as json. They are also great for logging, where different log messages may have different contents.

Column Database

Examples: Cassandra, HBase
In a column database a column consists of a key value pair. Collections of columns that are accessed together are called Column Families. Rows within a column family may have many columns, accessed together using a row key. Different rows may have different columns.
If that didn’t make much sense, another way to think of a Column family is as a Table, each of whose rows can have different columns.
It’s also possible to have super-columns, where the value of the key value pair is a collection of columns.
Logging is a great use case for Column databases and they are used heavily in content management systems.
Column databases are sensitive to the query used. Changing the queries may require changing the column family design.

Graph Database

Examples: NeoDB, OrientDB
These are useful when you need to traverse relationships between entities. They allow both entities and relationships to be stored as first class elements, and both entities and relationships can have properties. Relationships can also have directionality. So for example given two people ‘Bob’ and ‘Sue’ with a ‘Likes’ relationship between them, it could be one way, so that ‘Bob likes Sue’ is true but ‘Sue likes Bob’ is not.
Graph databases make it easy to construct complex queries involving many entity types. In SQL those queries would involve many joins across different tables.

Should you change?

As always, it depends. I attended an interesting talk by Lisa Phillips from Twitter recently where she said they have loads of cool NoSQL stuff, some of which they invented themselves, but still use MySQL very extensively, to the extent that the default position for all new projects is ‘use MySQL unless it can’t do what you want’.

It also looks like the two worlds are converging a little. For example recent additions to MySQL include a handler socket interface that bypasses the SQL layer, and MariaDB has also added dynamic columns. It’s also possible to use MySQL as a front end to InfiniDB, and the MariaDB team has experimented with using Cassandra as a storage engine.

More information:

NoSQL Distilled

An excellent introduction to the subject.

Seven Databases in seven weeks

Practical dives into seven real databases, illustrating the strengths of each. Covers: Postgres, Riak, HBase, MongoDB, CouchDB, NeoJ, Redis

Software Engineering Radio:

I also attended a surprisingly good conference in Oxford recently on Database technologies for developers, called All your Base, which I hope they run again next year.