Node.js can’t handle parallel npm install

Setting up CI for a node.js project, I discovered one of life’s little ironies.  Node.js bills itself as a programming framework with a “non-blocking I/O model”, but npm — the node.js package manager — which installs libraries needed by a node app relies on a single cache file structure (typically ~/.npm) which can cause problems if you try to run ‘npm install -d’ more than one at a time.

Technically this isn’t an issue with parallelism in node.js — since it’s the package manager, not the execution environment, but it does cause one to question how well thought out the architecture of node.js is.

We’ve all heard stories about how LinkedIn and others have used node.js to parallelize their web services, etc.  But if you care to listen you hear even more stories about pitfalls people run into using node.js.

I’m not bashing node.js, but it’s a good idea to remember that node.js is a young, relatively untried product with a lot of lessons to learned and a lot of people who don’t know basic principals jumping on the band wagon because it is new and sounds cool, thinking it’s a pancea for everything.

Reminds me of Ruby on Rails.

Training is the key to a successful software development business

What makes a software company successful? A book in 1987 gave the definitive answer: People

Specifically it’s software developers, but of course also the testers, analysts, project managers, and operations staff that surround them. And let’s not forget the bosses that manage them.

But how do you get (and keep) good people?

Microsoft started the trend of spoiling them with cokes, games, bean bags, flexible hours, etc. Or rather, they were famously the first company to continue the laid back atmosphere once they became a “real” business. Although that atmosphere has become largely artificial, and many people have realized that the primary motive has been to keep them in the office — and lots more have moved on to Google where the perks are slightly larger and the hipness factor is slightly higher.

Hipness is another strategy, but it’s ephemeral. Google had it for a while, but it’s fading. Apple has always had it for a small (but in recent years, rapidly increasing) minority. Netflix had it for about a year. But that withered with their stock.

Stock options were the supposed de facto solution, but it turns out that that really only worked with the first consumer software companies in the early 1980s — Microsoft, Oracle, etc. It looked like it was continuing with Netscape and into the dotcom boom, but ask any developer except Jamie Zawinski how their Netscape (or Amazon, or stock options turned out.

More money is the age old solution. Stock options are just another way of saying more money, but it turns out that most programmers want about $120K salary (varied based on the cost of living in their preferred location) and wouldn’t know what to do with more money. And would rather tell a job they don’t like to stick the extra few bucks if they’re not happy.

Google knows another secret about keeping developers happy — and that’s freedom. It famously gives them a half day to work on a (corporate approved) open source project of their choice goes a way towards keeping them happy.

Joel Spolsky thinks what they really want is a great work environment with big monitors and fast computers and private desks and hip furniture — but it turns out that the hip furniture is really what he’s selling. People work at Joel On Software to be associated with someone they think is hip, who treats them like their hip. Lots of people like that.

So keeping them happy is what’s needed. Pretty much all the above described methods attempts described above: Money, Perks, Coolness, Creative Freedom.

In fact a lot of great programmers would be happy in a closet hand punching cards on a 386 — if they could do their own thing. Creativity is what keeps them.

But not everyone is necessarily a top-notch programmer and self-motivated — or is creative. What do you do about the rest? Not everyone can afford to hire the best of the best and give them free time and toys and food and drinks and cool offices.

You train them. Developers like to learn. They like to learn on their own, but they also learn at different paces. They’re mostly introverted (using the old binary labels, not the newer pop-psych quadrants). They’re often frustrated by the way they end up having to do things because they don’t have the time to figure out how to do them better. They know there are better ways are out there, but they don’t always have all evening to study them out in their own time.

They’re either too busy playing pinball and foosball in the cafeteria waiting for the automated tests to approve their checkin so they can go back to their quiet office to play EverCraft or whatever the cool kids are playing these days — or they just want to spend some time with their own wife and kids, since 1990s Microsoft stock options made marrying a poor lonely geek socially acceptable.

But you can make those average developers better, some of those bad ones less dangerous, and even some of those genius ones better by training them. Everyone knows its hard to recruit and keep good developers, but few know the secret is to train them. To make them.

In the last 1990s, corporations learned something important. There was a massive shortage of programmers — and suddenly a huge demand for more programs. That hasn’t slackened. But they made a critical mistake. Afraid that they would lose their investment in training them to a startup competitor, they cut training budgets. They became wary of letting them try new technologies. They thought by keeping them ignorant, they could keep them from being stolen away.

That didn’t work out so well. But companies big and small still have an almost allergic reaction to training. I’m not talking seminars — I agree that the money spent sending developers to fancy offsite seminar training sessions was better spent on sending them to conventions where they could hang out with other geeks and get swag — but that’s not really training, that’s a vacation with moral support. There they could hear how programmers at other companies faced the same frustrations they did and hear lecturers say there’s a better way and nod their heads and buy their books.

Training doesn’t have to be expensive. In fact the best training is mentoring. Senior programmers don’t like to train junior programmers, but if you can get them too, you can rapidly increase their productivity. Like 5x. Like turning someone just out of college who doesn’t know what version control is and still thinks in a hybrid of BASIC and Microsoft Visual C. into the equivalent of a 5 year J2EE veteran in 1 year. Maybe even without the cynicism. (Just kidding, cynicism is the most valuable part of the training.)

Training is great in 2 more ways. Learning naturally develops into exploring. And once people start exploring (even if they’re just reinventing the wheel) they become creative. And mentoring naturally develops into leading.

You hire MBAs or oceanography majors to manage your programmers because you say you can’t find any senior programmers with leadership skills? Get them mentoring. Teaching new recruits who can optimize hello world in assembly but can’t create a hello world webapp in Ruby to be cynical architects is just what’s needed.

And here’s a little secret. You don’t have to hire CompSci majors for Carnegie Mellon to get decent programmers. You can turn anyone who understands algebra and can craft a coherent essay into a decent programmer with a year of training. And you can pay them a lot less than it would cost to send them to 4 year education at a state school. At least at first. After 12 months you might have to pay them the market rate. After 24 months, you’ll have to allow them creativity. After 48 months you’ll have to give them a chance to lead. But by then, they can start mentoring their peers who just got out of college.

And they will be loyal. People like training, they like creativity, and they like advancing their careers. They like money and perks and hipness too, so don’t skimp on that either — but you’ll be successful, so you can afford that.

Why did Agile become so popular?

In a recent post on his blog, Bob Marshall (, says that Agile Coaching is Evil.

He then gives the token denouncement that his post is not link bait, but then spends several paragraphs explaining how he thinks that the promises Agile Coaches make and fail to deliver on — even if unknowingly — qualifies as evil.

I’ll let others decide the morality of that, but I can affirm absolutely, by his own qualification, that Mr. Marshall was engaging in link baiting. But I’m also certain that it was deliberate, any protestation to the contrary notwithstanding — and that dishonesty *is* immoral — if not evil.

It appears that the flowchainsensei is marketing himself as a different brand of “coach”, having found the “Agile” appelation no longer to his liking.

It’s a good move, since there is a pretty popular backlash to “Agile” these days and so it’s almost as good marketing to say you can teach people how to work in a “not-Agile” way — the same way 10 years ago Agile folks claimed they could teach people how to work in a “not-Waterfall” way — with the major exception being that Agile was a label people used to describe themselves and “Waterfall” was a label used by Agile (and other) marketers to describe other people’s processes disparagingly.

There has been a strong backlash against “Agile” from the beginning, and contrary to the “Agile” acolytes claims, it wasn’t from management. It was from programmers, and has grown in popularity with manifestos like this.

But why did Agile become so popular?

I think it’s because almost everyone knows what “agile” means and knew their development process wasn’t. With the possible exception of Agile coaches, and definitely not any of the signatories of the Agile Manifesto.

Read it, and tell me if you see anything that fits the definition of the word “agile” in there?

Unless you consider writing documentation an unnecessary hindrance, and thus not writing it makes you more agile.

That single tenet was the most tempting part of the “Agile” process — and responsible for about 90% of the early converts among developers. It catered to the vanity of the “rockstar” developer personality — they were called “cowboys” back then — because they thought they were so good that they were above the process.

It’s common knowledge that most people think they’re better than they actually are, and their lack of experience prevents them from perceiving their lack of experience. They don’t know that they don’t know what they don’t know.

The idea that writing documentation was unnecessary was quickly disavowed by Agilists in discussions with managers (and others to whom accountability or maintenance was important) and they anticipated the need of such double-speak in the initial manifesto with the phrase:

while there is value in the items on the right (such as writing documentation),
we value the items on the left (such as not writing documentation) more.

As anyone who is familiar with decision making knows, when there is more value in one thing than another –and you have to make a choice– you choose the item that has more value. Especially if it costs less too!

If you’ve ever used a product backlog (or a prioritized task list), you’ve seen this in action. Things that get prioritized higher get done, things that have lower priority don’t get done. Because there is always something with a higher priority to do.

Chances are, the Agilists considered the word “agile” to be a generic one meaning essentially “good” and chose the label for it’s general sense of “coolness”. But it is possible that they too realized what so many who instinctively knew they wanted, and could patently see they did not have: that their development process was not agile, it was rigid.

Writing comprehensive documentation is rigid (as well as boring) — there’s no doubt about that — but so is writing unit tests. It’s an exercise is budgeting and resource management (not a development task) to determine if comprehensive unit tests or comprehensive documentation has a better payoff.

I’d guess from a cost/benefit perspective they’re about equal because documentation is easier to edit than code, and writing (and testing) against it is cheaper than writing code. But of course that’s a false dichotomy. There is a (variable) point at which there are diminishing returns for both writing documentation and writing unit tests.

And the choice isn’t between comprehensive documentation or comprehensive unit tests (neither can be completely comprehensive) — and neither one is a substitute for the other. You need both adequate documentation and adequate unit tests. Without the former, you can not hope to achieve the latter — unless your developers (the ones writing the unit tests) are determining the requirements.

I’ll completely skip over a discussion about whether unit tests are adequate, since I’ve addressed that before.

Doing things in short iterations is possibly the most universally implemented (and praised) practice of Agile. But short iterations doesn’t make one agile. It’s often disparaged as “Agilefall” or “Cascade” development. It can give the semblance of agility (like a higher resolution bitmap can give the semblance of curves) and can potentially help you practice limbering up if you’re particularly rigid — not through the iterative development process itself, but from the introspective effort that is supposed to accompany it — which is probably the least practiced part Agile.

But the reason the “retrospective” part is almost universally neglected (or scorned) is because it DOES NOT work. You can be forced to have meetings every two (or three, or one, or four weeks) where you write down in groups what went well and what didn’t, but unless your heart is in it, and there is unanimity on what is right and wrong (and how to fix it), it doesn’t have any benefit.

It’s really a misguided kindergarten-level way of avoiding decision making and responsibility. The technique doesn’t work on 5 year olds, and it doesn’t work any better on grown men and women.

And that’s the crux of it. Agile started out as a way for developers to take power from ineffective management (and hence it’s appeal, and it’s become a way for ineffective management to shift blame.

I agree with Bob Marshall that Agile Coaching is (if not evil), definitely not a good. It’s a case of those who can’t, teaching those who should be able to. And a deliberate lack of management. It’s snake oil, and every snake oil salesman knows they’re selling something of little value, and that’s evil.

They don’t have to know that it doesn’t work — they know that they are expending little effort and claiming large value. That contradicts the laws of thermodynamics. For every lever or pulley, there are a thousand perpetual motion machines for sale. Don’t expect someone at this stage to have invented something that’s better than the wheel. Even if they claim to have invented a better wheel (an incremental, rather than evolutionary change), you’d better check to see how good they are at making conventional wheels before you believe they’ve stumbled onto a better one.

Best case scenario, it’s just someone who doesn’t know how much they don’t know. But they darn well know they don’t know anything about the process they’re claiming to be able to improve.

The real issue is that we have managers who (even if they’re not inattentive) don’t know how to make software. They can’t effectively mentor developers (if they can’t actually teach them how to be developers (from experience) — and they can’t tell how to effective reward or discipline behavior if they don’t know how to manage.

Agile is not agile –it is definitely rigid– all the moreso the more purely it is practiced. The very concept that Agile has a purity measurement (and is said to be ineffective unless practiced 100% purely) smacks of rigidity (not to mention religiousity) and I think that’s a large cause of the backlash.

The solution isn’t an new process, movement, or manifesto. Although to those selling silver bullets, newness is advantageous, because it’s flaws have not been explored and it’s dogmatism is not yet apparent.

The crisis is not one of process but of leadership. And you can’t outsource that to a consultant or a wet behind the ears MBA who’s been pampered his whole life, or someone with a condescending attitude and a bunch of feel good aphorisms, whatever they label them.

Skip internal tag Jenkins Git

While setting up a build on Jenkins using the Git plugin (not sure what the Github plugin does, since it doesn’t handle checkout from github) I ran into the following error message:

Building in workspace /var/lib/jenkins/jobs/qasite-dashboard-selenium-tests/workspace
Checkout:workspace / /var/lib/jenkins/jobs/qasite-dashboard-selenium-tests/workspace - hudson.remoting.LocalChannel@f0f2eb
Using strategy: Default
Checkout:workspace / /var/lib/jenkins/jobs/qasite-dashboard-selenium-tests/workspace - hudson.remoting.LocalChannel@f0f2eb
Fetching changes from 1 remote Git repository
Fetching upstream changes from
Commencing build of Revision 3b4da0edc2c97cb8837129471c45510a7610a63f (origin/development)
Checking out Revision 3b4da0edc2c97cb8837129471c45510a7610a63f (origin/development)
FATAL: Could not apply tag jenkins-qasite-dashboard-selenium-tests-16
hudson.plugins.git.GitException: Could not apply tag jenkins-qasite-dashboard-selenium-tests-16
    at hudson.plugins.git.GitAPI.tag(
    at hudson.plugins.git.GitSCM$4.invoke(
    at hudson.plugins.git.GitSCM$4.invoke(
    at hudson.FilePath.act(
    at hudson.FilePath.act(
    at hudson.plugins.git.GitSCM.checkout(
    at hudson.model.AbstractProject.checkout(
    at hudson.model.AbstractBuild$AbstractRunner.checkout(
    at hudson.model.AbstractBuild$
    at hudson.model.ResourceController.execute(
Caused by: hudson.plugins.git.GitException: Error performing command: git tag -a -f -m Jenkins Build #16 jenkins-qasite-dashboard-selenium-tests-16
Command "git tag -a -f -m Jenkins Build #16 jenkins-qasite-dashboard-selenium-tests-16" returned status code 128:
*** Please tell me who you are.


  git config --global ""
  git config --global "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: empty ident   not allowed

	at hudson.plugins.git.GitAPI.launchCommandIn(
	at hudson.plugins.git.GitAPI.launchCommand(
	at hudson.plugins.git.GitAPI.launchCommand(
	at hudson.plugins.git.GitAPI.tag(
	... 12 more
Caused by: hudson.plugins.git.GitException: Command "git tag -a -f -m Jenkins Build #16 jenkins-qasite-dashboard-16" returned status code 128: 

Even after adding the git config info I was getting the same message.   I found this post on StackOverflow enlightening:

Turns out that somewhere buried under the job configuration for the git step (not clearly labeled as a step on Jenkins) if you click a button labeled “Advanced” and search for the text “Skip internal tag” there is a checkbox that you need to check if you don’t want Jenkins to create a tag on your remote repository after every build (yeah, it wants to do that every time, even if the build fails — and it will fail the build if it can’t.)

I’m liking Bamboo more and more, even though it’s buggy, has a weird shell (busybox, maybe?), has not nearly as many plugins as Jenkins, and is a resource hog.