Short video introduction:

It doesn’t work!

Disclaimer

Though I incorporated feedback from a lot of colleagues, this was never “the” SBP framework. Our company culture is as informal as it can be, with just enough structure. Making a framework such as this official is more than we need. We do not want people to think of their careers as a checklist to complete or as a ladder to climb. Instead, we suggest people mostly focus on growing their impact, picking up and mastering new competencies along the way.

Framework

The base framework is rather abstract, defining 10 coarse-grained competencies that apply to all engineering roles, such as “maintaining quality” or “planning and organizing”. The competencies have up to 7 different levels. These 10 competencies are grouped into Craftsmanship, Human, and Leadership categories.

The Progression app is organized into strict levels. To make the tool work for us, the online version of the SBP framework defines these levels too. But these are not actually used within SBP! It’s quite common to have some competencies at a very high level and others still much lower, and that’s ok with us. One common example is engineers with very highly rated craftsmanship skills but lower rated teamwork skills. The reverse also happens!

Roles

In this framework there are only three roles defined: engineers, tech leads, and technology officers.

Engineers are our individual contributors (ICs), an industry term we don’t usually use. When engineers lead others, they do so primarily by example, based on the content of their work.

Tech leads (TLs) are engineers that accept additional leadership responsibilities. Tech leads are not in charge of teams, we expect servant leadership of them.

Technology officers (TOs) are a lot like tech leads but with somewhat different leadership responsibilities. While tech lead engineers lead one specific team, our technology officers have a somewhat broader scope. They’re a bit further removed from the day-to-day engineering work, giving them time to focus on cross-team topics and to do work for which there isn’t a dedicated engineering team available, like presales.

TLs and TOs are leaders but not managers. At many companies with career ladders and levels, TLs and TOs might be considered staff or principal engineers, i.e. they are still on the “individual contributor” track. While some engineers progress by taking a TL or TO role, it’s important to realize there are engineers at a “staff” or “principal” level without a TL or TO role. Those engineers could match a different staff-plus archetype (like Solver or Architect), or just not match up with such a rigid framework at all. It’s also possible to switch from a TL or TO role to a regular engineering role.

Engineers at SBP do have additional roles at times that require additional competencies. For example, engineers can become coaches. These are not part of the framework.

Competencies

While Progression is optimized for defining pretty specific skills for pretty specific roles, this particular framework is broad and generic, and so is focused around competencies.

There are up to 7 different levels defined per competency.

While the basic competencies are the same for TLs and TOs as for engineers, our expectations of where our leaders will focus are a little different, and so for some skills there is a leadership (L) variant defined. Since Progression doesn’t support such variant definitions, there’s some overlap/copies.

Competency example: Client Orientation

SBP is an outsourcing company. Our engineers work in self-steering teams dedicated to a single customer. Because all our engineers have a lot of customer interaction, the framework is relatively heavy on communication compared to many other progression examples.

Progression for comparing roles & levels

One cool feature in Progression is how it allows you to put different roles and different levels side by side. This can help engineers think about which direction they might develop in next, though since at SBP we don’t actually use levels it’s of limited use for us.

Progression for check-ins

A recent addition to Progression is a new check-in feature that arranges a guided conversation between engineers and their managers (or in SBP its case with no line management, coaches). The idea is to agree together on career progression.

I like what they’ve built here. I imagine it’s a good way for HR departments to help beginning engineering managers have decent-quality factful conversations. But it’s not the right tool for Schuberg Philis.

It’s a great example of our “too much structure” rule kicking in. We have an internal 360-degree feedback process that is less structured than this, asking open questions instead, focusing on the important feedback conversation more and happily sacrificing some structured/detailed record-keeping in the process. Maybe when SBP is 10 times bigger that has to change?

Spreadsheet version

I created the Progression version of this framework based on my initial spreadsheet version of this career framework. I’ve found most people feel quite overwhelmed (at best) when confronted with this spreadsheet, while HR professionals appreciate the comprehensive overview.

Success!

Developing this framework helped me think through career guidance in great depth and that has helped make my advice to others clearer. I sometimes refer back to it when helping to weigh or appraise feedback, during salary discussions, or when writing job specs.

Putting the framework in Progression really helped to compare our framework to other engineering companies and to benchmark our internal levelless ranking system to common industry levels. Sometimes I can use it to explain to our younger engineers that no, they would not be ranked as principal engineers at Google, yet!

The process of iterating on the framework together was valuable. Asking and discussing feedback helped to align our leadership’s thinking about how we appraise and value our colleagues. Beyond leadership, trying to describe very different work in common terms (dev and ops, for example) helped the people involved to understand more about each other.

Our HR professionals happily adopted the framework, for example to develop and finetune our internal leadership training. For my HR colleagues relatively new to IT organizations it really helped them to see and understand both the breadth and the depth of our careers.

Failure!

This framework hasn’t helped our engineers or their coaches! My colleagues may look at it, perhaps understand it as reasonable and sensible…and then will never use it during their career planning or career development conversations.

People are unsure why, but they tell me the framework “doesn’t feel right”.

I believe one problem is that the level of abstraction is so high people don’t know how to apply it for themselves in a practical way. Making more concrete roles (like “java developer”) with more concrete skills (like “java programming”) could help, but that’s a lot of work to set up and maintain.

I believe another problem is that the setup is still too constrained: our actual appraisal system is more free-form. I imagine if we were to strictly define career levels that would then prompt more colleagues to use this framework. Especially if we linked the levels to salary ranges. And…I don’t know that would be positive. SBP tries to be a place where all our colleagues can “become who they are”, which for many colleagues can mean quite an “unbalanced” set of competencies.

The framework also hasn’t helped me! Our non-technical coaches have difficulty using the more technical competencies. You need an experienced technologist’s help to evaluate the quality of your solutions designs or implementation plans, to determine how competent you got. It sucks to have a conversation about those things with a non-technical coach.

So in the end even if my colleagues do use this framework, they will still ask for my help, and I save no time!

Fortunately, it’s the kind of conversation I love to have.

Permanent contract vs 1-year contract

Dutch employment law can get pretty complex, making the non-financial terms important when evaluating salary. One key factor is whether you’re looking at a permanent contract or not.

For more senior positions, permanent contracts are common. If you do your job well and the company is doing ok financially, your employer cannot easily fire you from a permanent contract, which makes these popular with risk-averse employees and unpopular with commitment-averse employers.

For more junior positions, 1-year contracts are common. 1-year contracts often get automatically extended, and then can convert into permanent contracts automatically after 3 years.

Getting a mortgage is harder without a permanent contract since banks are also risk-averse.

The base salaries for both contract types tend to be comparable, but permanent contracts usually have better non-base salary benefits leading to a better total benefits package.

Common benefits package components

A Dutch contract usually has a yearly salary listed for a full-time job. Usually full-time for developer jobs means 40 hours a week. If you work less hours you will be paid proportionally less, while if you structurally work more you typically do not get paid for that at all.

The base yearly salary includes 8% yearly holiday allowance, which is usually paid once a year in May. In May you receive 15.67% of your yearly salary, while in all other months you receive 7.67%.

Discretionary bonuses are now reasonably common in Dutch IT companies, paid out yearly, with 5%-20% of your yearly salary reasonably common, and 10% probably the most common percentage. Usually discretionary means that your manager informs you once a year if you get your bonus and how much you will get, i.e. your employer can decide for any reason not to pay you the bonus.

It is uncommon for Dutch employers to actually promise bonuses, or otherwise tie bonus payout to profits or performance targets in a transparent way. When that arrangement is written into your contract it is a performance bonus instead of the common discretionary bonus, which you may encounter for more commercial roles (i.e. a role as a presales engineer).

Stock options, shares, employee ownership plans, and profit sharing agreements are relatively uncommon for Dutch IT companies. Even when they are used, the “standard” approaches that exist in the US tech industry for stock options are not standard here, which means most US-centric advice on the internet about such things does not apply. The actual benefit of such programs relative to base salary can range from pretty neglible (more common) to pretty dominant (rare).

Cars in The Netherlands are expensive to own and drive, due to how taxes for them work as well as due to high taxes on gas. Lease cars are common, where your employer arranges a car for you (often of the type and with the options you choose) with a leasing company. Your employer pays for your car every month, including for any gas used. If you also use this car for non-work travel then you have to pay extra income tax based on the catalog value of the car. If you like driving a modern car this usually works out cheaper than other approaches. At some employers, lease cars are getting replaced with mobility budgets or public transport passes.

A full-time contract includes a mandatory minimum of 20 vacation days. These are a legal right; you might see them as a government-mandated benefit. Many employers include extra vacation days above 20, sometimes with various differing rules for exchanging days for cash, buying extra days, and so on. Besides vacation you also have rights to parental leave, sick days, and so on.

Paid overtime and standby shift compensation are rather uncommon for developers. If your contract does include paid overtime it is likely based on extra hours worked per week or month, which means tracking all hours. Night or weekend hours may be paid at 150% or 200%. Working a lot of overtime can mean quite significant additional income.

Most permanent contracts will include a pension, for which there are many different and confusing different setups that are hard to compare and value. It’s common these days that your employer pays some monthly amount into a fund on your behalf that is based on a combination of your base salary plus your age, with the amount being only a few percent for people under 30 and quite a lot for those close to retirement. The fine print around Dutch pensions is often more complex than for the rest of the employment contract combined.

Many employers also mention providing the things you need to do your job (like laptops, phones, and training budget) as benefits, or they point out lunch or snack arrangement or fun company events. While of course it’s nice if an employer provides such perks, when contrasting offers financially most of these are usually not significant. It’s also very rare for these perks to be promised contractually.

One aspect that is not considered a perk is what working from home arrangements have been made. Besides your own preferences around this, not having a daily commute may save you money & time, while not having your home office setup paid for may cost you money or health.

Salary ranges

Dutch developer salaries are perhaps €43k average, compared to €38k median salary across the country for all jobs. I’ve never heard of a developer making half that average, while double the average and more does happen. Most developers make close to and perhaps a bit less than the average, while a select few developers make a lot more. What’s going on?

• Senior developers switch into different higher-paying jobs that have different titles, like architect, manager, or consultant.
• Senior developers go freelance, so they write invoices instead of receive a salary.
• Senior developers don’t do things to increase their salary, instead doing things they find more fun or interesting.
• We have no idea what individual developers are worth, so there are few good arguments for employers to pay much higher than average.

Even for those salaried developers that do continue to hone their craft and gain the experience that clearly makes them more valuable, I believe the salary curve flattens out. A bit of this is simply due to how The Netherlands thinks about what’s a fair distribution of income. A bit of this is because developers don’t like salary negotiation so their employer ends up paying them what they can, instead of what they are worth.

Evaluating benefits packages

With all that basic understanding in place, you might now enjoy playing with this google sheet of example developer salaries that I invented to make the above graph:

The sheet might help as a baseline for evaluating different benefits packages and contrasting some different salary ladders against a simple baseline. The absolute values are probably less useful, though I did attempt to have them be reasonable based on my own experience and some general awareness of 2021 numbers at a couple different companies.

Sample goal: cool company where no-one worries about money

FAANG pay high salaries but not everyone wants to work for them. You may choose to work for a smaller company, or a cooler one, or one with better values. Perhaps then we can pay a bit less?

Let’s look for a really cool small company with solid values, and see what they pay. Bryan Cantrill’s Oxide Computer pays everyone who works at their company a $180k base salary (€160k). That is enough:

1. to live comfortably with 3 kids in the Bay Area;
2. to be in the top 5% of income in the US;
3. to not worry about money, even when putting extra organic raspberries in your shopping basket.

They also provide good benefits like healthcare and stock options. For them, €160k is enough to take money of the table as a employee demotivator, enough to assemble a great team.

The people that work at Oxide Computer could earn more at other employers. That’s ok: beyond some amount, more money simply matters less than other things, like pursuing autonomy, mastery, and purpose. What they offer candidates is something like join a great team, try to make something cool following decent principles, live comfortably, and have a small chance to become independently wealthy.

While it’s not a goal for them forever, Oxide so far has (quite) senior engineers only, of which Bryan writes “engineers at the earliest stages of their career are unlikely to be a fit for us right now”.

Let’s imagine we start a similarly cool company, with similar hiring goals, but in Amsterdam, The Netherlands. Let’s imagine we want to hire (someone like) Bryan as our first awesome senior engineer. He’ll get options or shares of course, and a nice pension, and so on, but what base salary should we pay him so money is not a worry?

€76.000?

The estimated modal income in The Netherlands for 2022 is €38k gross. €38k is the income where 50% of employees make less and 50% make more than that. An example of a job where you can expect to make about €38k is as a nurse with a college degree.

€76k is double that modal income. Ignoring market conditions, I consider double-modal quite a decent benchmark salary for individual contributor senior software engineers with a lot of experience.

• Until our recent housing crisis, a double-modal income was enough to buy a house, support a family, and otherwise live quite a comfortable life in The Netherlands;
• €76k is more than the average senior software engineer makes in The Netherlands, which is probably about €60k;
• Above-average does seem reasonable to me, after all this is Bryan Cantrill we’re trying to hire!
• It feels right to me.

Unfortunately, with the rising house prices, €76k salary is currently not enough for Bryan to buy a comfortable Amsterdam family home, and it’s definitely not enough for unlimited organic fruit. We should deal with our housing crisis and make healthy food cheaper, but in the meantime…

€100.000??

1. To live in a modern 4 bedroom house in the greater Amsterdam area, Bryan’s mortgage will be about €550k, for which he will need a salary of about €100k;
2. He’ll pay about 40% income tax, so will have net income of about €60k, which puts him in the top 5% of income in The Netherlands;
3. After base costs, perhaps €30k remains as disposable income, which should be plenty for all the raspberries he’d like, plus a couple of family ski trips a year, plus buying carbon offsets for that trip.

€100k ought to be enough of a base salary based on the goals we picked. It’s is a lot more than most software engineers make in The Netherlands, but $180k is of course also more than most software engineers make in the USA. We want excellent people so we will have to pay an excellent salary

€140.000???

Most expats prefer living in Amsterdam itself instead of merely close to it. It’s nice when the staff at the local organic grocery shop speaks fluent English as they point out the organic raspberries isle. To live in Amsterdam itself, Bryan’s cost of living goes up a lot, especially for housing.

1. He will need perhaps an €800k mortgage, for which he needs a salary of about €140k;
2. His taxes will then be a bit higher too, for a net income of about €80k, which now puts him in the top 2% of income in The Netherlands;
3. Perhaps €35k of that is now disposable income, still plenty for those more expensive Amsterdam raspberries.

Beyond just money, the living standards for the top 5%/2% of earners in The Netherlands are arguably quite amazing compared to the Bay Area. After all The Netherlands is one of the best places to live in the world! Rather than try to flesh out that argument here I’ll point out the excellent YouTube channel NotJustBikes.

So is this €140k enough to get Bryan to join us? Of course €140k isn’t €160k…but since The Netherlands is so awesome, it does seems somewhat reasonable to me to have to take a bit of a paycut to live here, just like it also seems reasonable to take a bit of a paycut to work for an amazing company such as Oxide Computer. That 12% paycut might be ok to Bryan, and is a bit less than the 15% cost of living difference between the Bay Area and Amsterdam, so day to day it may not feel like a big paycut to him.

€160.000????

It’s 2022 and many US tech companies are all-in on full-remote work, often no longer adjusting for cost of living when setting global salaries. More and more of these companies will pay those Bay Area salaries over here, so to be globally competitive in the job market €160k now really is the rate to match and that’s what our imaginary Amsterdam startup should probably do.

Wow. While €100k is already considered a high salary by most Dutch standards for just about any regular job, €160k plus benefits is more than most Dutch software engineers imagined they could make at any point in their careers. Over here that €160k makes you a part of the 1% even before considering the rest of your benefits package. Many software engineers are loyal employees who don’t like to change jobs, but finding out a life-changing amount of money is perhaps on offer (while building cool things with Bryan Cantrill, no less!) many of us are tempted!

On the other side of the negotiating table, as a hiring manager who pays their engineers €76k or perhaps €100k and thinks that it is quite a generous and reasonable amount, it can be jarring when we hear that those same engineers now earn 60-100% more somewhere else for the same (or more fun!) work. Should we try and somehow match the US salaries, or, should we accept we’re not a globally competitive employer and risk losing some of our best people? Ugh! That also doesn’t feel right.

How high will salaries go?

Over the last few quarters a lot of Dutch employers are bumping engineering salaries by 10-20% or more, German IT salaries are going up fast too, and probably this really is true across Europe. Employers are trying to remain competitive in the job market…but is a 10% bump enough when 60% or 100% more salary is on offer elsewhere?

Since the scarcity of senior software engineers won’t be fixed change quickly, and the BigTech companies can afford to pay a lot, it seems like the top global salaries are poised to still go up further in the short term, in turn creating further strain in our local job market, and keeping the money topic on the table for most IT employers and employees everywhere.

Will salaries come back down?

Perhaps. The global economy seems to be overheating. It could be we are in another bubble, which could burst, which could turn the scarcity of engineers into surplus of engineers really quickly as investment drops. Since no-one really understands how the global economy works, it’s hard to be sure.

(Disclaimers: I work in outsourcing, not at a startup. I work and live in the Amsterdam area but not in Amsterdam itself. This is not how compensation is set up where I work. I make a lot less money than Bryan Cantrill, who was also not consulted about this blog post (I owe you some raspberries, Bryan).)

To fight your own bias think: “I don’t know”. Question your own thoughts, knowledge and beliefs, often, more than is comfortable. This will improve your thinking and reduce your bias. The more you question yourself the easier it gets.

To help others fight their bias, lead by example. Say “I don’t know” out loud. By saying it you make it easier for them to think and say it too.

To think and work together, listen to and try to understand others with a mindset of not knowing. If our own position is unsure we are more open to ideas of others. As a leader in a group conversation you can say: “I don’t know, what do others think?”

I don’t know any of this for sure! I think the above is good advice because scientific and philosophical content points in this direction and (I think…) I have some experience with it myself. But don’t take my word for it!

Frank Meester is a Dutch philosopher who has written a new book

Waarom we de wereld niet rond krijgen: pleidooi voor inconsequentie
(“Why we can’t make the world whole: plea for inconsequence” – amazon | bol | libris)

I found the book’s form almost as interesting as its main ideas. On the one hand it’s a “properly” structured philosophical work with a clear argument that’s built up and examined from several perspectives, and supported with many references to prior work (including Kant, Wittgenstein, Sartre, Peirce). On the other hand it’s an accessible story that’s full of casual and witty sentences, anecdotes, and modern cultural references, never taking itself too seriously for too long, and taking care to provide simple explanations for all philosophical concepts referenced.

I think this dual form works well to get across the main idea. Meester’s main point in the book is to introduce his own new philosophy dubbed inconsequentialism, which is summarized in his main statements:

1. It isn’t possible to tell a consistent or consequent story about life.
2. Because total consequence is not possible, we do not have to find it annoying when we don’t quite succeed at it.
3. In practices almost everything is a little bit possible.
4. A little bit is already quite something.

At first these seemed almost trivial statements to me that could hardly be the basis for a full book, but Meester diggs into them and their impact in quite some detail. In the first part of the book Meester reviews logic and reality for their inherent (in)consistency, building his philosophy on top of work by primarily Wittgenstein and Kant, assisted with reference to modern physics and quantum physics. In part two he moves on to consequences of his inconsequentialism, looking at pragmatism, the hermeneutic circle, ethics, and determinism. In part three he contrasts his inconsequentialism with existentialism, humanism, absurdism, and performativity.

The weakest part of the book for me is where Meester dives into physics and specifically quantum physics, subject matter he admits he is not too deeply familiar with. For some reason it is pretty popular these days to take the small scale awkwardness of quantum physics to make large scale arguments about the nature of nature, and Meester does the same. But the instances where that scale-up makes sense in nature are a rare exception to the rule that in the large you can usually get by with Newtonian physics, a rule that ought to mostly carry over to philosophical constructs. Fortunately most of Meester’s argument still holds up even when rejecting his view on modern physics.

I guess Meester’s incosequentialism might be very Dutch. In many ways it’s a modest and moderate philosophy, down-to-earth and practical, forgiving and reasonably applicable to daily life. As far as philosophies go, this one lacks grandeur, does not strive for greatness, doesn’t strongly critique or fully reject the ideas that came before. Instead Meester is poised to get along about equally well with influential Greeks, German, French and American philosophers.

I guess if Meester’s very Dutch, I might also be very Dutch, because I find this book to be closer to my own personal philosophy than most other philosophical work I’ve read recently. By rejecting absolute/simple answers to typical philosophical puzzles as well as the absoluteness/simpleness of those puzzles themselves, there’s room for nuance. For example:

• It’s good to try to live the good life, even if it is hard to have an exact definition of what the good life is or to be perfect at living it.
• It’s good to have lofty dreams and ideals, while it’s also ok to have practical pragmatic compromise when it is not possible to reach the ideal.
• Compromising on ideals doesn’t make the ideals bad or useless.

I find such thoughts hopeful and comforting. If you can read Dutch I can recommend this as an accessible and enjoyable book, even if normally you don’t read philosophy. (As far as I can tell the book has no English translation yet but I’ll update this post if that changes.)

For example, I had an issue with cucumber + autotest + rails where I had both rails 3.1 and rails 3.2 apps using the same RVM. Since I was confused, and in a hurry, I figured one brute force option to unconfuse me would be to simply remove all old versions of all gems from the environment. I did just that, and thought I’d explain the process of incrementally coming up with the right shell script one liner.

First things first, let’s figure out what we have installed:

$ gem
...
  Usage:
...
    gem command [arguments...] [options...]
...
  Examples:
...
    gem list --local
...

Ok, I guess we want gem list:

$ gem list
*** LOCAL GEMS ***

actionmailer (3.2.2, 3.1.1)
...
ZenTest (4.7.0)

actionmailer is part of rails, and we can see there are two versions installed. Let’s figure out how to remove one of them…

$ gem help commands
GEM commands are:
...
    uninstall         Uninstall gems from the local repository
...
$ gem uninstall --help
Usage: gem uninstall GEMNAME [GEMNAME ...] [options]

  Options:
...
    -I, --[no-]ignore-dependencies   Ignore dependency requirements while
                                     uninstalling
...
    -v, --version VERSION            Specify version of gem to uninstall
...

Great. Let’s try it:

$ gem uninstall actionmailer -v 3.1.1

You have requested to uninstall the gem:
	actionmailer-3.1.1
rails-3.1.1 depends on [actionmailer (= 3.1.1)]
If you remove this gems, one or more dependencies will not be met.
Continue with Uninstall? [Yn]  y
Successfully uninstalled actionmailer-3.1.1

Ok, so we need to have it not ask us that question. From studying the command line options, the magic switch is to add -I.

So once we have pinpointed a version to uninstall, our command becomes something like gem uninstall -I $gem_name -v $gem_version. Now we need the list of gems to do this on, so we can run that command a bunch of times.

We’ll now start building our big fancy one-line script. I tend to do this by typing the command, executing it, and then pressing the up arrow to go back in the bash history to re-edit the same command.

Looking at the gem list output again, we can see that any gem with multiple installed versions has a comma in the output, and gems with just one installed version do not. We can use grep to filter the list:

$ gem list | grep ','
actionmailer (3.2.2, 3.1.1)
...
sprockets (2.1.2, 2.0.3)

Great. Now we need to extract out of this just the name of the gem and the problematic version. One way of looking at the listing is as a space-seperated set of fields: gemname SPACE (version1, SPACE version2), so we can use cut to pick fields one and three:

$ gem list | grep ',' | cut -d ' ' -f 1,3
...
gherkin 2.5.4)
jquery-rails 2.0.1,
...

Wait, why does the jquery-rails line look different?

$ gem list | grep ',' | grep jquery-rails
jquery-rails (2.0.2, 2.0.1, 1.0.16)

Ok, so it has 3 versions. Really in this instance we need to pick out fields 3,4,5,… and loop over them, uninstalling all the old versions. But that’s a bit hard to do. The alternative is to just pick out field 3 anyway, and run the same command a few times. The first time will remove jquery-rails 2.0.1, and then the second time the output will become something like

jquery-rails (2.0.2, 1.0.16)

and we will remove jquery-=rails 1.0.16.

We’re almost there, but we still need to get rid of the ( and , in our output.

$ gem list | grep ',' | cut -d ' ' -f 1,3 | sed 's/,//' | sed 's/)//'
...
childprocess 0.2.2
...
rack 1.3.5

Looking nice and clean.

To run our gem uninstall command, we know we need to prefix the version with -v

$ gem list | grep ',' | cut -d ' ' -f 1,3 | sed 's/,//' | sed 's/)//' | sed 's/ / -v /'
...
childprocess -v 0.2.2
...

Ok, so now at the start of the list we want to put gem uninstall -I . We can use the regular expression ‘^’ to match the beginning of the line. We’ll need sed to evaluate our regular expressions…

$ gem list | grep ',' | cut -d ' ' -f 1,3 | sed 's/,//' | sed 's/)//' | sed 's/ / -v /' | sed -r 's/^/gem uninstall -I/'
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
       sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]

Ugh. -r is the switch used in the GNU version of sed. I’m on Mac OS X which comes with BSD sed, which uses -E.

$ gem list | grep ',' | cut -d ' ' -f 1,3 | sed 's/,//' | sed 's/)//' | sed 's/ / -v /' | sed -E 's/^/gem uninstall -I /'
...
gem uninstall -I childprocess -v 0.2.2
...

Ok. That looks like its the list of commands that we want to run. Since the next step will be the big one, before we actually run all the commands, let’s check that we can do so safely. A nice trick is to echo out the commands.

$ gem list | grep ',' | cut -d ' ' -f 1,3 | sed 's/,//' | sed 's/)//' | sed 's/ / -v /' | sed -E 's/^/echo gem uninstall -I /' | sh
gem uninstall -I childprocess -v 0.2.2

Ok, so evaluating through sh works. Let’s remove the echo:

$ gem list | grep ',' | cut -d ' ' -f 1,3 | sed 's/,//' | sed 's/)//' | sed 's/ / -v /' | sed -E 's/^/gem uninstall -I /' | sh
...
Successfully uninstalled childprocess-0.2.2
Removing rails
Successfully uninstalled rails-3.1.1
...

I have no idea why that rails gem gets the extra line of output. But it looks like it all went ok. Let’s remove the ‘sh’ again and check:

$ gem list | grep ',' | cut -d ' ' -f 1,3 | sed 's/,//' | sed 's/)//' | sed 's/ / -v /' | sed -E 's/^/gem uninstall /'
gem uninstall jquery-rails -v 1.0.16

Oh, that’s right, the jquery-rails gem had two versions. Let’s uninstall that, then. We press the up arrow twice to get back the command line that ends with | sh, and run that. Great: we’re all done!

Let’s look at the final command again:

gem list | grep ',' | cut -d ' ' -f 1,3 | sed 's/,//' | sed 's/)//' | sed 's/ / -v /' | sed -E 's/^/gem uninstall -I /' | sh

1. gem list shows us the locally installed gems
2. | grep ',' limits that output to lines with a comma in it, that is, gems with multiple versions installed
3. | cut -d ' ' -f 1,3 splits the remaining lines by spaces, then picks fields one and three
4. | sed 's/,//' removes all , from the output
5. | sed 's/)//' removes all ) from the output
6. | sed 's/ / -v /' replaces the (one remaining) space with -v
7. | sed -E 's/^/gem uninstall -I /' puts gem uninstall -I at the start of every line
8. | sh evaluates all the lines in the output as commands

Note that, as a reusable program or installable shell script, this command really isn’t good enough. For example:

• It does not check the exit codes / statuses of the commands ran, instead it just assumes they run successfully
• It assumes the output of gem list will always match our expectations (for example, that the output header does not have a comma in it, or that gem names or versions cannot contain space, comma, or -- this may be true but I wouldn't know for sure)
• It assumes that -I is the only switch needed to prevent gem list from ever asking us questions

The best approach for a reusable command would probably be to write a script in ruby that used the RubyGems API. However, that would be much more work than writing our one-liner, which is “safe enough” for this use-once approach.

(For the record, this didn’t solve my rails + cucumber woes. Instead, it turns out I had run bundle install --without test at some point, and bundler remembers the --without. So I needed rm Gemfile.lock; bundle install --without ''.)

In 2011 I wrote a blog post about Oracle database license costs. Back then I took a look at the open source Spacewalk project using an Oracle Database. I was surprised it was not ridiculously expensive to set up, because a small Oracle Standard database doesn’t cost what a big Oracle Enteprise deployment costs.

In 2021 Oracle is even cheaper to get started with. There is a completely free tier at Oracle Cloud, essentially offering a hosted / managed version of Oracle Express. Oracle Standard now starts at $87/year.

This is not too impressive because everything else got cheaper too. About half of the monthly cost of the minimum Oracle Standard deployment on AWS is the license, which may be a bargain if Oracle is the right choice for your database. But the bigger the database the higher premium you pay for Oracle, especially if you switch to Oracle Enterprise, which can be 5x-10x more expensive.

The Spacewalk project made the effort to switch to postgres, before discontinuing itself completely in 2020.

Oracle Express free cloud hosting

You can get two managed databases, each with 1 CPU and 20GB storage, from Oracle Cloud, for free. With that you can also get two VMs with 0.2 CPU and 1 GB memory to work with those databases.

This is a more generous than the always free tier from AWS. From AWS you can permanently get 25GB of DynamoDB and 1 million Lambda requests per month. For databases and VMs the free offering for AWS is half of Oracle’s and it’s only for the first year: for RDS you get 1 CPU and 20GB of storage, for EC2 you also get 1 CPU.

Cheapest Oracle Standard license: $87/year

Amazon RDS has a license-included option for Oracle, the smallest db.t3.micro in a single availability zone is now $0.038/hour, or $193/year with a reserved instance. Besides the license this includes the virtual machine but still excludes storage and IO.

PostgreSQL on RDS by comparison is $0.018/hour, or $106/year with a reserved instance. If we substract that amount then we could say the cheapest oracle license on AWS is $0.020/hour or $87/year, perhaps half of what it cost a decade ago.

Realistic Oracle Standard license: pay 4x

You can’t do that much with a db.t3.micro instance running Oracle. For two reserved multi-AZ-replicating db.t3.2xlarge reserved instances with 8 CPU cores each, Oracle costs $12.4k/year while PostgreSQL comes to $3.3k/year. We could say a realistic Oracle Standard license cost for a realistic-size basic database setup is $9.1k/year, and Oracle Standard costs about 4 times more than PostgreSQL per core.

Typical Oracle Enterprise costs: pay 7x

For Oracle Enterprise, which remains quite expensive, on AWS you still have to bring your own license and roll your own deployment, though AWS provides a quick start for Oracle now. For any significant setup like that, maintenance effort is going to be significant.

You can buy a managed oracle enteprise database from Oracle Cloud. The smallest setup is a quarter of an X8 rack. It costs about $15/quarter-of-a-rack/hour, plus $1000/cpu/month, or about $106k/month for 96 cores with 44TB if the setup is fully utilized 24/7. AWS Aurora with postgres will be more like $16k/month, in other words Oracle Enterprise costs about 7 times more than PostgreSQL per core.

Comparing apple trees to orange juice

But that doesn’t mean Oracle is expensive for your needs. Aurora t3 cores are not the same as Exadata X8 cores. If Oracle can do something you need that PostgreSQL cannot, or if Oracle is much faster for your workload than PostgreSQL, it can still come out cheaper.

Even for a workload that supports both engines and even if the hardware is similar, tuning the database engine might make all the difference. Do you have someone tuning your database? I haven’t tuned a database for a decade, but it seems it can still easily increase performance by a factor of 10. If you spend $15k/month on a database, you should probably pay an expert $30k to make it fast.

Postgres growth

Even if these Oracle costs are not ridiculous, Postgres is still making inroads. Amazon (the shop, not AWS) switching off their Oracle was a major milestone back in 2019. Amazon reported 60% cost savings, which we may take with a grain of salt since they want to convince us to buy AWS services, but I doubt that things got more expensive for them.

Still, 60% savings is less than 86%, which would match that 7x number above. Probably AWS was not paying Oracle list prices, probably Oracle was a pretty good solution for some of their workloads, and probably AWS its costs managing its own infrastructure are lower than paying for Oracle Cloud.

On the other hand the internet is also full of examples of companies saving 90% or more, especially if they started off with applications that were not really a great fit for Oracle Database or if they were in a costly licensing audit fight with oracle.

Spacewalk itself started supporting PostgreSQL as a backend in 2011 which seems to be the default for a long while.

But spacewalk is discontinued and not many people care, because people manage their infra packages differently now. So with that context, was it worthwhile to make Spacewalk Oracle-independent? Did that investment effort back in 2011 pay itself back over the next decade? Given the number of Spacewalk installations, probably. How about the opportunity cost: could the project have invested that same effort in innovation instead?

2011 original post

I learned that spacewalk requires oracle the other day. To an open source weenie that sounds bad, but how much of a problem is it in practice?

Oracle XE is free, allowing up to 4GB of data, which seems like it would last you about 2 years of running spacewalk with a few thousand packages and a few hundred machines.

Oracle One costs $180 per user with a 5 user minimum, installable on one machine with two sockets (so up to 12 core for an AMD opteron). Including one year of support that adds up to about $1000.

But that’s a perpetual license. A 1-year oracle license is $36 per user, so adding in support, for one year or oracle on one machine you pay just shy of $200.

The next upgrade after oracle one is oracle standard, which costs $350 per user perpetual, or $70 per user per year.

So to get started with oracle you can use Oracle XE. When you run out of space you buy a decent database server machine and a $200 oracle license, allowing for 4 middleware nodes and one administrator. You have to pay that $200 once a year. Or you can run Oracle One in the cloud with Amazon RDS from $0.16 per hour.

When you need a resilient set up you will need oracle standard so you can cluster with Oracle RAC. You’ll need a license for each server (which can now have 4 sockets so up to 24 core), so your cost increases to 2 * 5 * 70 = $700/year.

You can also deploy oracle standard on Amazon RDS, where you’ll pay from $0.11 per hour per VM in addition to your license. Though it seems like at the moment, amazon RDS replication doesn’t work for oracle, so there’s probably no point in using that. Instead, you’ll have to switch to creating your own EBS-backed AMIs and installing oracle into them (or, perhaps, use an oracle-provided AMI).

I can find nothing that states there’s a limit to Oracle RAC scale with oracle standard. So you could potentially build big, BIG clusters this way.

$350/server/year is not exactly free of course, but this actually gets you really good clustering. That bit of capex gets you the ease of deployment and use of Oracle RAC. For some use cases, it’s probably easily the cheapest option if you take into account the opex of doing anything else. For example if Oracle Locator, the subset of Oracle Spatial that’s available in the standard edition, is good enough for you, this seems easier than figuring out your own clustered PostGIS setup. Postgres or mysql replication would’ve been just fine for spacewalk though.

Oracle Enterprise has additional features that aren’t really relevant for most scale-out (webby) stuff: using more than 4 sockets, some of the advanced administration/audit/backup tools, advanced security features, oracle spatial, compression, table partitioning, and some other bits. Enterprise still costs about a gazillion dollars.

We have to erase this idea that “oracle is ridiculously expensive” out of our brains. It just isn’t true anymore: there’s some very reasonable oracle database offerings for reasonable prices. Instead, the idea should now be “oracle enterprise is ridiculously expensive”. For things like spacewalk, where you don’t need the advanced stuff, it seems oracle is now almost always a very reasonable backend that probably can have a competitive TCO.

• Download ubuntu server, I chose the 64-bit LTS iso (10.04).
• create new empty VM called vanilla-ubuntu.
• Resize the disk image to 10GB.
• set RAM to 256MB.
• disable just about all options including audio support, usb support, printer support, file sharing, etc.
• set to NAT networking.
• attach ubuntu iso to cdrom drive.
• Accept all installer defaults, but select OpenSSH server as additional system to install
• Log in as created user
• run sudo passwd root to set the root password
• sudo apt-get update && sudo apt-get -y dist-upgrade
• to work around this bug, sudo rm /etc/motd.tail
• sudo shutdown -h now.
• open the settings pane, open the hard disk panel, and select “clean up disk”.
• disconnect and remove the floppy drive. Disconnect the DVD drive.
• (optional), open the directory containing the VM location in the terminal and mv *.vmwarevm/* . && rm -r *.vmwarevm. When you next open the VM (you will need to use File > Open), select “I moved it”.
• Make a backup copy of the virtual machine files, this is your clean VMWare-independent base VM.
• Start the virtual machine.
• in the Fusion menu, select Virtual Machine > Install VMWare tools, and install the tools:

  mkdir /mnt/cdrom
  mount /dev/cdrom /mnt/cdrom
  cd /tmp
  tar zxf /mnt/cdrom/VMwareTools*.tar.gz
  ./vmware-install.pl --default
  cd
  rm -Rf /tmp/vmware-tools-distrib
  

• Shut down the virtual machine
• Make a backup copy of the virtual machine files, this is your clean VMWare-ready base VM.

One thing that’s easy to notice is how lightning fast ubuntu starts up. The other thing to notice is just how old CentOS / RHEL 5 really is, and IMNSHO how overdue RHEL 6 was (and CentOS 6 is):

# uname -r
2.6.32-31-server
# python -V
Python 2.6.5
# perl -v
This is perl, v5.10.1 ...
# apt-cache show ruby1.9.1 | grep Version
Version: 1.9.1.378-1

# uname -r
2.6.18-238.9.1.el5
# python -V
Python 2.4.3
# perl -v
This is perl, v5.8.8 ...
# yum info ruby.x86_64 | grep Version
Version    : 1.8.5

Getting spacewalk installed is a bit of work since it (a) is huge, (b) needs an oracle database. But the published installation instructions worked fine for me:

Get a new Centos VM: ./make-centos32.py spacewalk. Note that Oracle express is 32 bit-only, so I decided to just do everything as a 32 bit install. Change the VM settings to use 1.5GB ram which seems like it is just about enough for development/demo purposes (specified minimum is 2GB).

Now, follow spacewalk instructions to set up Oracle Express:

$ sudo bash
# cat >>/etc/hosts <>/etc/hosts <   oracle-instantclient11.2-basic-11.2.0.2.0.i386.rpm \
>   oracle-instantclient11.2-sqlplus-11.2.0.2.0.i386.rpm

# export SWREPO=http://spacewalk.redhat.com/yum/1.4
# export EPEL=http://download.fedora.redhat.com/pub/epel
# rpm -Uvh $SWREPO/RHEL/5/i386/spacewalk-repo-1.4-1.el5.noarch.rpm
# rpm -Uvh $EPEL/5/i386/epel-release-5-4.noarch.rpm
# rpm -Uvh $SWREPO//RHEL/5/i386/spacewalk-client-repo-1.4-1.el5.noarch.rpm

# yum -y install oracle-lib-compat

# /etc/init.d/oracle-xe configure
HTTP port for Oracle Application Express: 9055
Database listener port: 1521
Password for SYS/SYSTEM: changeme12
Start at boot: y
# sqlplus 'sys/@//localhost/XE as sysdba'
SQL> create user spacewalk identified by spacewalk default tablespace users;
SQL> grant dba to spacewalk;
SQL> quit

# sqlplus spacewalk/spacewalk@//localhost/XE
SQL> alter system set processes = 400 scope=spfile; 
SQL> alter system set "_optimizer_filter_pred_pullup"=false scope=spfile; 
SQL> alter system set "_optimizer_cost_based_transformation"=off scope=spfile; 
SQL> quit  
# /sbin/service oracle-xe restart

Well that wasn’t so bad. I seem to recall installing oracle as being much harder and taking at least a day or two if you know what you’re doing. Maybe it’s just that the XE edition is so much easier, or maybe Oracle have just learned a trick or two in the 5? (wow!) years since I last touched it.

Now we’re ready to install spacewalk. The right yum repo configs were already imported, above, so we don’t need to do that again.

# yum install curl
# curl -o /tmp/gpg-key http://www.redhat.com/security/37017186.txt
# mv /tmp/gpg-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
# yum install spacewalk-oracle
...
Transaction Summary
=================================
Install     287 Package(s)
Upgrade       0 Package(s)

Total download size: 198 M
...

One of these packages results in the enabling of SELINUX. Let’s disable it again:

# echo 0 > /selinux/enforce
# vi /etc/selinux/config # change to SELINUX=disabled
# reboot

And now we can configure:

spacewalk-setup --disconnected
Database service name (SID)? XE
Username? spacewalk
Password? spacewalk
Admin Email Address? root@localhost
Should setup configure apache's default ssl server
    for you (saves original ssl.conf) [Y]? y
CA certificate password? spacewalk
Re-enter CA certificate password? spacewalk
Organization? spacewalk
Organization Unit [spacewalk.sandbox]? spacewalk
Email Address [root@localhost]? root@localhost
City? London
State? UK
Country code (Examples: "US", "JP", "IN",
    or type "?" to see a list)? GB

Now open https://spacewalk.sandbox/ in a web browser and create the admin account spacewalk/spacewalk.

Now to figure out how to make use of this beast….here’s a pretty good presentation from DevOps London about it:

Dev Ops meeting at youDevise, 3/3/2011 (Part 1 of 2) from youdevisetech on Vimeo.