> Let’s say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while.
This is one of my favorite blog posts, and it can basically be encapsulated in the idea of "innovation tokens." It is one of the most useful concepts I have had as a PM / eng leader in my career. It helps actually make the the right tradeoffs, and helps even more in explaining those tradeoffs to colleague of all levels. Highly recommend.
NickNaraghi
2 bare-metals with Linux, Postgres, HAProxy and PHP (or Python/Django) work perfectly fine for 99% of apps that businesses need. This will run with 99.99% uptime, 4-hour warranty from Dell/HPE (failover to the other server). Is also somewhat vertically scalable (upgrade RAM/SSD). Kids who finish high school can be taught to own and run this.
But .... if you run a multi cloud hybrid setup with kubernetes, service mesh, data [lake|pond|ocean] and millions of other fancy words in tech at each and every layer, you resume would look so awesome and you sound wicked smart. And the VP gets 600M budget for AWS and 600 developers, SREs, DevOps, PMO. It is not that things won't run, humans have perverse incentives.
I'm certain a lot of porn/adult industry run their setup like I mentioned with a Romanian dude running the entire infrastructure for $15K - $20K.
thelastgallon
I'll push back against this, despite it being so popular. I dislike the arbitrary "innovation tokens" and I think this entire concept really blurs the lines and feels sort of unserious.
Engineers should understand requirements, risks, tradeoffs, and potential gains. New technology may be right for that. Novel approaches may be right for that. "Novel" or "New" are only proxies and they're weak.
For example, I may think "New" means untested, but is that true? What if a new project has Jepsen testing, a fuzzing suite, massive compute running tons of oracle tests, etc? I should just say "Choose well tested" instead of "Choose old" - lots of old software is very poorly tested.
Maybe I think that "Old" implies better documentation, but does it? Lots of older projects have insane cruft and weird edge cases that are undocumented and accumulated over years.
Why do we need a metaphor? Why is "innovation token" helpful?
If you're incapable of evaluating a technology in terms of these properties, you aren't a serious developer and "boring" will not save you.
Sit down, write our your requirements, determine candidate solutions, and choose them based on their fit. "Boring" means nothing, it's a vague proxy term. "Well tsted", "performant for our use case", "developers know it", etc mean something.
> MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.
Literally every one of these has caused hilarious and disastrous failures for me in my career. But yep, boring.
> If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens.
What if you know NodeJS really well? Or MongoDb? What if you have empirical, verifiable reasons for why they fit better?
I'm a bit tired of "simple" and "boring" and other nonsense words in this field taking up the air in the room that should be spent evaluating solutions on their actual merits.
insanitybit
Some of this may have been a reaction to the era of Javascript framework churn. There were way too many different technologies for doing roughly the same job.
They all more or less worked.
On the other hand, IBM was late getting into integrated circuits. They had Solid Logic Technology, automated machinery for putting transistors into ceramic substrates to make tiny but discrete circuits. That's what powered the IBM System/360. Worked, but kept mainframe prices high and made IBM late to minicomputers.
Innovation when the problem is hard is more interesting. Look at the history of US long range bombers. The B-29 was effective, but underpowered, and had a lot of trouble getting off the ground fully loaded. So, after WWII, the next development was the B-36, which answered the question "what if we scaled up the B-29?"[1] Six propellers, and four jet engines (added late in the design cycle). Was a sky barge, but it worked, as long as no one was trying hard to shoot it down. No flyable aircraft remain. That was the boring technology approach.
After that came the B-47, which answered the question "what if we scaled up a jet fighter to bomber size?"[2] The B-47 was all jets, no props. It needed solid-fuel rocket boosters (!) to help it get off the ground, and a drag chute to slow it down on landing. It was terrible to fly; its operating speed and altitude were too near the "coffin corner" where stall and Mach buffet meet.[3] No flyable aircraft remain.
Then came the B-52. New engines. New airframe. New design. [4] That's the high-risk approach. It worked. Hundreds are still in active service. Outlasted most of its successors, the B-58 Hustler (a supersonic intercontinental bomber), the B-70, the F-111, etc.
On the commercial side, we have Boeing, whose most successful airliner is a variant of the B-737, which first flew in 1967. But that's another story.
I love this post. It’s also interesting to revisit in the age of agents.
Using the language of the article, I’d say “push all your innovation tokens into agents” is probably a good move. This means the tech your agents work with should all be boring tech.
Another way of saying this is “use in-distribution technology”. If agents are substantially better at Rust than Zig, probably you should use Rust, even if Zig is “better”. The amount that Zig is better is going to get swamped by the amount that in-distribution agents are better.
(This is not a claim that Rust actually is better, just a hypothetical fact pattern for discussion.)
This advice holds up, but there are caveats to keep in mind. Two off the top of my head:
1. I once worked for a company that had a large Cassandra cluster that was primarily serving the role of a distributed append-only log. This role is as perfect a fit for Cassandra as I can imagine. When we needed a distributed database for authentication, we decided to use Cassandra for it since we had in-house expertise (it was "boring technology") and a cluster we could piggy-back on for a while. When we needed a distributed database for a key-value service, we used the same reasoning and chose Cassandra again. There were many problems with Cassandra that I won't go into, but I will say that straying too far from Cassandra's comfort zone stretched it to a breaking point. And we had some downtime and rough nights because of it. Sometimes boring technology isn't enough, you also need boring workloads for it.
2. I was an early adopter for Rust. Before 1.0, it wasn't clear to most people whether the language would mount to anything. To me, I saw a formal proof assistant being put into programmer's hands and could tell right away it had a bright future. Rust was not a boring technology back then. You might say it is now (for some use cases). Adopting shiny new technologies that measurably improve confidence is not a risk. What's foolish is getting comfortable and complacent with familiar, aka boring, technologies that are difficult to use correctly.
jason_oster
Love this post. Surprisingly controversial; it hasn't made me very many engineering friends.
conrs
I wish there was a jobs board for companies that are somehow vetted for this type of engineering culture.
So many jobs are sold as “we’re pragmatist's” and when you show up there’s 5 devs, 50 repos and most of the work is discussing if x requirement should be a new micro service. The product is usually an web app with 10 entities and and an API.
Suppose it’s keeping people in jobs
cliche
Software that works year-after-year has never been a commodity. It's boring on the surface. It doesn't get the flashy posts. But I'll choose reliable over new in almost all cases.
comments (10)
This is one of my favorite blog posts, and it can basically be encapsulated in the idea of "innovation tokens." It is one of the most useful concepts I have had as a PM / eng leader in my career. It helps actually make the the right tradeoffs, and helps even more in explaining those tradeoffs to colleague of all levels. Highly recommend.
NickNaraghi
But .... if you run a multi cloud hybrid setup with kubernetes, service mesh, data [lake|pond|ocean] and millions of other fancy words in tech at each and every layer, you resume would look so awesome and you sound wicked smart. And the VP gets 600M budget for AWS and 600 developers, SREs, DevOps, PMO. It is not that things won't run, humans have perverse incentives.
I'm certain a lot of porn/adult industry run their setup like I mentioned with a Romanian dude running the entire infrastructure for $15K - $20K.
thelastgallon
Engineers should understand requirements, risks, tradeoffs, and potential gains. New technology may be right for that. Novel approaches may be right for that. "Novel" or "New" are only proxies and they're weak.
For example, I may think "New" means untested, but is that true? What if a new project has Jepsen testing, a fuzzing suite, massive compute running tons of oracle tests, etc? I should just say "Choose well tested" instead of "Choose old" - lots of old software is very poorly tested.
Maybe I think that "Old" implies better documentation, but does it? Lots of older projects have insane cruft and weird edge cases that are undocumented and accumulated over years.
Why do we need a metaphor? Why is "innovation token" helpful?
If you're incapable of evaluating a technology in terms of these properties, you aren't a serious developer and "boring" will not save you.
Sit down, write our your requirements, determine candidate solutions, and choose them based on their fit. "Boring" means nothing, it's a vague proxy term. "Well tsted", "performant for our use case", "developers know it", etc mean something.
> MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.
Literally every one of these has caused hilarious and disastrous failures for me in my career. But yep, boring.
> If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens.
What if you know NodeJS really well? Or MongoDb? What if you have empirical, verifiable reasons for why they fit better?
I'm a bit tired of "simple" and "boring" and other nonsense words in this field taking up the air in the room that should be spent evaluating solutions on their actual merits.
insanitybit
On the other hand, IBM was late getting into integrated circuits. They had Solid Logic Technology, automated machinery for putting transistors into ceramic substrates to make tiny but discrete circuits. That's what powered the IBM System/360. Worked, but kept mainframe prices high and made IBM late to minicomputers.
Innovation when the problem is hard is more interesting. Look at the history of US long range bombers. The B-29 was effective, but underpowered, and had a lot of trouble getting off the ground fully loaded. So, after WWII, the next development was the B-36, which answered the question "what if we scaled up the B-29?"[1] Six propellers, and four jet engines (added late in the design cycle). Was a sky barge, but it worked, as long as no one was trying hard to shoot it down. No flyable aircraft remain. That was the boring technology approach.
After that came the B-47, which answered the question "what if we scaled up a jet fighter to bomber size?"[2] The B-47 was all jets, no props. It needed solid-fuel rocket boosters (!) to help it get off the ground, and a drag chute to slow it down on landing. It was terrible to fly; its operating speed and altitude were too near the "coffin corner" where stall and Mach buffet meet.[3] No flyable aircraft remain.
Then came the B-52. New engines. New airframe. New design. [4] That's the high-risk approach. It worked. Hundreds are still in active service. Outlasted most of its successors, the B-58 Hustler (a supersonic intercontinental bomber), the B-70, the F-111, etc.
On the commercial side, we have Boeing, whose most successful airliner is a variant of the B-737, which first flew in 1967. But that's another story.
[1] https://www.youtube.com/watch?v=vKQYG_fA2uM
[2] https://www.youtube.com/watch?v=l1-urTRxeEM
[3] https://en.wikipedia.org/wiki/Coffin_corner_%28aerodynamics%...
[4] https://www.youtube.com/watch?v=k8EURBL53_k
Animats
Using the language of the article, I’d say “push all your innovation tokens into agents” is probably a good move. This means the tech your agents work with should all be boring tech.
Another way of saying this is “use in-distribution technology”. If agents are substantially better at Rust than Zig, probably you should use Rust, even if Zig is “better”. The amount that Zig is better is going to get swamped by the amount that in-distribution agents are better.
(This is not a claim that Rust actually is better, just a hypothetical fact pattern for discussion.)
theptip
iand675
1. I once worked for a company that had a large Cassandra cluster that was primarily serving the role of a distributed append-only log. This role is as perfect a fit for Cassandra as I can imagine. When we needed a distributed database for authentication, we decided to use Cassandra for it since we had in-house expertise (it was "boring technology") and a cluster we could piggy-back on for a while. When we needed a distributed database for a key-value service, we used the same reasoning and chose Cassandra again. There were many problems with Cassandra that I won't go into, but I will say that straying too far from Cassandra's comfort zone stretched it to a breaking point. And we had some downtime and rough nights because of it. Sometimes boring technology isn't enough, you also need boring workloads for it.
2. I was an early adopter for Rust. Before 1.0, it wasn't clear to most people whether the language would mount to anything. To me, I saw a formal proof assistant being put into programmer's hands and could tell right away it had a bright future. Rust was not a boring technology back then. You might say it is now (for some use cases). Adopting shiny new technologies that measurably improve confidence is not a risk. What's foolish is getting comfortable and complacent with familiar, aka boring, technologies that are difficult to use correctly.
jason_oster
conrs
So many jobs are sold as “we’re pragmatist's” and when you show up there’s 5 devs, 50 repos and most of the work is discussing if x requirement should be a new micro service. The product is usually an web app with 10 entities and and an API.
Suppose it’s keeping people in jobs
cliche
euthymiclabs