Home on Erik Bernhardsson

Software companies buying software: a story of ecosystems and vendors

Wed, 25 Feb 2026 00:00:00 +0000

Why are new startups growing so fast? Why is the wage distribution getting larger for software engineers? Why do I love infrastructure? Is open source dead?

I think a lot of these things, and some more things, all have their roots in a big shift in how we build software. Or rather, how we buy software. Software development today is a lot about using the right vendors rather than building technology yourself.

It's hard to write code for computers, but it's even harder to write code for humans

Fri, 27 Sep 2024 00:00:00 +0000

Writing code for a computer is hard enough. You take something big and fuzzy, some large vague business outcome you want to achive. Then you break it down recursively and think about all the cases until you have clear logical statements a computer can follow. Computers are very good at following logical statements.

Predicting solar eclipses with Python

Sun, 07 Apr 2024 00:00:00 +0000

As I am en route to see my first total solar eclipse, I was curious how hard it would be to compute eclipses in Python. It turns out, ignoring some minor coordinate system head-banging, I was able to get something half-decent working in a couple of hours.

Simple sabotage for software

Wed, 13 Dec 2023 00:00:00 +0000

CIA produced a fantastic book during the peak of World War 2 called Simple Sabotage. It laid out various ways for infiltrators to ruin productivity of a company. Some of the advice is timeless, for instance the section about “General interference with Organizations and Production”:

What I have been working on: Modal

Wed, 07 Dec 2022 00:00:00 +0000

Long story short: I’m working on a super cool tool called Modal. Please check it out — it lets you run things in the cloud without having to think about infrastructure. Scaling out, scheduling, containerization, using GPUs, setting up webhooks, and all kinds of other stuff. It’s primarily meant for data teams. We aren’t quite live, but you can sign up for our waitlist.

We are still early with the cloud: why software development is overdue for a change

Wed, 19 Oct 2022 00:00:00 +0000

This is is in many respects a successor to a blog post I wrote last year about what I want from software infrastructure, but the ideas morphed in my head into something sort of wider.

The genesis

I encountered AWS in 2006 or 2007 and remember thinking that it’s crazy — why would anyone want to put their stuff in someone else’s data center? But only a couple of years later, I was running a bunch of stuff on top of AWS.

σ-driven project management: when is the optimal time to give up?

Tue, 05 Apr 2022 00:00:00 +0000

Hi! It’s your friendly project management theorician. You might remember me from blog posts such as Why software projects take longer than you think, which is a blog post I wrote a long time ago positing that software projects completion time follow a log-normal distribution.

Storm in the stratosphere: how the cloud will be reshuffled

Tue, 30 Nov 2021 00:00:00 +0000

Here’s a theory I have about cloud vendors (AWS, Azure, GCP):

Cloud vendors¹ will increasingly focus on the lowest layers in the stack: basically leasing capacity in their data centers through an API.
Other pure-software providers will build all the stuff on top of it. Databases, running code, you name it.

We currently have cloud vendors that offer end-to-end solutions from the developer experience down to the hardware:

What is the right level of specialization? For data teams and anyone else.

Fri, 23 Jul 2021 00:00:00 +0000

This isn’t as much of a blog post as an elaboration of a tweet I posted the other day:

I think this specialization of data teams into 99 different roles (data scientist, data engineer, analytics engineer, ML engineer etc) is generally a bad thing driven by the fact that tools are bad and too hard to use

Building a data team at a mid-stage startup: a short story

Wed, 07 Jul 2021 00:00:00 +0000

I guess I should really call this a parable.

The backdrop is: you have been brought in to grow a tiny data team (~~4 people) at a mid-stage startup (~~$10M annual revenue), although this story could take place at many different types of companies.

Software infrastructure 2.0: a wishlist

Mon, 19 Apr 2021 00:00:00 +0000

Software infrastructure (by which I include everything ending with *aaS, or anything remotely similar to it) is an exciting field, in particular because (despite what the neo-luddites may say) it keeps getting better every year! I love working with something that moves so quickly.

What's Erik up to?

Thu, 01 Apr 2021 00:00:00 +0000

I joined Better in early 2015 because I thought the team was crazy enough to actually change one of the largest industries in the US. For six years, I ran the tech team, hiring 300+ people, probably doing 2,000+ interviews, and according to GitHub I added 646,941 lines of code and removed 339,164. But I also got married, had two kids, bought an apartment and renovated it! From time to time, there was some intense periods of hard work.

Giving more tools to software engineers: the reorganization of the factory

Wed, 16 Dec 2020 00:00:00 +0000

It’s a popular attitude among developers to rant about our tools and how broken things are. Maybe I’m an optimistic person, because my viewpoint is the complete opposite! I had my first job as a software engineer in 1999, and in the last two decades I’ve seen software engineering changing in ways that have made us orders of magnitude more productive. Just some examples from things I’ve worked on or close to:

Developer experience as a competitive advantage

Tue, 06 Oct 2020 00:00:00 +0000

I spent a ton of time looking at different software providers, both as a CTO, and as a ~~nerd~~ “advanced” consumer who builds stuff in my spare time. In the last 10 years, there has been an order of magnitude more products that cater directly to developers, through APIs, SDKs, and tooling. I’m pretty psyched about this trend. As the cost of building software goes down, that drives up the demand for software engineers. That then drives up the demand for even more products built for software engineers. That then drives down the cost of building software even more!

Mortality statistics and Sweden's "dry tinder" effect

Wed, 23 Sep 2020 00:00:00 +0000

We live in a year of about 350,000 amateur epidemiologists and I have no desire to join that “club”. But I read something about COVID-19 deaths that I thought was interesting and wanted to see if I could replicated it through data. Basically the claim is that Sweden had an exceptionally “good” year in 2019 in terms of influenza deaths causing there to be more deaths “overdue” in 2020.

How to set compensation using commonsense principles

Mon, 08 Jun 2020 00:00:00 +0000

Compensation has always been one of the most confusing parts of management to me. Getting it right is obviously extremely important. Compensation is what drives our entire economy, and you could look at the market for labor as one gigantic resource-allocating machine in the same way as people look at the stock market as a gigantic resource-allocating machine for investments.

Never attribute to stupidity that which is adequately explained by opportunity cost

Tue, 10 Mar 2020 00:00:00 +0000

Hanlon’s razor is a classic aphorism I’m sure you have heard before: Never attribute to malice that which can be adequately explained by stupidity.

I’ve found that neither malice nor stupidity is the most common reason when you don’t understand why something is in a certain way. Instead, the root cause is probably just that they didn’t have time yet. This happens all the time at startups (maybe a bit less at big companies, for reasons I’ll get back to).

How to hire smarter than the market: a toy model

Mon, 13 Jan 2020 00:00:00 +0000

Let’s consider a toy model where you’re hiring for two things and that those are equally valuable. It’s not very important what those are, so let’s just call them “thing A” and “thing B” for now. For one set of abilities, the scatter plot looks like this:

What can startups learn from Koch Industries?

Thu, 19 Dec 2019 00:00:00 +0000

I recently finished the excellent book Kochland. This isn’t my first interest in Koch—I read The Science of Success by Charles Koch himself a couple of years ago.

Charles Koch inherited a tiny company in 1967 and turned it into one of the world’s largest ones. That’s impressive! Just a quick disclaimer just to get it out of the way. You may know the Koch brothers as the climate deniers who funded the Tea Party. I don’t understand this disconnect between being so brilliant in one field, and extremely ignorant in another. But my curiosity tells me there’s something worth learning from most notable people, despite what I may think of their opinions and Koch Industries turns out ot be a particularly interesting case study.

We're hiring at Better

Mon, 09 Dec 2019 00:00:00 +0000

Just a quick note that my team is always hiring at Better. A lot of new people have been joining the team here in NYC lately—the tech team has actually grown from 35 to 60 in just ~3 months. We are primarily looking for senior software engineers and/or engineering managers. But we would love to talk if you have less experience too! Our main tech stack is mostly TypeScript and Python.

Buffet lines are terrible, but let's try to improve them using computer simulations

Wed, 16 Oct 2019 00:00:00 +0000

My company has a buffet every Friday, and the lines grow to epic proportions when the food arrives. I’ve suspected for years that the “classic” buffet line system is a deeply flawed and inefficient method, and every time I’m stuck in the line has made me more convinced.

Miscellaneous unsolicited (and possibly biased) career advice

Thu, 26 Sep 2019 00:00:00 +0000

No one asked for this, but I’m something like ~12 years into my career and have had my fair share of mistakes and luck so I thought I’d share some.

Honestly, I feel like I’ve mostly benefitted from luck. Some of the things I did on a whim turned out to be excellent choices many years later. Some of the things were clear blind spots in hindsight. If I could give my 12 years younger self a bunch of career advice, here are some of those things.

Modeling conversion rates using Weibull and gamma distributions

Mon, 05 Aug 2019 00:00:00 +0000

This is a blog post originally featured on the Better engineering blog. If you want to link to this article or share it, please go to the original post URL! Separately, I’m sorry it’s been so long with no posts on this blog. Between kids, moving, and being a startup CTO, I’ve been busy. I have a few posts coming down the pipe though, so stay tuned…

Why software projects take longer than you think: a statistical model

Mon, 15 Apr 2019 00:00:00 +0000

Anyone who built software for a while knows that estimating how long something is going to take is hard. It’s hard to come up with an unbiased estimate of how long something will take, when fundamentally the work in itself is about solving something. One pet theory I’ve had for a really long time, is that some of this is really just a statistical artifact.

Headcount goals, feature factories, and when to hire those mythical 10x people

Thu, 21 Feb 2019 00:00:00 +0000

When I started building up a tech team for Better, I made a very conscious decision to pay at the high end to get people. I thought this made more sense: they cost a bit more money to hire, but output usually more than compensates for it. Many fellow CTOs, some went for the other side of the spectrum. This was a mystery to me, until it all made sense to me.

Data architecture vs backend architecture

Thu, 10 Jan 2019 00:00:00 +0000

A modern tech stack typically involves at least a frontend and backend but relatively quickly also grows to include a data platform. This typically grows out of the need for ad-hoc analysis and reporting but possibly evolves into a whole oil refinery of cronjobs, dashboards, bulk data copying, and much more. What generally pushes things into the data platform is (generally) that a number of things are

The hacker's guide to uncertainty estimates

Mon, 08 Oct 2018 00:00:00 +0000

It started with a tweet:

New years resolution: every plot I make during 2018 will contain uncertainty estimates
— Erik Bernhardsson (@bernhardsson) January 7, 2018

Why? Because I’ve been sitting in 100,000,000 meetings where people endlessly debate whether the monthly number of widgets is going up or down, or whether widget method X is more productive than widget method Y. For almost any graph, quantifying the uncertainty seems useful, so I started trying. A few months later:

I don't want to learn your garbage query language

Thu, 30 Aug 2018 00:00:00 +0000

This is a bit of a rant but I really don’t like software that invents its own query language. There’s a trillion different ORMs out there. Another trillion databases with their own query language. Another trillion SaaS products where the only way to query is to learn some random query DSL they made up.

Business secrets from terrible people

Thu, 16 Aug 2018 00:00:00 +0000

I get bored reading management books very easily and lately I’ve been reading about a wide range of almost arbitrary topics. One of the lenses I tend to read through is to see different management styles in different environments.

New approximate nearest neighbor benchmarks

Sun, 17 Jun 2018 00:00:00 +0000

As some of you may know, one of my side interests is approximate nearest neighbor algorithms. I’m the author of Annoy, a library with 3,500+ stars on Github as of today. It offers fast approximate search for nearest neighbors with the additional benefit that you can load data super fast from disk using mmap. I built it at Spotify to use for music recommendations where it’s still used to power millions (maybe billions) of music recommendations every day.

Missing the point about microservices: it's about testing and deploying independently

Mon, 04 Jun 2018 00:00:00 +0000

Ok, so I have to first preface this whole blog post by a few things:

I really struggle with the term microservices. I can’t put my finger on exactly why. Maybe because the term is hopelessly ill-defined, maybe because it’s gotten picked up by the hype train. Whatever. But I have to stick to some type of terminology so let’s just roll with it.
This blog post might be mildly controversial, but I’m throwing it out there because I’ve had this itchy feeling for so long and I can’t get rid of it. I respect it if you want to disagree vehemently, and maybe there’s something both of us can learn.
I have a weird story. My first “real” company, Spotify, used a service-oriented architecture from scratch. I also spent some time at Google which used a service-oriented architecture. So basically since 2006 I’ve been continuously working in what people now call a “microservice architecture”. It didn’t even occur to me that some people might want to build things as monoliths. So I guess I’m coming at it from a different direction than many other. Either way, there were particular non-standard reasons why Spotify and Google had to do this that I’ll get back to later.

Let’s start by talking about iteration speed!

Interviewing is a noisy prediction problem

Wed, 02 May 2018 00:00:00 +0000

I have done roughly 2,000 interviews in my life. When I started recruiting, I had so much confidence in my ability to assess people. Let me just throw a couple of algorithm questions at a candidate and then I’ll tell you if they are good or not!

Waiting time, load factor, and queueing theory: why you need to cut your systems a bit of slack

Tue, 27 Mar 2018 00:00:00 +0000

I’ve been reading up on operations research lately, including queueing theory. It started out as a way to understand the very complex mortgage process (I work at a mortgage startup) but it’s turned into my little hammer and now I see nails everywhere.

Lessons from content marketing myself (aka blogging) for five years

Wed, 07 Mar 2018 00:00:00 +0000

I started writing this blog in late 2012, partly because I felt like it would help me improve my English and my writing skills, partly because I kept having a lot of random ideas in my head and I wanted to write them down somewhere. I honestly never cared too much about finding a particular niche, I just wanted to write down stuff that I found interesting. I set up a Wordpress blog on my crappy Swedish virtual private server.

New benchmarks for approximate nearest neighbors

Thu, 15 Feb 2018 00:00:00 +0000

UPDATE(2018-06-17): There are is a later blog post with newer benchmarks!

One of my super nerdy interests include approximate algorithms for nearest neighbors in high-dimensional spaces. The problem is simple. You have say 1M points in some high-dimensional space. Now given a query point, can you find the nearest points out of the 1M set? Doing this fast turns out to be tricky.

I'm looking for data engineers

Sun, 28 Jan 2018 00:00:00 +0000

I’m interrupting the regular programming for a quick announcement: we’re looking for data engineers at Better. You would be the first one to join and would work a lot directly with me.

Some fun things you could work on (these are all projects I’m working on right now):

Books I consumed in 2017

Wed, 17 Jan 2018 00:00:00 +0000

Turns out having a toddler isn’t super compatible with reading. I used to read ~100 books/year as a teenager, but it has slowly deteriorated to maybe 20-30 books, at most. And I don’t even finish all of them because life is too short! Some books are just not that interesting. So what were some of the books worth mentioning?

Plotting author statistics for Git repos using Git of Theseus

Wed, 03 Jan 2018 00:00:00 +0000

I spent a few days during the holidays fixing up a bunch of semi-dormant open source projects and I have a couple of blog posts in the pipeline about various updates. First up, I made a number of fixes to Git of Theseus which is a tool (written in Python) that generates statistics about Git repositories. I’ve written about it previously on this blog. The name is a horrible pun (I’m a dad!) on Ship of Theseus which is a philosophical thought experiment about what happens if you replace every single part of a boat — is it still the same boat ⁉️ 🤔

Toxic meeting culture

Fri, 29 Dec 2017 00:00:00 +0000

I spent six years at a company that went from 50 people to 1500 and one contributing factor leading to my departure was that I went from a “maker” to a person stuck in meetings every day. It wasn’t that I wanted to do that, but everyone else kept dragging me into meetings.

Learning from users faster using machine learning

Tue, 12 Dec 2017 00:00:00 +0000

I had an interesting idea a few weeks ago, best explained through an example. Let’s say you’re running an e-commerce site (I kind of do) and you want to optimize the number of purchases.

Let’s also say we try to learn as much as we can from users, both using A/B tests but also using just basic slicing and dicing of the data. We are looking at how many people convert (buy our widgets) but a constant problem is there’s just too much uncertainty.

Annoy 1.10 released, with Hamming distance and Windows support

Sun, 26 Nov 2017 00:00:00 +0000

I’ve been a bit bad at posting things with a regular cadence lately, partly because I’m trying to adjust to having a toddler, partly because the hunt for clicks has caused such a high bar for me that I feel like I have to post something Pulitzer-worthy. But things are always cooking, so let’s break this pattern with a quick notice on something I’ve been working on!

Why conversion matters: a toy model

Mon, 30 Oct 2017 00:00:00 +0000

There are often close relationships between top level business metrics. For instance, it’s well known that retention has a super strong impact on the valuation of a subscription business. Or that the % of occupied seats is super important for an airline. A fun little toy model that I can up with generates a curious relationship between conversion rates and revenue.

On the Equifax breach and how to really prevent identity theft

Tue, 26 Sep 2017 00:00:00 +0000

A funny thing about being a foreigner is how you realize people take broken things for granted. I’m going to go out on a limb here claiming that the US has a pretty dumb banking system. I could talk about it all day, but right now I want to focus on a very particular piece of it: how to verify your identity online.

The number of letters in the word for each number

Wed, 06 Sep 2017 00:00:00 +0000

Just for fun, I generated these graphs of the number of letters in the word for each number. I really spent about 10 minutes on this (ok…possibly also another 40 minutes tweaking the plots):

The software engineering rule of 3

Tue, 29 Aug 2017 00:00:00 +0000

Here’s a ~~dumb~~ extremely accurate rule I’m postulating* for software engineering projects: you need at least 3 examples before you solve the right problem.

This is what I’ve noticed:

Don’t factor out shared code between two classes. Wait until you have at least three.
The two first attempts to solve a problem will fail because you misunderstood the problem. The third time it will work.
Any attempt at being smart earlier will end up overfitting to coincidental patterns.

(Note that #1 and #2 are actually pretty different implications. But let’s get back to that later.)

Machine, Platform, Crowd

Sat, 19 Aug 2017 00:00:00 +0000

I just bought Machine, Platform, Crowd: Harnessing Our Digital Future and discovered that it mentions my blog – in particular the post When machine learning matters.

Ok, I lied a little bit. I didn’t discover it serendipitously. Someone actually emailed me saying I was mentioned, and so I ordered the book for same-day delivery. But I was seriously planning to read the book anyway – having read both The Second Machine Age and Rage Against the Machine – they are great books and I’m not being biased.

Google diversity memo, global warming, Pascal's wager, and other stuff

Mon, 14 Aug 2017 00:00:00 +0000

There’s about 765 million blog posts about the diversity “memo” that leaked out of Google a couple of weeks ago. I think the case for any biological difference is pretty weak, and it bothers me when people refer to an “interest gap” as anything else than caused by the environment. Maybe because I have a daughter, maybe because I have too many female friends who told me stories how they were held back or discriminated against.

Fun with trigonometry: the world's most twisted coastline

Wed, 12 Jul 2017 00:00:00 +0000

I just spent a few days in Italy, on the Ligurian coast. Even though we were on the west side of Italy, the Mediterranean sea was to the east, because the house was situated on a long bay. But zooming in even more, there were parts of the coast that were even more twisted – to the point where it had turned a full 360 degress so you ended up having the sea to the west again.

Optimizing for iteration speed

Thu, 06 Jul 2017 00:00:00 +0000

I’ve written before about the importance of iterating quickly but I didn’t necessarily talk about some concrete things you can do. When I’ve built up the tech team at Better, I’ve intentionally optimized for fast iteration speed above almost everything else. What are some ways we did that?

Conversion rates – you are (most likely) computing them wrong

Tue, 23 May 2017 00:00:00 +0000

How hard can it be to compute conversion rate? Take the total number of users that converted and divide them with the total number of users. Done. Except… it’s a lot more complicated when you have any sort of significant time lag.

The mathematical principles of management

Sun, 09 Apr 2017 00:00:00 +0000

I’ve read about 100 management books by now but if there’s something that always bothered me it’s the lack of first principles thinking. Basically it’s a ton of heuristics. And heuristics are great, but when you present heuristics as true objectives, it kind of clouds the underlying objectives (and you end up with weird proxy cults like the Agile movement 👹 – not that I disagree with it, I just wish they could derive it from a more systematic understanding of project management).

The eigenvector of "Why we moved from language X to language Y"

Wed, 15 Mar 2017 00:00:00 +0000

I was reading yet another blog post titled “Why our team moved from <language X> to <language Y>” (I forgot which one) and I started wondering if you can generalize it a bit. Is it possible to generate a N * N contingency table of moving from language X to language Y?

Why I went into the mortgage industry

Fri, 17 Feb 2017 00:00:00 +0000

I just realized last Thursday that I have spent two full years at Better, incidentally on the same day as we announced a $15M round led by Kleiner Perkins. So it was a good point to reflect a bit and think back – what the F led me to abandon my role managing the machine learning team at Spotify? To join some random startup in the world’s most boring industry? So here’s my justification why I love being where I am:

Language pitch

Wed, 01 Feb 2017 00:00:00 +0000

Here’s a fun analysis that I did of the pitch (aka. frequency) of various languages. Certain languages are simply pronounced with lower or higher pitch. Whether this is a feature of the language or more a cultural thing is a good question, but there are some substantial differences between languages.

Functional programming is the libertarianism of software engineering

Tue, 10 Jan 2017 00:00:00 +0000

This is a pretty dumb post, in which I argue that functional programming has a lot of the bad parts of libertarianism and a lot of the good parts:

Both ideologies strive to eliminate [the] state. (ok, dumb dad joke)
Both ideologies are driven by a set of dogmatic axioms rather than a practical goal:
Libertarianism wants to reduce the government because any involvement distorts free markets. I always struggled to see what the underlying objective function is (it doesn’t seem to be maximization of people’s utility). 🤔
Functional programming wants to reduce side effects and make everything pure, often by enforcing onerous type systems. But why? Again I don’t see an ultimate objective here. IMO it should start from the principle that the goal of a programming language should be to make the programmers as productive as possible. For instance, the little research that exists has shown that most bugs have little to with typing and I’d expect something similar to apply to mutable state. In fact the largest class seems to be poor error handling (ok, typing isn’t necessarily related to FP, but in practice I find that strong typing and FP have highly overlapping fan clubs).
Both camps invoke obscure cases in history as a proof of success: libertarianists (more so anarchists I guess) often talks about Spain during the civil war, Somalia, or sometimes Singapore. Haskell acolytes are very eager to bring up Facebook’s spam filtering.
YET – and this is the surprising part imho – both ideologies are ~90% correct (source: my opinion). Which really surprises me given that they start from a (imo) arbitrary set of axioms.
Even if you are a die hard bolshevik, you benefit from an understanding of how interventions distort markets, how incentives matter, and how entrepreneurship is the driver of progress.
Even if you are coding in Visual Basic, you can level up your skills by learning FP: making functions pure when needed, avoid state, avoid reassign variables, avoid mutable data structures, write pipelines of data transformations, and all that jazz that FP has taught us to cherish.

The half-life of code & the ship of Theseus

Mon, 05 Dec 2016 00:00:00 +0000

As a project evolves, does the new code just add on top of the old code? Or does it replace the old code slowly over time? In order to understand this, I built a little thing to analyze Git projects, with help from the formidable GitPython project. The idea is to go back in history historical and run a git blame (making this somewhat fast was a bit nontrivial, as it turns out, but I’ll spare you the details, which involve some opportunistic caching of files, pick historical points spread out in time, use git diff to invalidate changed files, etc).

Are data sets the new server rooms?

Tue, 01 Nov 2016 00:00:00 +0000

This blog post Data sets are the new server rooms makes the point that a bunch of companies raise a ton of money to go get really proprietary awesome data as a competitive moat. Because once you have the data, you can build a better product, and no one can copy it (at least not very cheaply). Ideally you hit a virtuous cycle as well, where usage of your system once it takes of gives even more data, which makes the system even better, which attracts more users…

Pareto efficency

Tue, 25 Oct 2016 00:00:00 +0000

Pareto efficiency is a useful concept I like to think about. It often comes up when you compare items on multiple dimensions. Say you want to buy a new TV. To simplify it let’s assume you only care about two factors: price and quality. We don’t know what you are willing to pay for quality – but we know that everything else equals:

State drift

Thu, 08 Sep 2016 00:00:00 +0000

I generally haven’t written much about software architecture. People make heuristics into religion. But here is something I thought about: how to build in self-correction into systems. This has been something just vaguely sitting in my head lacking a clear conceptual definition until a whole slew of things popped up today that all had the exact same issue at its core. I’m going to refer to it as state drift lacking a better term for it.

When machine learning matters

Fri, 05 Aug 2016 00:00:00 +0000

I joined Spotify in 2008 to focus on machine learning and music recommendations. It’s easy to forget, but Spotify’s key differentiator back then was the low-latency playback. People would say that it felt like they had the music on their own hard drive. (The other key differentiator was licensing – until early 2009 Spotify basically just had all kinds of weird stuff that employees had uploaded. In 2009 after a crazy amount of negotiation the music labels agreed to try it out as an experiment. But I’m getting off topic now.)

Subway waiting math

Sat, 09 Jul 2016 00:00:00 +0000

Why does it suck to wait for things? In a previous post I analyzed a NYC subway dataset and found that at some point, quite early, it’s worth just giving up.

This isn’t a proof that the subway doesn’t run on time – in fact it might actually proves that the subway runs really well. The numbers indicate that it’s not worth waiting after 10 minutes, but it’s a rare event and usually involves something extraordinary like a multi-hour delay. You should roughly give up after some point related to the normal train frequency, and 10 minutes is not a lot at all. Conversely if the trains ran hourly, it probably would had been worth waiting an hour or more. My analysis gave me a lot of respect for the job MTA is doing.

Approximate nearest news

Thu, 02 Jun 2016 00:00:00 +0000

As you may know, one of my (very geeky) interests is Approximate nearest neigbor methods, and I’m the author of a Python package called Annoy.

I’ve also built a benchmark suite called ann-benchmarks to compare different packages. Annoy was the world’s fastest package for a few months, but two things happened.

What is your motivation?

Tue, 24 May 2016 00:00:00 +0000

I’ve been trying to learn Clojure. I keep telling people I meet that I really want to learn Clojure, but still every night I can’t get myself to spend time with it. It’s unclear if I really want to learn Clojure or just want to have learned Clojure?

Dollar cost averaging

Tue, 26 Apr 2016 00:00:00 +0000

(I accidentally published an unfinished draft of this post a few days ago – sorry about that).

There’s a lot of sources preaching the benefits of dollar cost averaging, or the practice of investing a fixed amount of money regularly. The alleged benefit is that when the price goes up, well, then your stake is worth more, but if the price goes down, then you get more shares for the same amount of money. According to Wikipedia, it “minimises downside risk”, about.com says it “drastically reduces market risk”, and an article on Nasdaq.com claims that it’s a “smart investment strategy”.

Why organizations fail

Mon, 18 Apr 2016 00:00:00 +0000

One of my favorite business hobbies is to reduce some nasty decision down to its absolute core objective, decide the most basic strategy, and then add more and more modifications as you have to confront the complexity of reality (yes I have very lame hobbies thanks I know).

NYC subway math

Mon, 04 Apr 2016 00:00:00 +0000

Apparently MTA (the company running the NYC subway) has a real-time API. My fascination for the subway takes autistic proportions and so obviously I had to analyze some of the data. The documentation is somewhat terrible, but here’s some relevant code for how to use the API:

Exploding offers are bullshit

Wed, 16 Mar 2016 00:00:00 +0000

I do a lot of recruiting and have given maybe 50 offers in my career. Although many companies do, I never put a deadline on any of them. Unfortunately, I’ve often ended up competing with other companies who do, and I feel really bad that this usually tricks younger developers into signing offers. On numerous occasions, I’ve gotten an email halfway through the interview process

Meta-blogging

Sat, 12 Mar 2016 00:00:00 +0000

(This is not a very relevant/useful post for regular readers – feel free to skip. I thought I would share it so people can find it on Google.)

My blog blew up twice in a week earlier this year when I landed on Hacker News. The first time I was asleep so I didn’t notice that the site went down. The second time I did notice, and scrambled to reconfigure Apache & MySQL to handle the load.

Iterate or die

Wed, 02 Mar 2016 00:00:00 +0000

Here’s a conclusion I’ve made building consumer products for many years: the speed at which a company innovates is limited by its iteration speed.

I don’t even mean throughput here. I just mean the cycle time. Invoking Little’s law this is also related to the total inventory of features not being deployed yet.

My issue with GPU-accelerated deep learning

Wed, 03 Feb 2016 00:00:00 +0000

I’ve been spending several hundred bucks renting GPU instances on AWS over the last year. The speedup from a GPU is awesome and hard to deny. GPUs have taken over the field. Maybe following the footsteps of Bitcoin mining there’s some research on using FPGA (I know very little about this).

Some more font links

Mon, 25 Jan 2016 00:00:00 +0000

My blog post about fonts generated lots of traffic – it landed on Hacker News, took down my site while I was sleeping, and then obviously vanished from HN before I woke up. But it also got retweeted by a ton of people.

Analyzing 50k fonts using deep neural networks

Thu, 21 Jan 2016 00:00:00 +0000

For some reason I decided one night I wanted to get a bunch of fonts. A lot of them. An hour later I had a bunch of scrapy scripts pulling down fonts and a few days later I had more than 50k fonts on my computer.

I believe in the 10x engineer, but...

Fri, 08 Jan 2016 00:00:00 +0000

The easiest way to be a 10x engineer is to make 10 other engineers 2x more efficient. Someone can be a 10x engineer if they do nothing for 364 days then convinces the team to change programming language to a 2x more productive language.
A motivated 10x engineer in one team could be a demotivated 0.5x engineer in another team (and vice versa).
A average 1x engineer could easily become a 5x engineer if surrounded by 10x engineers. Engagement and work ethics is contagious.
The cynical reason why 10x engineers aren’t paid 10x more salary is that there is no way for the new employer to know. There is no “10x badge”.
…but also, a 10x engineer can go to a new company and become an 1x engineer because of bad focus / bad engagement / tech stack mismatch.
So unfortunately there’s less economic rationality for companies to pay 10x salaries to 10x engineers (contrary to what Google or Netflix says)
There’s no such thing as a 10x engineer spending time on something that never ends up delivering business value. If something doesn’t deliver business value, it’s 0x.
If you build something that the average engineer would not have been able to build, no matter how much time, that can make you 100x or 1000x, or ∞x. Quoting Alexander Scott: There is no number of ordinary eight-year-olds who, when organized into a team, will become smart enough to beat a grandmaster in chess.
Most of the 10x factor is most likely explained by team and company factors (process, tech stack, etc) and applies to everyone in the team/company. Intra-team variation is thus much smaller than 10x (even controlling for the fact that companies tend to attract people of equal caliber). Nature vs nurture…
I’ve never met the legendary “10x jerk”. Anecdotally the outperforming engineers are generally nice and humble.
Don’t get hung up on the exact numbers here, it’s just for illustration purposes. I.e. someone introduced a bug in the trading system of Knight Capital that made them lose $465M in 30 minutes. Did that make it a -1,000,000x engineer? (and btw it had more to do with company culture). The numbers aren’t meant to be taken literally.

Books I read in 2015

Fri, 01 Jan 2016 00:00:00 +0000

Early last year when I left Spotify I decided to do more reading. I was planning to read at least one book per week and in particular I wanted to brush up on management, economics, and technology. 2015 was also a year of exclusively non-fiction, which is a pretty drastic shift, since I grew up reading fiction compulsively for 20 years.

More MCMC – Analyzing a small dataset with 1-5 ratings

Sat, 05 Dec 2015 00:00:00 +0000

I’ve been obsessed with how to iterate quickly based on small scale feedback lately. One awesome website I encountered is Usability Hub which lets you run 5 second tests. Users see your site for 5 seconds and you can ask them free-form questions afterwards. The nice thing is you don’t even have to build the site – just upload a static png/jpg and collect data.

There is no magic trick

Sat, 28 Nov 2015 00:00:00 +0000

(Warning: super speculative, feel free to ignore)

As Yogi Berra said, “It’s tough to make predictions, especially about the future”. Unfortunately predicting is hard, and unsurprisingly people look for the Magic Trick™ that can resolve all the uncertainty. Whether it’s recruiting, investing, system design, finding your soulmate, or anything else, there’s always an alleged shortcut.

Installing TensorFlow on AWS

Thu, 12 Nov 2015 00:00:00 +0000

Curious about Google’s newly released TensorFlow? I don’t have a beefy GPU machine, so I spent some time getting it to run on EC2. The steps on how to reproduce it are pretty brutal and I wouldn’t recommend going through it unless you want to waste five hours of your live.

Looking for smart people

Wed, 04 Nov 2015 00:00:00 +0000

I haven’t mentioned what I’m currently up to. Earlier this year I left Spotify to join a small startup called Better. We’re going after one of the biggest industries in the world that also turns out to be completely broken. The mortgage industry might not be the #1 industry you pictured yourself in, but it’s an enormous opportunity to fix a series of real consumer problems and join a company that I predict will be huge.

MCMC for marketing data

Sat, 31 Oct 2015 00:00:00 +0000

The other day I was looking at marketing spend broken down by channel and wanted to compute some simple uncertainty estimates. I have data like this:

<th>
  Total spend
</th>

<th>
  Transactions
</th>

<td>
  2292.04
</td>

<td>
  9
</td>

<td>
  1276.85
</td>

<td>
  2
</td>

<td>
  139.59
</td>

<td>
  3
</td>

<td>
  954.98
</td>

<td>
  5
</td>


Channel A
Channel B
Channel C
Channel D

Of course, it’s easy to compute the cost per transaction, but how do you produce uncertainty estimates? Turns out to be somewhat nontrivial. I don’t even think it’s possible to do a t-test, which is kind of interesting in itself.

Interview with a Data Scientist: Erik Bernhardsson

Wed, 28 Oct 2015 00:00:00 +0000

I was featured in Peadar Coyle’s interview series interviewing various “data scientists” – which is kind of arguable since (a) all the other ppl in that series are much cooler than me (b) I’m not really a data scientist. Anyway, reposting the full interview:

Nearest neighbors and vector models – epilogue – curse of dimensionality

Tue, 20 Oct 2015 00:00:00 +0000

This is another post based on my talk at NYC Machine Learning. The previous two parts covered most of the interesting parts, but there are still some topics left to be discussed. To go back and read the meaty stuff, check out

Nearest neighbors and vector models – part 2 – algorithms and data structures

Thu, 01 Oct 2015 00:00:00 +0000

This is a blog post rewritten from a presentation at NYC Machine Learning on Sep 17. It covers a library called Annoy that I have built that helps you do nearest neighbor queries in high dimensional spaces. In the first part, I went through some examples of why vector models are useful. In the second part I will be explaining the data structures and algorithms that Annoy uses to do approximate nearest neighbor queries.

Nearest neighbor methods and vector models – part 1

Thu, 24 Sep 2015 00:00:00 +0000

This is a blog post rewritten from a presentation at NYC Machine Learning last week. It covers a library called Annoy that I have built that helps you do (approximate) nearest neighbor queries in high dimensional spaces. I will be splitting it into several parts. This first talks about vector models, how to measure similarity, and why nearest neighbor queries are useful.

Presentations about Spotify music recommendations

Tue, 22 Sep 2015 00:00:00 +0000

A couple of people in my old team have been around talking about how Spotify does music recommendations and put together some quite good presentations.

First one is Neville Li’s presentation about Scala Data Pipelines @ Spotify:

Antipodes

Tue, 08 Sep 2015 00:00:00 +0000

I was playing around with D3 last night and built a silly visualization of antipodes and how our intuitive understanding of the world sometimes doesn’t make sense. Check out the visualization at bl.ocks.org!

Basically the idea is if you fly from Beijing to Buenos Aires then you can have a layover at any point of the Earth’s surface and it won’t make the trip longer.

Software Engineers and Automation

Sun, 16 Aug 2015 00:00:00 +0000

Every once in a while when talking to smart people the topic of automation comes up. Technology has made lots of occupations redundant, so what’s next?

Switchboard operator, a long time ago

What about software engineers? Every year technology replaces parts of what they do. Eventually surely everything must be replaced? I just ran into another one of these arguments: Software Engineers will be obsolete by 2060.

coin2dice

Fri, 24 Jul 2015 00:00:00 +0000

Here’s a problem that I used to give to candidates. I stopped using it seriously a long time ago since I don’t believe in puzzles, but I think it’s kind of fun.

Let’s say you have a function that simulates a random coin flip. It returns “H” or “T”. This is the only random generator available. How can write a new function that simulates a random dice roll (1…6)?
Is there any method that guarantees that the second function returns in finite time?
Let’s say you want to do this $$ n $$ times where $$ n \to \infty $$ . What’s the most efficient way to do it? Efficient in terms of using the fewest amount of coin flips.

The first part is old, I think. The second and third part are follow up questions that I came up with.

Benchmark of Approximate Nearest Neighbor libraries

Sat, 04 Jul 2015 00:00:00 +0000

Annoy is a library written by me that supports fast approximate nearest neighbor queries. Say you have a high (1-1000) dimensional space with points in it, and you want to find the nearest neighbors to some point. Annoy gives you a way to do this very quickly. It could be points on a map, but also word vectors in a latent semantic representation or latent item vectors in collaborative filtering.

More Luigi alternatives

Thu, 02 Jul 2015 00:00:00 +0000

The workflow engine battle has intensified with some more interesting entries lately! Here are a couple I encountered in the last few days. I love that at least two of them are direct references to Luigi!

3D in D3

Sun, 21 Jun 2015 00:00:00 +0000

I have spent some time lately with D3. It’s a lot of fun to build interactive graphs. See for instance this demo (will provide a longer writeup soon).

D3 doesn’t have support for 3D but you can do projections into 2D pretty easily. It’s just old school computer graphics. I ended up adding an animated background to this blog based on an experiment. The math is simple.

The hardest challenge about becoming a manager

Fri, 05 Jun 2015 00:00:00 +0000

Note: this post is full of pseudo-psychology and highly speculative content. Like most fun stuff!

I became a manager back in 2009. Being a developer is fun. You have this very tangible way to measure yourself. Did I deploy something today? How much code did I write today? Did I solve some really cool machine learning problem on paper?

The lane next to you is more likely to be slower than yours

Thu, 28 May 2015 00:00:00 +0000

Saw this link on Hacker News the other day: The Highway Lane Next to Yours Isn’t Really Moving Any Faster

The article describes a phenomenon unique to traffic where cars spread out when they go fast and get more compact when they go slow. That’s supposedly the explanation.

Better precision and faster index building in Annoy

Tue, 26 May 2015 00:00:00 +0000

Sometimes you have these awesome insights. A few days ago I got an idea for how to improve index building in Annoy.

For anyone who isn’t acquainted with Annoy – it’s a C++ library with Python bindings that provides fast high-dimensional nearest neighbor search.

Annoy – now without Boost dependencies and with Python 3 Support

Sun, 03 May 2015 00:00:00 +0000

Annoy is a C++/Python package I built for fast approximate nearest neighbor search in high dimensional spaces. Spotify uses it a lot to find similar items. First, matrix factorization gives a low dimensional representation of each item (artist/album/track/user) so that every item is a k-dimensional vector, where k is typically 40-100. This is then loaded into an Annoy index for a number of things: fast similar items, personal music recommendations, etc.

Ping the world

Sun, 26 Apr 2015 00:00:00 +0000

I just pinged a few million random IP addresses from my apartment in NYC. Here’s the result:

Some notes:

What’s going on with Sweden? Too much torrenting?
Ireland is likewise super slow, but not Northern Ireland
Eastern Ukraine is also super slow, maybe not surprising given current events.
Toronto seems screwed too, as well as part of NH and western PA.
Russia has fast internet.

The world:

Black Box Machine Learning in the Cloud

Wed, 22 Apr 2015 00:00:00 +0000

There’s a bunch of companies working on machine learning as a service. Some old companies like Google, but now also Amazon and Microsoft.

Then there’s a ton of startups: PredictionIO ($2.7M funding), BigML ($1.6M funding), Clarifai, etc, etc. Here’s a nice map from Bloomberg showing some of the landscape.

It's called Berkson's paradox!

Thu, 09 Apr 2015 00:00:00 +0000

As noted by multiple tweets, my previous post describes a phenomenon denoted Berkson’s paradox.

Here’s another example: Why Are Handsome Men Such Jerks?

Norvig's claim that programming competitions correlate negatively with being good on the job

Tue, 07 Apr 2015 00:00:00 +0000

I saw a bunch of tweets over the weekend about Peter Norvig claiming there’s a negative correlation between being good at programming competitions and being good at the job. There were some decent Hacker News comments on it.

Pinterest open sources Pinball

Sat, 14 Mar 2015 00:00:00 +0000

Pinterest just open sourced Pinball which seems like an interesting Luigi alternative. There’s two blog posts: Pinball: Building workflow management (from 2014) and Open-sourcing Pinball (from this week). The author has a comment in the comments thread on Hacker News:

The relationship between commit size and commit message size

Thu, 26 Feb 2015 00:00:00 +0000

Wow I guess it was more than a year ago that I tweeted this. Crazy how time flies by. Anyway, here’s my rationale:

When I update one line of code I feel like I have to put in a long explanation about its side effects, why it’s fully backwards compatible, and why it fixes some issue #xyz.
When I refactor 500 lines of code, I get too lazy to write anything sensible, so I just put “refactoring FooBarController”. Note: don’t do at home!

I decided to plot the relationship for Luigi: {% include 2015-02-26-the-relationship-between-commit-size-and-commit-message-size.html %}

My favorite management failures

Sun, 22 Feb 2015 00:00:00 +0000

For most people straight out of school, work life is a bit of a culture shock. For me it was an awesome experience, but a lot of the constraints were different and I had to learn to optimize for different things. It wasn’t necessarily the technology that I struggled with. The hardest part was how to manage my own projects and my time, as well as how to grow and make impact as an engineer. I’ve listed some of my biggest mistakes, which are also mistakes I see other (mostly junior) engineers make.

Leaving Spotify

Wed, 11 Feb 2015 00:00:00 +0000

Febrary 6 was my last day at Spotify. In total I spent more than six years at Spotify and it was an amazing experience.

I joined Spotify in Stockholm in 2008, mainly because a bunch of friends from programming competitions had joined already. Their goal to change music consumption seemed ridiculous at that point, but six years later I think it’s safe to say they actually succeeded.

Scala Data Pipelines for Music Recommendations

Tue, 13 Jan 2015 00:00:00 +0000

Chris Johnson‘s presentation from Data Day Texas:

Everything I learned about technical debt

Tue, 30 Dec 2014 00:00:00 +0000

I just made it to Sweden suffering from jet lag induced insomnia, but this blog post will not cover that. Instead, I will talk a little bit about technical debt.

The concept of technical debt always resonated with me, partly because I always like the analogy with “real” debt. If you take the analogy really far, there are some curious implications. I always like to think of the “interest rate” of software development. Debt is really just borrowing from the future, with some interest rate. You are getting a free lunch right now, but you need to pay back 1.2 free lunches in a few months. That’s the interest rate. In a software project the equivalent could be to pick a database that will have scalability issues later, or to make all member variables of some class public. You are doing it because it makes it easier to do things now but you will have to pay the cost of that later.

I already found the best gifs

Sun, 28 Dec 2014 00:00:00 +0000

Just search for “hackers gif“.

There you go. Fun for your work emails for the next 500 years. From the awesome movie Hackers. That movie together with The Warriors convinced me that I wanted to live in NYC when I was like… 14 years old.

A brief history of Hadoop at Spotify

Sat, 20 Dec 2014 00:00:00 +0000

I was talking with some data engineers at Spotify and had a moment of nostalgia.

2008

I was writing my master’s thesis at Spotify and had to run a Hadoop job to extract some data from the logs. Every time I started running the job, I kept hearing this subtle noise. I kept noticing the correlation for a few days but I was too intimidated to ask. Finally people starting cursing that their machines had gotten really slow lately and I realized we were running Hadoop on the developer’s desktop machines. No one had told me. I think back then we had only GB’s of log data. I remember running less on the log and I would recognize half the usernames because they were my friends.

Luigi Presentation @ NYC Data Science, Dec 16, 2014

Wed, 17 Dec 2014 00:00:00 +0000

Luigi talk tomorrow

Tue, 16 Dec 2014 00:00:00 +0000

At NYC Data Science meetup! Unfortunately the space is full but the talk will be livestreamed – check out the meetup web page for a link tomorrow.

Deep learning for… Go

Thu, 11 Dec 2014 00:00:00 +0000

This is the last post about deep learning for chess/go/whatever. But this really cool paper by Christopher Clark and Amos Storkey was forwarded to me by Michael Eickenberg. It’s about using convolutional neural networks to play Go. The authors of the paper do a much better job than I would ever have done of modeling move prediction in Go and show that their model beat certain Go engines.

Deep learning for… chess (addendum)

Mon, 08 Dec 2014 00:00:00 +0000

My previous blog post about deep learning for chess blew up and made it to Hacker News and a couple of other places. One pretty amazing thing was that the Github repo got 150 stars overnight. There was also lots of comments on the Hacker News post that I thought were really interesting. (See this skeptical comment for instance).

Deep learning for... chess

Sat, 29 Nov 2014 00:00:00 +0000

I’ve been meaning to learn Theano for a while and I’ve also wanted to build a chess AI at some point. So why not combine the two? That’s what I thought, and I ended up spending way too much time on it. I actually built most of this back in September but not until Thanksgiving did I have the time to write a blog post about it.

Optimizing things: everything is a proxy for a proxy for a proxy

Sat, 22 Nov 2014 00:00:00 +0000

Say you build a machine learning model, like a movie recommender system. You need to optimize for something. You have 1-5 stars as ratings so let’s optimize for mean squared error. Great.

Then let’s say you build a new model. It has even lower mean squared error. You deploy it. This model turns out to give a lower mean squared error. You roll it out to users and the metrics are tanking. Crap! Ok so maybe mean squared error isn’t the right thing to optimize for.

Luigi conquering the world

Sat, 15 Nov 2014 00:00:00 +0000

I keep forgetting to buy a costume for Halloween every year, so this year I prepared and got myself a Luigi costume a month in advance. Only to realize I was going to be out of town the whole weekend. If anyone wants a Luigi costume, let me know!

Annoying blog post

Tue, 11 Nov 2014 00:00:00 +0000

I spent a couple of hours this weekend going through some pull requests and issues to Annoy, which is an open source C++/Python library for Approximate Nearest Neighbor search.

I set up Travis-CI integration and spent some time on one of the issues that multiple people had reported. At the end of the day, it turns out the issue was actually caused by a bug in GCC 4.8. Some crazy compiler optimization introduced between 4.6 and 4.8 caused this loop to be removed:

The Filter Bubble is Silly and you Can't Guess What Happened Next

Fri, 10 Oct 2014 00:00:00 +0000

I’m at RecSys 2014, meeting a lot of people and hanging out at talks. Some of the discussions here was about the filter bubble which prompted me to formalize my own thoughts.

I firmly believe that it’s the role of a system to respect the user’s intent. Any sensible system will optimize for user’s long-term happiness by providing info back to the user that s/he finds useful. This holds true as long as a system isn’t (a) stupid and recommends the wrong content (b) trying to push its own agenda, that may or may not be hidden.

Detecting corporate fraud using Benford's law

Tue, 07 Oct 2014 00:00:00 +0000

Note: This is a silly application. Don’t take anything seriously.

Benford’s law describes a phenomenon where numbers in any data series will exhibit patterns in their first digit. For instance, if you took a list of the 1,000 longest rivers of Mongolia, or the average daily calorie consumption of mammals, or the wealth distribution of German soccer players, you will on average see that these numbers start with “1” about 30% of the time. I won’t attempt at proving this, but essentially it’s a result of scale invariance. It doesn’t apply to all numerical series, like IQ or shoe size, but this pattern turns out to pop up in a lot of places.

Running Theano on EC2

Tue, 19 Aug 2014 00:00:00 +0000

Inspired by Sander Dieleman’s internship at Spotify, I’ve been playing around with deep learning using Theano. Theano is this Python package that lets you define symbolic expressions (cool), does automatic differentiation (really cool), and compiles it down into bytecode to run on a CPU/GPU (super cool). It’s built by Yoshua Bengio’s deep learning team up in Montreal.

In defense of false positives (why you can't fail with A/B tests)

Wed, 30 Jul 2014 00:00:00 +0000

Many years ago, I used to think that A/B tests were foolproof and all you need to do is compare the metrics for the two groups. The group with the highest conversion rate wins, right?

Recurrent Neural Networks for Collaborative Filtering

Sat, 28 Jun 2014 00:00:00 +0000

I’ve been spending quite some time lately playing around with RNN’s for collaborative filtering. RNN’s are models that predict a sequence of something. The beauty is that this something can be anything really – as long as you can design an output gate with a proper loss function, you can model essentially anything.

Where do locals go in NYC?

Tue, 17 Jun 2014 00:00:00 +0000

One obvious thing to anyone living in NYC is how tourists cluster in certain areas. I was curious about the larger patterns around this, so I spent some time looking at data. The thing I wanted to understand is: what areas are dominated by tourists? Or conversely, what areas are dominated by locals?

How to build up a data team (everything I ever learned about recruiting)

Sun, 08 Jun 2014 00:00:00 +0000

During my time at Spotify, I’ve reviewed thousands of resumes and interviewed hundreds of people. Lots of them were rejected but lots of them also got offers. Finally, I’ve also had my share of offers rejected by the candidate.

The power of ensembles

Thu, 24 Apr 2014 00:00:00 +0000

From my presentation at MLConf, one of the points I think is worth stressing again is how extremely well combining different algorithms works.

In this case, we’re training machine learning algorithms on different data sets (playlists, play counts, sessions) and different objectives (least squares, max likelihood). Then we combine all the models using gradient boosted decision trees training on a smaller but higher quality data set. Finally, we validate on a third data set, in this case looking at recall for a ground truth data set of related artists.

MLConf 2014

Sat, 12 Apr 2014 00:00:00 +0000

Just spent a day at MLConf where I was talking about how we do music recommendations. There was a whole range of great speakers (actually almost 2/3 women which was pretty cool in itself).

Here are my slides:

Music recommendations using cover images (part 1)

Tue, 01 Apr 2014 00:00:00 +0000

Scrolling through the Discover page on Spotify the other day it occurred to me that the album is in fact a fairly strong visual proxy for what kind of content you can expect from it. I started wondering if the album cover can in fact be used for recommendations. For many obvious reasons this is a kind of ridiculous idea, but still interesting enough that I just had to explore it a bit. So, I embarked on a journey to see how far I could get in a few hours.

Luigi success

Sat, 22 Mar 2014 00:00:00 +0000

So Luigi, our open sourced workflow engine in Python, just recently passed 1,000 stars on Github, then shortly after passed mrjob as (I think) the most popular Python package to do Hadoop stuff. This is exciting!

Welcome Echo Nest!

Sat, 22 Mar 2014 00:00:00 +0000

In case you missed it, we just acquired a company called Echo Nest in Boston. These people have been obsessed with understanding music for the past 8 years since it was founded by Brian Whitman and Tristan Jehan out of MIT Medialab.

Momentum strategies

Mon, 03 Mar 2014 00:00:00 +0000

Haven’t posted anything in ages, so here’s a quick hack I threw together in Python on a Sunday night. Basically I wanted to know whether momentum strategies work well for international stock indexes. I spent a bit of time putting together a strategy that buys the stock index if the return during the previous n days was positive, otherwise doesn’t do anything. I ran this strategy for a basket of approximately 20 stock markets.

Ratio metrics

Thu, 23 Jan 2014 00:00:00 +0000

We run a ton of A/B tests at Spotify and we look at a ton of metrics. Defining metrics is a little bit of an art form. Ideally you want to define success metrics before you run a test to avoid cherry picking metrics. You also want to define a metric that has as high signal to noise ratio. And of course, most importantly, your metric should ideally correlate to high level business impact as much as possible.

Benchmarking nearest neighbor libraries in Python

Sun, 12 Jan 2014 00:00:00 +0000

Radim Rehurek has put together an excellent summary of approximate nearest neighbor libraries in Python. This is exciting, because one of the libraries he’s covering, annoy, was built by me.

After introducing the problem, he goes through the list of contestants and sticks with five remaining ones. Finally, the benchmarks pits annoy against FLANN. Although FLANN seems to have roughly 4x better performance, somewhat surprisingly, Radim concludes annoy is the “winner”. Yay!

More recommender algorithms

Fri, 20 Dec 2013 00:00:00 +0000

I wanted to share some more insight into the algorithms we use at Spotify. One matrix factorization algorithm we have used for a while assumes that we have user vectors $$ bf{a}_u $$ and item vectors $$ bf{b}_i $$ . The next track $$ i $$ for a user is now given by the relation

Microsoft's new marketing strategy: give up

Thu, 12 Dec 2013 00:00:00 +0000

I think it’s funny how MS at some point realized they are not the cool kids and there’s no reason to appeal to that target audience. Their new marketing strategy finally admits what’s been long known: the correlation between “business casual” and using Microsoft products:

Bagging as a regularizer

Fri, 06 Dec 2013 00:00:00 +0000

One thing I encountered today was a trick using bagging as a way to go beyond a point estimate and get an approximation for the full distribution. This can then be used to penalize predictions with larger uncertainty, which helps reducing false positives.

Model benchmarks

Sat, 02 Nov 2013 00:00:00 +0000

A lot of people have asked me what models we use for recommendations at Spotify so I wanted to share some insights. Here’s benchmarks for some models. Note that we don’t use all of them in production.

statself.com

Fri, 18 Oct 2013 00:00:00 +0000

Btw I just put something up online that I spent a couple of evenings in my couch putting together: it’s a website where you can track any numerical data on the web. Want to know how many Twitter followers you have? Temperature in NYC? Go to statself.com and start tracking it.

Implicit data and collaborative filtering

Mon, 16 Sep 2013 00:00:00 +0000

A lot of people these days know about collaborative filtering. It’s that Netflix Prize thing, right? People rate things 1-5 stars and then you have to predict missing ratings.

While there’s no doubt that the Netflix Prize was successful, I think it created an illusion that all recommender systems care about explicit 1-5 ratings and RMSE as the objective. Some people even distrust me when I talk about the approach we take at Spotify.

Vote for our SXSW panel!

Wed, 04 Sep 2013 00:00:00 +0000

If you have a few minutes, you should check out mine and Chris Johnson‘s panel proposal. Go here and vote: http://panelpicker.sxsw.com/vote/24504

Algorithmic Music Discovery at Spotify

****Spotify crunches hundreds of billions of streams to analyze user’s music taste and provide music recommendations for its users. We will discuss how the algorithms work, how they fit in within the products, what the problems are and where we think music discovery is going. The talk will be quite technical with a focus on the concepts and methods, mainly how we use large scale machine learning, but we will also some aspects of music discovery from a user perspective that greatly influenced the design decisions.

What's up with music recommendations?

Sat, 17 Aug 2013 00:00:00 +0000

I just answered a Quora question about what, if any, are the differences in the algorithms that are behind recommendations for music and movies.

Of course, every media type is different. For instance, there’s fundamental reasons why latent factor models works really well for music and movies, as opposed to location recommendations where I suspect graph based models are more powerful. People recommendations is another animal and I’m sure beer recommendations has its own domain-specific quirks.

3D

Mon, 12 Aug 2013 00:00:00 +0000

Andy Sloane decided to call my 2D visualization and raise it to 3D.

(Looks a little weird in the iframe but check out the link). It’s based on a LDA model with 200 topics, so the artists tend to stick to clusters where each cluster is a topic. The embedding also uses t-SNE but in three dimensions (obviously).

2D embedding of 5k artists = WIN

Sun, 11 Aug 2013 00:00:00 +0000

I’m at KDD in Chicago for a few days. We have a Spotify booth tomorrow, and I wanted to put together some cool graphics to show. I’ve been thinking about doing a 2D embedding of the top artists forever since I read about t-SNE and other papers so this was a perfect opportunity to spend some time on it.

Delivering Music Recommendations

Fri, 09 Aug 2013 00:00:00 +0000

I’ve turned into a lazy bastard and I’m just posting presentations on this blog, but here’s one from Rohan Singh at Spotify talking about the backend infrastructure of the Discover page.

ML+Hadoop at NYC Predictive Analytics

Sat, 03 Aug 2013 00:00:00 +0000

I was just at the NYC Predictive Analytics meetup talking about how we build machine learning algorithms using Hadoop to power music recommendations.

Great meetup, where we had two speakers, me and Blake Shaw from Foursquare. Blake talked about how they use machine learning at Foursquare, using Hadoop (and Luigi), and he uploaded his slides here!

HubSpot's Picture Shows how to Maintain Monocultures in the 21st Century

Sun, 28 Jul 2013 00:00:00 +0000

I thought this article about the company culture at HubSpot is kind of funny. “HubSpot’s Awesome Presentation Shows how to Create a 21st Century Culture”.

Just FYI: You’re not different. You’re a bunch of white hipsters aged 25-30 dressed up in the same theme. That’s not being different.

More Luigi: Presentation from OSCON

Sat, 27 Jul 2013 00:00:00 +0000

I was in Portland, OR for a few days hanging out at OSCON. Was fun. I also talked a bit about Luigi:

Next week I’m presenting at the NYC Predictive Analytics meetup together with Blake Shaw from Foursquare. The topic is ML + Hadoop. Will be fun!

Optimizing over multinomial distributions

Wed, 24 Jul 2013 00:00:00 +0000

Sometimes you have to maximize some function $$ f(w_1, w_2, ldots, w_n) $$ where $$ w_1 + w_2 + ldots + w_n = 1 $$ and $$ 0 le w_i le 1 $$ . Usually, $$ f $$ is concave and differentiable, so there’s one unique global maximum and you can solve it by applying gradient ascent. The presence of the constraint makes it a little tricky, but we can solve it using the method of Lagrange multipliers. In particular, since the surface $$ w_1 + w_2 + ldots + w_n $$ has the normal $$ (1, 1, ldots, 1) $$ , the following optimization procedure works:

More Luigi!

Wed, 26 Jun 2013 00:00:00 +0000

Continuing in the same spirit of shameless self-promotion, here’s some recent Luigi press:

Reddit thread
A Guide to Python Frameworks for Hadoop (slides from the NYC Hadoop User Group)
This presentation from the Open Analytics NYC meetup about how Foursquare uses Luigi

hdfs2cass

Wed, 19 Jun 2013 00:00:00 +0000

Just open sourced hdfs2cass which is a Hadoop job (written in Java) to do efficient Cassandra bulkloading. The nice thing is that it queries Cassandra for its topology and uses that to partition the data so that each reducer can upload data directly to a Cassandra node. It also builds SSTables locally etc. Not an expert at Cassandra so I’ll stop describing those parts before I embarrass myself.

NoDoc

Sun, 16 Jun 2013 00:00:00 +0000

We had an unconference at Spotify last Thursday and I added a semi-trolling semi-serious topic about abolishing documentation. Or NoDoc, as I’m going to call this movement. This was meant to be mostly a thought experiment, but I don’t see it as complete madness.

Wikiphilia

Sun, 02 Jun 2013 00:00:00 +0000

I’ve been obsessed with Wikipedia for the past ten years. Occasionally I find some good articles worth sharing and that’s why I created the wikiphilia Twitter handle. Just a long stream of stuff that for one reason or another may be interesting.

Spotify's Discovery page

Fri, 31 May 2013 00:00:00 +0000

The Discovery page, the new start page in Spotify, is finally out to a fairly significant percentage of all users. Really happy since we have worked on it for the past six months. Here’s a screen shot:

Fermat's principle

Tue, 21 May 2013 00:00:00 +0000

I was browsing around on the Internet and the physics geek in me started reading about Fermat’s principle. And suddenly something came back to me that I’ve been trying to suppress for many years – how I never understood why there’s anything fundamental about the principal of least time.

Snakebite

Tue, 07 May 2013 00:00:00 +0000

Just promoting Spotify stuff here: check out the Snakebite repo on Github, written by Wouter de Bie. It’s a super fast tool to access HDFS over CLI/Python, by accessing the namenode directly over sockets/protobuf.

Spotify’s developer blog features a nice blog post outlining what it’s useful for. I think this kicks ass and there will definitely be some kind of Luigi integration coming up at some point

Stuff that bothers me: “100x faster than Hadoop”

Sat, 27 Apr 2013 00:00:00 +0000

The simple way to get featured on big data blog these days seem to be

Build something that does 1 thing super well but nothing else
Benchmark it against Hadoop
Publish stats showing that it’s 100x faster than Hadoop
$$$

Spark claims their 100x faster than Hadoop and there’s a lot of stats showing Redshift is 10x faster than Hadoop. There’s a bunch of papers with similar claims. I spent five minutes Googling “Xx faster than Hadoop” and found a ton of other stats.

Presentation about Luigi

Fri, 26 Apr 2013 00:00:00 +0000

I like the editing!

Being data driven

Sat, 13 Apr 2013 00:00:00 +0000

I picked up an issue of Foreign Affairs while flying back to NYC from SFO. It features this long interview with U.S. General Stanley McChrystal and I thought it was pretty interesting how striking some of the similarities are between fighting in a war and developing software.

Annoy

Fri, 12 Apr 2013 00:00:00 +0000

Annoy is a simple package to find approximate nearest neighbors (ANN) that I just put on Github. I’m not trying to compete with existing packages, but Annoy has a couple of features that makes it pretty useful. Most importantly, it uses very little memory and can put everything in a contiguous blob that you can mmap from disk. This way multiple processes can share the same index.

More Luigi!

Fri, 22 Mar 2013 00:00:00 +0000

Elias Freider just talked about Luigi at PyData 2013:

The presentation above is much better than one I put together a few weeks ago. In case anyone is interested I'll include it too:

ML at Twitter

Wed, 27 Feb 2013 00:00:00 +0000

I recently came across this paper describing how they do ML at Twitter.

TL;DR Their approach is pretty interesting. Everything is a Pig workflow and then they do everything as UDF’s.

This approach seems pretty interesting. As long as your data can be expressed as small atomic machine learning functions, I’m sure it works great. But there’s so much more than that. All small slicing, transforming etc is so much easier to express in a language like Python. I’m still not really comfortable with Pig as a language to power these data flows.

I'm featured in Mashable

Wed, 06 Feb 2013 00:00:00 +0000

This article from today in Mashable describes some of the fun stuff I get to work with:

Erik Bernhardsson is technical lead at Spotify, where he helped to build a music recommendation system based on large-scale machine learning algorithms, mainly matrix factorization of big matrices using Hadoop. He moved into this role after heading the Business Intelligence team, where he collected, aggregated and made sense of all the data at Spotify, whether that’s ad-hoc insights, A/B testing, visualization or ad optimization.

Slides from NYC Machine Learning talk

Sun, 27 Jan 2013 00:00:00 +0000

Slides from the talk. Slightly edited because (a) some of the slides make little sense taken out of context (b) Slideshare seem to have problem converting some of the stuff.

Collaborative filtering at Spotify from Erik Bernhardsson

NYC Machine Learning meetup

Tue, 22 Jan 2013 00:00:00 +0000

From the NYC Machine Learning talk I had last week:

Haven’t looked at it yet except briefly. Unfortunately the quality isn’t the best.

Momentum and mean reversion might just be volatility bias

Sun, 13 Jan 2013 00:00:00 +0000

The Economist just published an article called The best, the worst and the ugly. By looking at historical performance for mutual funds, they find strong support for momentum and mean reversion. Picking the best or the worst fund over the previous five years gives great returns over the next five years.

Calculating cosine similarities using dimensionality reduction

Wed, 05 Dec 2012 00:00:00 +0000

This was posted on the Twitter Engineering blog a few days ago: Dimension Independent Similarity Computation (DISCO)

I just glanced at the paper, and there’s some cool stuff going on from a theoretical perspective. What I’m curious about is why they didn’t decide to use dimensionality reduction to solve such a big problem. The benefit of this approach is that it scales much better (linear in input data size) and produces much better results. The drawback is that it’s much harder to implement.

Tumblr's awesome project names

Sun, 18 Nov 2012 00:00:00 +0000

Not sure how I managed to miss this, but I’m watching this Tumblr presentation and they talk about their projects named after Arrested Development topics: Gob, Parmesan, Buster, Jetpants, Oscar, George and Motherboy.

Still, the best software project name is probably still Apple’s BHA.

A neat little trick with time decay

Mon, 29 Oct 2012 00:00:00 +0000

Something that pops up pretty frequently is to implement time decay, especially where you have recursive chains of jobs. For instance, say you want to keep track of a popularity score. You calculate today’s output by reading yesterday’s output, discounting it by $$ exp(-lambda Delta T) $$ and then adding some hit count for today. Typically you choose $$ lambda $$ so that $$ exp(-lambda Delta T) = 0.95 $$ for a day or something like that. We do this to generate popularity scores for every track at Spotify.

Luigi: complex pipelines of tasks in Python

Sun, 21 Oct 2012 00:00:00 +0000

I’m shamelessly promoting my first major open source project. Luigi is a Python module that helps you build complex pipelines of batch jobs, handle dependency resolution, and create visualizations to help manage multiple workflows. It also comes with Hadoop support built in (because that’s where really where its strength becomes clear).

About