What's Erik up to?

I joined Better in early 2015 because I thought the team was crazy enough to actually change one of the largest industries in the US. For six years, I ran the tech team, hiring 300+ people, probably doing 2,000+ interviews, and according to GitHub I added 646,941 lines of code and removed 339,164. But I also got married, had two kids, bought an apartment and renovated it! From time to time, there was some intense periods of hard work.

Better has gotten to a pretty amazing place. It's one of the largest startups in NYC (by several metrics, like valuation or headcount) and it has a world class engineering team that makes me insanely proud. I think it will keep being super successful in the future as it deconstructs the housing market down to its core principles (people want somewhere to live!) and builds something that solves these things with a much better experience and less waste.

Anyway, after six years of Better, I'm moving on to do something new. I always had an itch to do my own thing, and I was originally planning to do that back in 2015 when I left Spotify. But Vishal (the CEO of Better) convinced me to spend a year or two with him, and learn how the sausage is made. How to raise money, how to work with the board, how to run a company, and all that stuff. I joined and I got stuck for a lot longer than I thought!

At some point, I decided that I need to do my own thing, or I'm never going to do it. Since Jan 1 this year, Diane Yu is now running the tech team. We were super lucky to find her and I feel confident that she's going to take Better to the next level.

With Diane at Better, I took a 6 week semi-vacation in the Caribbean with my family, working 2-3 days a week and taking care of two crazy girls the rest of the time.

Anyway, since a few weeks ago, I'm back in NYC ready for some serious crunch mode now.

So what's next?

So what am I doing next? I'm extremely psyched to share with the world that I will be joining Oracle to work as a Senior Implementation Engineer, on Blockchain solutions for mid-size maritime companies!

The opportunities are unlimited. The Suez blockage shows how vulnerable we are, and I'm 1000% convinced it would have been prevented if they had used a decentralized P2P network powered by huge FPGA-rigs in Inner Mongolia computing quadrillions of hash values every second.

Sorry, just kidding… it's April Fools day, after all. But I am leaving Better, for real.

So what am I focused on next? I'm extremely determined that I want to start my own thing (meaning, don't try to hire me, it's probably a waste of time), and it's highly likely it will be something in the data engineering/science tools/infra space.

Why data?

I've spent most of my career working in data in some shape or form. At Spotify, I was entirely focused on it. At Better, I was a CTO with responsibility for all of technology, but for a long time I was also the manager of the data team, and I always kept dabbling with data problems in my spare time.

Data as a subfield of software engineering has a crazy growth rate. 10 years ago, companies didn't have big data teams. But these days, a lot of startups have 20%+ of their tech team dedicated to data (engineering/science). At the same time, I look at backend/frontend engineers and see how many amazing tools they have gotten in the last 10 years. Not just tools, but also processes. I truly think they are an order of magnitude more productive than they used to be.

I don't see this productivity revolution for data teams! I see teams wasting lots of time with nonsense tasks, like writing YAML or waiting for jobs to finish. But I also see other teams going around data teams and building up their own “analyst” teams. To me this reflects on the extreme demand for people who are actually skilled in working with data. How can we make them more productive? And I think a lot of it boils down to giving them better tools.

I'm interested in about 100 subproblems in this field and I'm hacking on prototypes for several of these things, but just to pick a few:

• Workflow scheduling. I built Luigi a long time ago, which I think was kind of the first widely used one. Later, Airflow has become the de-facto standard. But I still don't think it's anywhere close to where it could be in terms of turbocharging data teams.
• All the work around workflow scheduling. Lineage, documentation, observability, et cetera. I think there's an opportunity to solve a lot of those problems at the same time.
• The chasm between SQL and Python. What to use for what, and how to increase the interoperability.
• How to run data jobs. I don't think Kubernetes is the answer. It's hard to work with, not great for ad-hoc jobs, and fundamentally in the age of cloud, why can't we have a much more elastic pool of resources available?
• Visualization. I kind of want something that's both matplotlib and WYSIWYG at the same time?

I could keep adding to this list for hours, these are just some things off the top of my mind.

Anyway, feel free to drop me a line if you're interested in anything related to this. For the next few months, I'm going to be knee deep in code hacking on prototypes, so I love any break where I can talk to people who care about these things! I don't want to put my email here, but I'm sure you can figure out how to reach me.

🍻