Erik Bernhardsson    About

ML+Hadoop at NYC Predictive Analytics

I was just at the NYC Predictive Analytics meetup talking about how we build machine learning algorithms using Hadoop to power music recommendations. Great meetup, where we had two speakers, me and Blake Shaw from Foursquare. Blake talk...

More Luigi: Presentation from OSCON

I was in Portland, OR for a few days hanging out at OSCON. Was fun. I also talked a bit about Luigi: Next week I’m presenting at the NYC Predictive Analytics meetup together with Blake Shaw from Foursquare. The topic is ML + Hadoop. Wil...

Optimizing over multinomial distributions

Sometimes you have to maximize some function where and . Usually, is concave and differentiable, so there’s one unique global maximum and you can solve it by applying gradient ascent. The presence of the constraint makes it a litt...

More Luigi!

Continuing in the same spirit of shameless self-promotion, here’s some recent Luigi press: Reddit thread A Guide to Python Frameworks for Hadoop (slides from the NYC Hadoop User Group) This presentation from the Open Analytics NY...

hdfs2cass

Just open sourced hdfs2cass which is a Hadoop job (written in Java) to do efficient Cassandra bulkloading. The nice thing is that it queries Cassandra for its topology and uses that to partition the data so that each reducer can upload d...

NoDoc

We had an unconference at Spotify last Thursday and I added a semi-trolling semi-serious topic about abolishing documentation. Or NoDoc, as I’m going to call this movement. This was meant to be mostly a thought experiment, but I don’t se...