Snakebite

Just promoting Spotify stuff here: check out the Snakebite repo on Github, written by Wouter de Bie. It's a super fast tool to access HDFS over CLI/Python, by accessing the namenode directly over sockets/protobuf.

Spotify's developer blog features a nice blog post outlining what it's useful for. I think this kicks ass and there will definitely be some kind of Luigi integration coming up at some point