How I slashed a SQL query runtime from 380 hours to 12 with two Unix commands
I was trying to run a simple join query on MariaDB (MySQL) and its performance was horrendous. Here's how I cut down the query's run time from over 380 hours to under 12 hours by executing part of it with two simple Unix commands.
Continue reading "How I slashed a SQL query runtime from 380 hours to 12 with two Unix commands"
Modular SQL Queries with Unit Tests
I'm sure I'm not the only person on earth facing a complex and expensive analytical processing task. The one I've been working on for the past couple of years, runs on the GHTorrent 98.5 GB data set of GitHub process data. It comprises 99 SQL queries (2599 lines of SQL code in total) and takes more than 20 hours to run on a hefty server. To make the job's parts run efficiently and reliably I implemented simple-rolap, a bare-bones relational online analytical processing tool suite. To ensure the queries produce correct results, I wrote RDBUnit, a unit testing framework for relational database queries. Here is a quick overview on how to use the two.
Continue reading "Modular SQL Queries with Unit Tests"
This column is about a tool we no longer have: the continuous rise of the CPU clock frequency. We were enjoying this trend for decades, but in the past few years, progress stalled. CPUs are no longer getting faster because their makers can’t handle the heat of faster-switching transistors. Furthermore, increasing the CPU’s sophistication to execute our instructions more cleverly has hit the law of diminishing returns. Consequently, CPU manufacturers now package the constantly increasing number of transistors they can fit onto a chip into multiple cores—processing elements—and then ask us developers to put the cores to good use.
Continue reading "Faking it"