blog dds

2018.11.27

Java Stream Methods and Unix Pipeline Commands: A Dictionary

While preparing my class notes for functional programming in Java I was struck between the neat correspondence between many Java Stream methods and Unix commands. I decided to organize the most common of these in a dictionary form that allows the mapping between the two. I'd very much welcome comments regarding common patterns that I've missed.

Continue reading "Java Stream Methods and Unix Pipeline Commands: A Dictionary"

2018.11.16

Debugging had to be discovered!

I start my Communications of the ACM article titled Modern debugging techniques: The art of finding a needle in a haystack (accessible from this page without a paywall) with the following remarkable quote. "As soon as we started programming, [...] we found to our surprise that it wasn't as easy to get programs right as we had thought it would be. [...] Debugging had to be discovered. I can remember the exact instant [...] when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs." A Google search for this phrase returns close to 3000 results, but most of them are cryptically attributed as "Maurice Wilkes, discovers debugging, 1949". For a scholarly article I knew I had to do better than that.

Continue reading "Debugging had to be discovered!"

2018.08.05

How I slashed a SQL query runtime from 380 hours to 12 with two Unix commands

I was trying to run a simple join query on MariaDB (MySQL) and its performance was horrendous. Here's how I cut down the query's run time from over 380 hours to under 12 hours by executing part of it with two simple Unix commands.

Continue reading "How I slashed a SQL query runtime from 380 hours to 12 with two Unix commands"

2018.04.03

How to Perform Set Operations on Terabyte Files

The Unix sort command can efficiently handle files of arbitrary size (think of terabytes). It does this by loading into main memory all the data that can fit into it (say 16GB), sorting that data efficiently using an O(N log N) algorithm, and then merge-sorting the chunks with a linear complexity O(N) cost. If the number of sorted chunks is higher than the number of file descriptors that the merge operation can simultaneously keep open (typically more than 1000), then sort will recursively merge-sort intermediate merged files. Once you have at hand sorted files with unique elements, you can efficiently perform set operations with them through linear complexity O(N) operations. Here is how to do it.

Continue reading "How to Perform Set Operations on Terabyte Files"

2018.02.10

The Shoemaker's Children Go Barefoot

Earlier today I submitted the camera-ready version of a technical briefing on mining Git repositories, which Georgios Gousios and I will be presenting at the 2018 International Conference on Software Engineering. I was struck by the complexity and inefficiency of the administrative process.

Continue reading "The Shoemaker's Children Go Barefoot"

2017.11.19

Reviving the 1973 Unix Programmer's Manual

The 1973 Fourth Edition of the Unix Programmer's Manual doesn't seem to be available online in typeset form. This is how I managed to recreate it from its source code.

Continue reading "Reviving the 1973 Unix Programmer's Manual"

2017.11.17

How I Recovered my Firefox Tab Groups

When quit and restarted Firefox today I received an unwelcomed shock. All my tab groups, which I maintained using the Tab Groups by Quicksaver plugin, were gone! This happened because it upgraded to Firefox Quantum (57), whose API does not maintain backward compatibility with the one used by the plugin. Although I knew the plugin would one day stop working, I thought there would be some last-minute warning and chance to export the tab groups.

Continue reading "How I Recovered my Firefox Tab Groups"

2017.10.03

An Embarrassing Failure

My colleague Georgios Gousios and I are studying the impact of software engineering research in practice. As part of our research, we identified award-winning and highly-cited papers, and asked their authors to complete an online survey. Each survey was personalized with the author's name and the paper's title and publication venue. After completing a trial and a pilot run, I decided to contact the large number of remaining authors. This is when things started going horribly wrong.

Continue reading "An Embarrassing Failure"

2017.09.15

Who are the Publishers of Computer Science Research?

To answer this question, I downloaded the DBLP database and used the DOI publisher prefix of each publication to determine its publisher. I grouped the 3.4 million entries by publisher and joined the numeric prefixes with the publisher names available in the list of Crossref members. Based on these data, here is a pie chart of the major publishers of computer science research papers.

Continue reading "Who are the Publishers of Computer Science Research?"

2017.09.14

The Origins of Malloc

The 1973 Fourth Edition Unix kernel source code contains two routines, malloc and mfree, that manage the dynamic allocation and release of main memory blocks for in-memory processes and of continuous disk swap area blocks for swapped-out processes. Their implementation and history can teach us many things regarding modern computing.

Continue reading "The Origins of Malloc"


Creative Commons License Last update: Thursday, December 6, 2018 9:42 pm
Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a Creative Commons Attribution-Share Alike 3.0 Greece License.