Posts in 2018

 

Java Stream Methods and Unix Pipeline Commands: A Dictionary

While preparing my class notes for functional programming in Java I was struck between the neat correspondence between many Java Stream methods and Unix commands. I decided to organize the most common of these in a dictionary form that allows the mapping between the two. I'd very much welcome comments regarding common patterns that I've missed.

Continue reading "Java Stream Methods and Unix Pipeline Commands: A Dictionary"

Debugging had to be discovered!

I start my Communications of the ACM article titled Modern debugging techniques: The art of finding a needle in a haystack (accessible from this page without a paywall) with the following remarkable quote. "As soon as we started programming, [...] we found to our surprise that it wasn't as easy to get programs right as we had thought it would be. [...] Debugging had to be discovered. I can remember the exact instant [...] when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs." A Google search for this phrase returns close to 3000 results, but most of them are cryptically attributed as "Maurice Wilkes, discovers debugging, 1949". For a scholarly article I knew I had to do better than that.

Continue reading "Debugging had to be discovered!"

How I slashed a SQL query runtime from 380 hours to 12 with two Unix commands

I was trying to run a simple join query on MariaDB (MySQL) and its performance was horrendous. Here's how I cut down the query's run time from over 380 hours to under 12 hours by executing part of it with two simple Unix commands.

Continue reading "How I slashed a SQL query runtime from 380 hours to 12 with two Unix commands"

How to Perform Set Operations on Terabyte Files

The Unix sort command can efficiently handle files of arbitrary size (think of terabytes). It does this by loading into main memory all the data that can fit into it (say 16GB), sorting that data efficiently using an O(N log N) algorithm, and then merge-sorting the chunks with a linear complexity O(N) cost. If the number of sorted chunks is higher than the number of file descriptors that the merge operation can simultaneously keep open (typically more than 1000), then sort will recursively merge-sort intermediate merged files. Once you have at hand sorted files with unique elements, you can efficiently perform set operations with them through linear complexity O(N) operations. Here is how to do it.

Continue reading "How to Perform Set Operations on Terabyte Files"

The Shoemaker's Children Go Barefoot

Earlier today I submitted the camera-ready version of a technical briefing on mining Git repositories, which Georgios Gousios and I will be presenting at the 2018 International Conference on Software Engineering. I was struck by the complexity and inefficiency of the administrative process.

Continue reading "The Shoemaker's Children Go Barefoot"

Become a Unix command line wizard
edX MOOC on Unix Tools: Data, Software, and Production Engineering
Debug like a master
Book cover of Effective Debugging
Compute with style
Book cover of The Elements of Computing Style
Syndication
This blog is also available as an RSS feed:

Category Tags
AWS (4)
Android (2)
Apple (9)
C (20)
C++ (16)
COVID-19 (1)
Cloud (1)
Computers (58)
Databases (4)
Debugging (8)
Design (1)
Discussion (6)
Electronics (15)
Environment (1)
FreeBSD (26)
Funny (14)
GSIS (5)
Google (6)
Government (3)
Hacks (26)
Hardware (25)
History (9)
Information systems (1)
Internet (12)
Java (26)
Linux (7)
Machine learning (1)
Management (27)
Microsoft (11)
One Laptop Per Child (3)
Open source (57)
Opinion (28)
Parenting (11)
Perl (13)
Photos (13)
Politics (4)
Programming (108)
Python (2)
R (1)
Raspberry Pi (5)
Risks (6)
Scala (1)
Science (33)
Security (26)
Sights (19)
Smartphones (3)
Software (22)
Software engineering (90)
Standards (7)
System administration (46)
Teaching (9)
Technology (28)
Testing (2)
Tips (43)
Tools of the Trade (52)
Travel (9)
UML (6)
Unix (47)
Web (31)
Windows (15)
Writing (45)
XML (10)
vim (5)
Archive
Complete contents (366)
2020 (12)
2019 (4)
2018 (5)
2017 (20)
2016 (7)
2015 (6)
2014 (5)
2013 (13)
2012 (17)
2011 (14)
2010 (13)
2009 (40)
2008 (40)
2007 (41)
2006 (48)
2005 (44)
2004 (30)
2003 (7)

Last update: Thursday, August 27, 2020 7:26 pm

Creative Commons Licence

Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.