That moment when you spend all weekend generating a whole lotta data, only to realise there was an error in the input set and everything you created is invalid and needs to be thrown away…

I generated 1.1TB of string data for a project, overnight. It’s just one big text file on a disk. Now I just have to grep through it to find the particular patterns I need… that 1.1TB will probably come down to 500-600GB by the end of it, but I can see the pattern-matching process taking the rest of the weekend…

Python and command-line utilities have been super useful at generating this data, and definitely helped the process along. As a reminder to myself, these are the commands I’m using to “post-process” the data:

Look for lines in input.csv which don’t match this pattern, and echo them to output.csv:

$ grep -vE "([A-K]{3}),\1" input.csv > output.csv

Split output.csv into files 800MB in size, called data_n, where n is an 8-digit incremental number (e.g. data_00000001):

$ split -a 8 -d -b 800M output.csv data_

For each data file in the directory, give it the .csv extension:

$ for f in data*; mv "$f" "$f.csv"; done

Reposting: Jodi on Twitter

“Pia Klemp, the German ship captain who rescued migrants in the Mediterranean, as she refuses a medal from the mayor of Paris.”


“I’m not a humanitarian. I am not there to ‘aid’. I stand in solidarity. We do not need medals. We do not need authorities deciding who is a ‘hero’ and who is ‘illegal’. In fact they are in no position to make this call, because we are all equal.

“What we need are freedom and rights. It is time we call out hypocrite honouring and fill the void with social justice. It is time we cast all medals into spearheads of revolution!

“Documents and housing for all!

“Freedom of movement and residence!”

Pia Klemp, August 2019

Cool. Sometime recently, Flickr migrated their signup process to their own native system. I’ve been curious about Flickr under its new owners, but as recently as a couple of months ago, signing up still required creating a Yahoo account — something there was zero chance of me doing.

I’m not sure what I’d use Flickr for nowadays, but I do have fond memories of the service in its heyday of the mid-2000s. I can syndicate from this site to my Flickr photostream, although that means I’ll probably have to start using the full-sized original images in posts instead of the resized versions I use currently.

For whatever reason, I’m finding I’m just not “feeling it” with any of the miniatures I’ve been painting recently… not so much with the models themselves, more with the results of my efforts. I know I can often be overly critical of my own work, but this feels different to usual