08 May 2018

Setting up LUKS on Xubuntu 18.04

Spent some time this weekend setting up LUKS on my work laptop in preparation for some international travel.  This document is as much notes for me so I can reproduce the process as it is for my three dear readers who are probably bots.  Followed these directions, and for the most part they worked alright.  I did run into one issue: when running the refresh grub script from step 6 of this page, a number of x86_64-efi files did not exist.  I was able to work around this issue by running
apt-get install grub-efi-amd64-bin
, after which the script ran successfully.  I don't know why grub wasn't properly installed earlier in the setup procedure, but there you go.

I also made two small improvements to the process.  During paranoid setup, I used the AES noise fill from here:

openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero > /dev/sdxy 
In the check and finalize procedure, there's a note that you now have to manually run the update grub script every time you update the kernel.  But I know I'm going to forget to do that, so I googled a bit and found this thread, which suggested adding the script to /etc/kernel/postinst.d/ .  So I did that, and we'll see if it comes back to bite me in the ass and render this machine unbootable in a year or so.

17 March 2018

Peopleware, Slack, and Design Patterns

I read Peopleware last week, and was surprised to find that it referenced Christopher Alexander's work on design patterns in architecture, as these relate to laying out working spaces for developers.  I was struck by the notion of an intimacy gradient, where a household has public areas (a dining room or sitting room), then shared private areas (a kitchen, perhaps), and then individually-private areas (bedrooms, studies).  Peopleware proposed that we lay out spaces for programming teams in a similar way; meeting areas, then common areas, then private offices.

This all seems very reasonable, but I find myself lately working on geographically distributed teams reshuffled about quarterly, with most of our communication happening through slack.  There is a persistent pattern where a new team is formed (administratively; hardly to the level of unit cohesion that Peopleware would ask of something called a team), and someone creates a slack channel for that team and invites all the formal members of the team.  Gradually, more and more people from outside the team end up in the channel; sometimes they're experts in a particular thing that are called in for a particular question, sometimes they're managers or executives who invite themselves, sometimes they're just curious engineers trying to maintain situational awareness (I have been guilty of this). As the number of people increase, conversations tend to get sidetracked (or be off topic from the start), bystander effects increase, the SNR drops, and the utility to the team deteriorates.

To use Peopleware and Alexander's language: a shared private space for a team becomes a public space over time.  While we have the technical means to kick people out of channels (or should I say rooms?), it's never been done; it's outside the norms of our microculture.  In short: we're wusses.

My solution, on realizing the nature of this problem,was to create a direct message set with the other members of my team (so now we have effectively a public channel where management can ask us questions, and a private channel where we can figure out our collective answer before replying).  So far this has been very productive, and it is immune to gradual dilution.  It does produce a potential gap in organizational memory, though - if a new person is added to the team, they can't search that DM channel's history.

This realized parallel between slack and the architecture of working spaces raises some interesting questions about the social parts of the web generally.  I am sorely tempted to go hunt down a copy of Alexander's book, A Pattern Language, and see if there are more elements applicable to the design of digital spaces where people live and work, again by the "room" metaphor.

While investigating Alexander's work, though, I learned that it was the inspiration for the whole software design patterns set of memes.  I had heard of them, but had mostly seen them in code that I considered quite ugly, and so found them distasteful.  Reading more about them now, I have not found any evidence strong enough to shake that distaste.  I think at the very least, the method used to arrive at them is unsound.  When Alexander derived architectural design patterns, he did so from a thousand years of functional architecture, developed at global levels of parallelism and leaving behind concrete artifacts amenable to public study.  When the Gang of Four derived software design patterns, they did so from a bare half-century of object-oriented programming (which, as for example Norvig notes, is hardly the only reasonable idiom for programming).  Software engineering as a discipline does not have the corpus of high-quality, environment-adapted folk-work that architecture does.  Most of what we build is crap that evaporates in a decade or less, and the projects that survive do so not because of their great and highly-functional architecture but on the whims of the market.  It is still very early days to be inductively drawing software design patterns with anything like the authority that Alexander can achieve with architecture.

11 March 2018

Reading / Status, 11 March 18

The Google SRE book, up through chapter 26.  Chapter 25, on pipelines, and 26, on data integrity, were actually relevant to my work, so that was neat.  I was somewhat disappointed with Chapter 22, on cascading failures - they're a topic of great interest to me, but the treatment here was very practical, specific to distributed computing environments, rather than as a general phenomenon.  I was impressed by the degree to which PAXOS is central to Google's production systems (having previously considered PAXOS an academic / military-industrial curiosity with little commercial application).

Peopleware, 2nd Edition (because I'm cheap).  This is an excellent book, and agrees mightily with my experience as a software engineer and briefly as a team lead.  I'm considering springing for the 3rd edition.

Related to Peopleware's argument that most differences in programmer productivity are a product of the work environment, particularly distractions, Dan Luu on programmer moneyball and Abe Winter contra slack.

Paul Graham's Beating the Averages, and subsequently parts of ANSI Common LISP, particularly the macros chapter.  I was kind of unimpressed, which might mean that I didn't really understand it.

Parts of Thinking in C++, Second Edition, which stackoverflow recommended in answer to a query about "C++ for C programmers".

Parts of Cracking the Coding Interview.  Not an especially insightful book; I could see this having been very useful to me when I was an undergrad interviewing for internships, and it was a decent refresher on basic topics, but I guess the main thing I learned is that the bar for algorithmic knowledge might be lower than I thought and I shouldn't try to cram (say) optimal max-flow/min-cut and convex hull algorithms before interviews.

In sum: things which name and explain my dissatisfactions with my current employer, and resources for acquiring a new employer (maybe I should be trying to change things here instead of just bailing?).  Not so much stuff that I'm really interested in reading for its own sake.

15 February 2018

Reading / Status, 15 Feb 18

Been a while, still alive.  Got into a relationship right around the time of last post, ended in January, so now I'm back (to the extent that I was ever here).  Interesting things I've read (or done) recently:

Gwern on spaced repetition, started using mnemosyne for data related to a couple of new hobbies / capabilities I want to bring online this year.  Related and linked from Gwern, SuperMemo's guidelines for writing flashcards.

KB6NU's Tech and General ham radio exam study guides, followed by passing both of those licensing exams (spaced repetition helped).

Started supplementing vitamin D and fish oil (since I have very little sunshine or fish in my life).  Mood seems slightly increased generally, though I have had difficulty focusing (really need to start meditating regularly again).  A multi-month skin condition cleared up in 48 hours with the vitamin D supplementation, which was neat.  Also began playing with caffeine pills twice a week, rhodiola rosea once a week, and irregular panax ginseng.  Stimulants really don't do me any good without focus though.

Relatedly, SlateStarCodex on placebo - I'm willing to chalk up improved mood to regression to the mean following relationship termination, but the skin thing seemed pretty real.

Been having persistent fasciculation in my left eyelid, pretty annoying.  Have been getting ~7.5 hours of melatonin-assisted sleep most nights, though frequently interrupted in wee hours of morning.  Doesn't seem to be a magnesium deficiency; brocolli and blackstrap haven't done anything for it.  Might just be stress.

Most of the Google SRE Book.  Error budget seems like a great concept.  I like the line "every line of code is a liability."  Pleasantly surprised to find unattributed quotes from John Gall / General Systemantics.  Most of the book is not really applicable to my current work, though.

Evolving a Decompiler.  Very cool, I should really download and play with their code.

Several posts at Path Sensitive, along with the first half of this paper that he linked to (I intend to finish it).

Gwern's book review on a history of naval operations analysis.  I tend to roll pretty Chesterton, so this gave me some things to chew on.

Spolsky came up at work, so I read Field Guide to Developers (I really need to get around to reading this copy of Peopleware on my shelf), Guerilla Guide to Interviewing, and Paul Graham's Beating the Averages.

Martin Jambon's Universal Career Advice.

GDB's startup-with-shell off command, for when you're debugging programs that LD_PRELOAD libraries that break bash.  Relatedly, ld.so man page.  Lots of grungy environment variables to play with.

04 September 2017

Reading / Links, 4 Sept 17

Stuff I've been reading in the last week or two:

Network Science, chapters 3 and 4. Pretty funny; he throws some shade on Erdos and Strogatz.  The editing / proofreading continues to disappoint, but the material is decent.  The main thing I want out of this book is an understanding of cascade failures (which he claims to have a good model for in the introduction); a graph theory refresher doesn't hurt though.

The Mind Illuminated, Chapter 4, Interlude 4, beginning of Chapter 5.  Interlude 4 was very interesting - consciousness is quantized and cut up into frames, like network packets, and dullness is packets dropping.  I really wish he'd include footnote references for the science behind this stuff, given that he's a neuro guy...  Given chapters 3, 4, and 5, it seems like I'm somewhere in late phase 3 or early phase 4 (modulo the fact that my practice is still irregular).

The Systems Bible.  Has nothing to do with systems programming, except inasmuch as programmers build systems.  Describes ways in which complex systems evolve and dysfunction.  Not at all rigorous, to the point where it doesn't bother to define "system", but some parallels with Rao's Gervais Principle in the organizational context (organizations constructed with backdoors allowing actual work to get done, eventually collapse under their own entropy) and with some of Scott's criticisms of high modernism in Seeing Like a State (the designed system opposes its own intended function and scales in unpredictable ways).  Also seems sort of linked to The Dispossessed, with its point about the emergence of effectively-bureaucratic systems under anarchist conditions.

Introduction to the DWARF Debugging Format.  I'm looking for stupid dwarf tricks, and was excited to find that DWARF contains at least two sorts of bytecode for generating tables of debugging information.

Relatedly, Funky File Formats.

Documentation on ptrace, proc, ELF, bpf, more bpf...  There's all kinds of fun stuff in /proc that I didn't know about.

23 August 2017

Paper Notes - Predicting Bugs from History, 2008

Notes on this chapter of what seems to be a book on "software evolution".

"The defects we measure from history can only be mapped to components because they have been fixed."  That seems rather optimistic...

"This mistake manifests itself as an error in some development artefact, be it requirements, specification, or a design document."  This is getting concerningly software-engineering-ish...

Sort of weird that everything is module-scoped.  I guess it makes sense for very large systems, but I'd tend to think of things more on the function / basic-block level.

Also weird that all of their coupling metrics (except class coupling) are for global variables.  Any / all shared state is a potential bug source; global just happens to be the most egregious possible case (except for system-globals, like files on disk and registry entries...).

"Never blindly trust a metric."  Wiser words are rarely written.

Not a whole lot of surprises.  Typical complexity metrics correlate with bugs.  Churn correlates with bugs.  Tricky problem domains correlate with bugs (though I feel like their example of Eclipse compiler internals vs GUI is sort of disingenuous; if the compiler internals are a little broken, Eclipse cannot perform its core function, but if the UI is a little broken, often end users can work around it or just live with it.  So is it a function of the inherent difficulty of the problem domain, or the centrality of that problem domain to the function of the project?).  Buggy dependencies correlate with bugs, but fall off with distance.  Would've been interesting to see the d=4 case for "domino effect in Windows Server 2003".

Sort of bummed that their references weren't included.

Potential follow-ups:

20 August 2017

Paper Notes - Valgrind 2007

I've been doing some work with Valgrind recently, and the suggested way to get a big-picture understanding of how Valgrind works was to read this paper. Having read it, I think this is a good recommendation.  Some notes, primarily for my own benefit.

Dynamic recompilation seems very similar to my under-informed understanding of QEMU's approach.  Substantially more complex than our hacked-up approach to static binary instrumentation.  Would probably be a lot easier to implement nowadays with LLVM than it was in 2007.  Interesting loading procedure, though it has the same issue that PIN does where it shares an address space with its target (and a target seeking to interfere with analysis will likely be able to).  The dispatcher / scheduler translation execution mechanism is also interesting; doesn't do translation block chaining like QEMU does (we ran into an issue with QEMU's tb-linking a couple weeks ago), but has a very tight "dispatcher" mechanism that checks a cache and executes known / hot translations, with the slower "scheduler" as fallback.  Coming from writing system call models in PIN, the events system sounds pretty great; I wonder how much of Valgrind's syscall models are stealable for use in other dynamic instrumentation frameworks.

Follow-up topics I should read more on: