You are not logged in.

sprezzatech blog #0018

Dirty South Supercomputers and Waffles
UNIX, HPC, cyberwarfare, and chasing perf in Atlanta.

SprezzOS Weeklyish Oblations, Objections, and News (SWOON) Vol. 1 Issue 6.
Thu Apr 25 12:40:54 EDT 2013

SprezzOS Weeklyish Oblations, Objections, and News (SWOON)

Volume I, Issue 6. 2013-04-25. Editor: Nick Black.



I. INTRODUCTORY RITES

“It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.”

—Alfred North Whitehead, An Introduction to Mathematics (1911)


“Hey there, fella, whose soul are you tormenting now?”

—Clutch, “Binge and Purge” (1993)


Well, SWOON seems to have become more Monthlyish than Weeklyish of late, but SMOON just doesn't have the same ring to it (“S-M-O-O-N, that spells SMOON!”), so we'll let things stand. I've spent a lot of time recently preparing for the class I'll be teaching this summer at Georgia Tech, playing around with the newly complete C++11 support in LLVM, preparing Sprezzatech's initial hardware line, hacking on OpenGL and XCB, and doing some work to keep everything funded. Busy, busy, busy, as they said in the Books of Bokonon.

Work on RAPTORIAL continues, but my focus has shifted to improving performance of the core, rather than outlying pieces like apt-file. With the aptitude maintainers having publicly stated that they're “uninterested in performance” and generally unwilling to accept patches devoted towards that end, I've gained significant liberty and freedom of action—essentially, I've accepted that Debian isn't going to welcome this work anytime soon, and thus I needn't make such an effort of keeping my changes minimal. So, I've been reading the dpkg, aptitude and APT sources, familiarizing myself with the dirty depths of the APT stack. I think that, with a bit of effort, APT could be made fast enough that never again will I sit there, wondering “what the hell is taking you so long?” Not only APT itself, but the package-building infrastructure (e.g. dh-shlibdeps, dh-install, etc) can all be vastly accelerated. Hurrah, differentiation!

fish is one of the coolest programs I've seen in awhile.



Κύριε, ἐλέησον.


II. LITURGY OF THE WORD


Updates ahoy! As always, this list is neither autoritative nor exhaustive: I'm currently working on moving us to kernel 3.8.8, and that should be done soon.


καὶ εἰς ἕνα Κύριον Ἰησοῦν Χριστὸν τὸν υἱὸν τοῦ Θεοῦ τὸν Μονογενῆ.


III. LITURGY OF THE EUCHARIST


With 2318 source packages in SprezzOS, all of them maintained by moi, I think I've got room to speak about Debian's vaunted packaging practices. That said, I do not speak the following as a Debian Developer, nor do I consider myself tremendously well-informed regarding Debian as a whole. I'm sure there will be some incorrect statements and omissions in the next few paragraphs, and I welcome your corrections. My main thrusts, however, I expect to stand.

Debian packaging:
  1. Has at least three major implementations
  2. Is nowhere completely documented or even described
  3. Cannot scale without adding more developers
  4. Is unsuitable for analysis at the source level
  5. Is painfully slow to learn, debug, and perform
  6. Is rife with violations of what standards do exist
 The definition of a debian package is based around the binary deliverable, aka the “deb”, as described in deb(5). This format has been stable since Debian 0.93. It is used by SprezzOS, Ubuntu, and any other Debian derivative, and works well enough. The ar archive within a deb contains as its second member a gzipped tar archive control.tar.gz, of which the file control is a mandatory member.

But that's about it. You can build this ar archive with debuild, or dpkg-buildpackage or debian/rules binary or ar or cat. It appears that the autobuilders use dpkg-buildpackage, looking at the porting guide, so let's use that as our minimum bar for packaging materials -- you ought build in a fresh chroot + all listed deps.

Except what do you build? The process of autobuilding relies on a source package. But when you're doing the packaging itself, you haven't yet built a source package. When you're taking over a package, there's no assurance that you can find the original tarballs on the original source, even when that source still exists. This is one purpose of watch files—to point at the original source, thus facilitating automatic checks for new versions. Obviously, this is not a complete solution (packages move sometimes), but it allows the vast majority of upstream updates to be automatically detected. Unfortunately, Debian doesn't require that these files be correct or included (it's recommended now, but I'd estimate a good 20% of source packages to have incorrect watchfiles). The first thing I do when I take over a package for SprezzOS is validate and correct the watchfile. Some upstream packages, however, are released in a format not amenable to uscan + dpkg-buildpackage, and require some intermediate processing. This processing is nowhere codified and seems to be done totally ad hoc; either way, it breaks our attempts to automatically update.

Thankfully, these changes tend to be oriented towards e.g. DFDL satisfaction, and are thus inappropriate for SprezzOS. Once the upstream is being properly unpackaged, we blow away any debian/ directory it might have shipped, and copy in our own. At this point, we run into any number of problems in the packaging itself, i.e. that would not cause us problems when building from a local tarball without intent to package: It is this last that is the most time-consuming. For the first issue, we drop the (usually questionable in the first place) tool specification, and almost always succeed on rebuild. The second issue can be addressed with the right additions to a debhelper or CDBS-based build, but one must verify that these additions have been properly made, and rebuild on failure after fixing up the install manifests and, possibly, the control file, and it hardly seems the best possible solution. The third issue is always caught, though it forces a manifest+control edit + rebuild. Avoiding this generally requires use of wildcards, which are inapplicable in the presence of fine-grained package division (as occurs in e.g. most KDE packages, and most packages installing multiple shared libraries). autotools issues aren't generally too big an issue, and can probably be considered universal anyway. Same thing can probably be said about new libraries. That soname changes aren't automatically handled is probably the single most annoying and stupid issue, and infuriates me every time I rebuild a library. I don't want to hear excuses, either. If you can bitch about it, which you certainly do, you can detect it. The few dozen ad hoc methods which extract the soname from within rules, and use that to set variables in control.in, are even worse. Every one of those ought have been time spent solving the real problem. Everything about this pisses me off all the time.

The fine-grained division of packages that suffuses Debian like a thousand paper cuts are their own problem.

But it's the free-form rules file that provides me the most constant serious problems. Who thought that a Turing-complete logic programming system was the best way to build more than, like, three packages? You can use your own free-form Make; you can use debhelper, or you can use cdbs. There's almost as many packaging implementations for this one format as there are build systems in use, and just like the build systems, they're all fucking terrible. They are embarrassments. I've been shocked and profoundly disturbed over this past half-year of SprezzOS development, having run Debian for thirteen years or so now, at how the sausage is made. If we consider obviously future-unproofed chicanery having clearly superior and immediately identifiable improvements as “bugs” —which they certainly are to the poor SprezzOS bastard who has to come along and update your package for the first time in years—then rules files probably have, on average, bugs proportional to and greater than their length. I.e. more than one bug per line. When I open a rules file of more than 10 lines, I deflate. When it's more than 30 lines, I look around really hard for something else to do, like check the price of blue-ring octopodes or practice knotting my dick into a pretzel. When it's, say, gcc-4.8 I waste twenty to forty curse-dense hours trying to get it to work, and eventually throw up my hands.

CDBS is actually pretty good. If every package had a watchfile and a CDBS-based rulesfile, I'd be a pretty happy guy. Happier than I am today, for sure.

More on this topic at a later date. Know that a solution inchoate is forming itself in my fetid mind.



Mysterium fidei.


IV. COMMUNION RITE


Not so many patches accepted upstream recently...Allegro5, Muffin, argyle, maybe something else. Actually, I guess that's one a week, which has been pretty steady since we started—it just seems less impressive now? Who knows.


Agnus Dei, qui tollis peccata mundi, miserere nobis.


V. CONCLUDING RITE

Ite, missa est.


Per ardua ad astra.
SprezzOS logo