The little Aguaxaca project has been humming along for a month now. Ingesting data as it comes, and I decided to take 30 minutes to take a look at the mess I’m now in. *starts panicking*

Lots of things here are mostly for my own sake. Sorry if that’s noise in your RSS reader.

Where were we?

Currently, I get a list of water-delivery spots (locations) from “@SOAPA_Oax”’s images on X/Twitter, and store lots of stringy data as a result. I’m gathering data, so the schema is 💩 on purpose. I should probably trash alter it once I’m more confident about the shape of the “source material”, and how to parse this stuff properly.

For each new delivery report, I create a new deliveries record with the location name as a string. Lots of duplication. This will obviously not scale, but it’s fine for the initial exploration phase and volume of data. The goal is to build a clean directory of locations and child locations, and store deliveries with a foreign key from locations for example…

Where we got

This is a small overview of locations names, after ±1 month of collecting raw data. This should be a full panorama of all the locations (or 99% of them, hopefully?) that get water deliveries in the city of Oaxaca. Yay. 🎉

You can follow along on agua.cypr.io, just to see how bad things are, and feel some Schadenfreude at my expense.

We can already see that locations are further divided in “sub-locations”. For example the Infonavit location includes 1a etapa, 2a etapa, (…), each with distinct delivery schedules, most of the time. Oh boy. 🥲

The same pattern is repeated for the Guadalupe Victoria location, and many others. For all these, we could have a simple parent-child relation. For example, the location Infonavit would have several sub-locations like 1a etapa, 2a etapa, etc. Sub-locations are optional, and sometimes a “parent” location will also be advertised in a delivery schedule, without more details. I mean, it could be “Yay! everyone gets water today!”, or “meh… typing these names is annoying”.

All locations have a distinct type, like colonia, ejido, fraccionamiento, unidad, etc. This is usually coherent, good. We can keep that as is, or spawn a separate table. I doubt we’ll see more than 10 different location types though, so it’s more about DB size and performance than anything (I mean storing a foreign-key integer instead of a string).

So, what do we know?

1. As I mentioned, lots of location names have sub-locations. These are usually appended between parens in the delivery announcement. consistency here.
2. The guiding rule could be that the “top” location is whatever is not inside parens, and then extract details about sub-locations to be as thorough as possible.
3. I should probably capitalize/downcase all names to get a semblance of
4. Different sub-locations can get deliveries on different days, so we do need the granularity of location and sub-location.

The parent-child relation would still allow some sort of tree-like structure, and granularity if we want to make something out of the data, for example send notifications when a specific sub-location gets water, or is likely to get water in the coming days.

Show us the data

Here’s some select data for a few locations, if you wanted a more concrete overview.

📍 Yalalag (parte baja)

Location would be “Yalalag”, with a sub-location like “Parte Baja”.

📍 Reforma (parte baja), and Reforma (parte alta)

Location would be “Reforma”, with 2 sub-locations like “Parte Alta”, and “Parte Baja”.

Now, some less obvious ones, that would require a better parser.

📍 Circuito Panorámica del Fortín (sectores 1, 2 y 3)

Location would be “Circuito Panorámica del Fortín”, with 3 sub-locations here:

• Sector 1,
• Sector 2, and
• Sector 3.

📍 Circuito Madero (de la Iglesia del Ex Marquezado a la Av. Tecnológico)

Location: Circuito Madero

Sub-location: de la Iglesia del Ex Marquezado a la Av. Tecnológico

These are street or landmarks names, no use in trying to parse this further.

📍For the Guadalupe Victoria location, we get a lot of variants:

• Guadalupe Victoria (sectores 3 y 4 parte baja)
• Guadalupe Victoria (sector 5 parte baja)
• Guadalupe Victoria (sectores 3, 4, 5 y 6 parte alta)
• Guadalupe Victoria (sector 6 parte baja)
• Guadalupe Victoria (sección Oeste sector 2)
• Guadalupe Victoria (2ª sección Oeste sector 1)
• Guadalupe Victoria (sector 1, 2ª sección Oeste)

It looks like we have a bunch of “sectors”. Some sectors have additional distinctions like “parte alta” and “parte baja”. From this list, I gather:

Location: Guadalupe Victoria Sub-locations:

• Sector 1
• Sector 2
• (…)
• Sector 6
• sección Oeste
• 2ª sección Oeste

💭 I’d auto-create sub-locations by parsing the text (splitting on commas, “y” conjunctions, etc.), but there’s probably not much use trying to go beyond one level: meaning “sector 6”, “sector 6 parte baja” and “sector 6 parte alta” are 3 sub-locations of “Guadalupe Victoria”.

📍 Dolores

• Dolores (zona Oriente parte baja)
• Dolores (sector Poniente parte baja)
• Dolores (sector Poniente parte alta)
• Dolores (parte baja)

Location: Dolores

Sub-locations (so far…):

• Zona Oriente Parte Baja
• Sector Poniente Parte Baja
• Sector Poniente Parte Alta
• Parte Baja

I guess some days you write “sector”, and some other days “zona”. It’s probably the same thing. Just great.

📍América Sur

We get the usual “parte alta” and “parte baja” sub-locations, but with a small twist:

• América Sur (parte baja)
• América Sur (parte baja y alta)

Location: América Sur

Sub-locations: Parte Baja, and Parte Alta

(Yes, there’s also América Norte, it’s a different location.)

📍San Juan Chapultepec

San Juan Chapultepec was both advertised as agencia when it comes without sub-locations, and a colonia when it comes with several sub-locations. Let’s focus on the latter here.

• San Juan Chapultepec y sus Barrios
• San Juan Chapultepec (parte baja)
• San Juan Chapultepec (parte media)
• San Juan Chapultepec (parte alta y sus Barrios)

That last one is 🏆. I guess we could have this hierarchy:

Location: San Juan Chapultepec

Sub-locations: “Parte Baja”, “Parte Media”, “Parte Alta”, and well… “Barrios” 😑.

This breaks the assumption that sub-locations are always between parentheses. Damn it.

📍Centro

The central district locations are also all over the place. Here’s a list of locations we get searching for “centro”:

• Centro Sur y Medio (de la calle Abasolo a Periférico Sur)
• Centro Sur y Medio (de la calle Abasolo a Periférico Sur)
• Centro Sur Poniente
• Centro Sur Poniente
• Centro Norte (de Constitución a Calzada Niños Héroes de Chapultepec)
• Centro Norte (de Constitución a Calzada Niños Héroes de Chapultepec)
• Centro Sur Poniente (sector Huzares)
• Centro Medio (de calle Colón a Abasolo)

Locations:

• Centro Sur,
• Centro Norte,
• Centro Medio, and probably
• Centro Sur Poniente.

Sub-Locations (again street names and landmarks):

• de la calle Abasolo a Periférico Sur
• de Constitución a Calzada Niños Héroes de Chapultepec
• sector Huzares
• de calle Colón a Abasolo

📍Volcanes

Here’s a list of locations names we’ve seen:

• Volcanes (sector 3 zona norte)
• Volcanes (sectores 4 y 5)
• Volcanes (sector 1 parte baja)
• Volcanes (sector 1 parte baja)
• Volcanes (sectores 4 y 5)
• Volcanes (zona Norte, sector 3)

Location: Volcanes

Sub-Locations:

• Sector 1 Parte Baja
• Sector 3
• Sector 3 Zona Norte
• Sector 4
• Sector 5
• Zona Norte

In this case as with others, I wonder whether these are actually 3 different locations: “sector 3”, “sector 3 zona norte”, and “zona norte”… The delivery schedules mention both at different dates:

• Aug 27: Volcanes (sector 3 zona norte)
• Aug 1st: Volcanes (zona Norte, sector 3)

Given the delivery interval this could be a single location, where the publisher used a different name format. 🤷‍♂️

Next steps of my Now You’re An ETL Programmer phase

1. Write a parser to extract the sub-locations, from location “names”.
2. Store locations and sub-locations properly in a locations table.
3. Cleanup and migrate the existing data.
4. Make a useful “product” (plot delivery dates per location(s), provide notifications, etc.).

If you’ve read through all of this, sorry there’s no big insight. Just my brain trying to map real world stuff to bytes. Part of me wants to see if I can offload task #1 to an LLM: with a sufficiently intricate prompt, it could work, but I’m not sure I can trust the machine here, because the text is sometimes quite ambiguous. Being surprised would be nice, because the alternative is a hand-written parser with lots of unit tests.

Hi there! Long time no see. Not that I expect to publish frequently these days. This is the pitch for a little open-source project that’s kept me busy a few evenings.

Some months ago, we moved back to the city of Oaxaca. We’ve lived in the countryside nearby for years, but it was soon very clear that “nearby” was actually pretty far when thousands of people try to use the same roads at the same time, to get to work or school, and public transport is shit.

I can’t fix that though, so we found a house 10 minutes from school, and stay in the city a good chunk of the year now. With that change, we became again dependent on the city’s water distribution network. Something with a pretty name like Sistema Operador de los Servicios de Agua Potable y Alcantarillado (de Oaxaca). Usually, they’re online here, but not all the time. 🥲

This public organization manages the water distribution network of the whole urban area of Oaxaca. For various socio-ecological reasons, water doesn’t run all the time in the city (or state) of Oaxaca. You’re expected to store whatever amount you receive monthly (-ish). There’s not enough water (or pressure) to distribute to everyone at the same time, so not all districts receive water on the same day. (You should not drink that water by the way.)

The distribution schedules are only advertised on X/Twitter, and Facebook. This is annoying, because the information is never really archived anywhere, and is published as JPEGs images (a workaround for text lengths limit, I guess). That’s not too friendly for visually impaired folks either. In Oaxaca, you do with what little resources you have.

I like to know when my home will receive water, and I’d also like to know when it did previously, to better manage my consumption. I don’t want to use Twitter/X, or Facebook though. I use them as little as possible, with a hazmat suit, and tears in my eyes.

So, that’s why I started this thing called Aguaxaca. It is an open-source project, that gathers water distribution schedules through a private Nitter instance. The images are parsed using a normal OCR an LLM, and the schedules are stored and published for all to see — in particular me, but I consider this a minor public service, not something that will turn a profit.

The goals are simple:

1. archive this stuff,
2. get notifications when it’s my turn (essential),
3. get historical records and plot some stats (would be nice),
4. compute the likely date of next delivery (would be extra nice).

I’m currently in “phase 1”, discovering the complete list of distribution sectors. The public data I scrape isn’t exactly great, or consistent, but with some genAI prompt magic, I should be able to turn this mess into a proper DB. That’s the plan anyway!

Also yes, the name… sucks, but it’s easy to remember if you know a few words of Spanish. If you got this far, thanks for reading.

Recently, I learned from MySQL 8.0 docs, that:

If you want to make your queries as fast as possible, look out for Extra column values of Using filesort and Using temporary, or, in JSON-formatted EXPLAIN output, for using_filesort and using_temporary_table properties equal to true.

So, I guess that I should probably stay away from these filesort and temporary table things. That sounds fair 🤔

You already know where this is going, but let’s see what EXPLAIN has to say about this suspiciously large query.

> EXPLAIN select '' -- Lots and lots of gibbe^Wperfectly valid SQL.
+----+--------------+-----------------------------------+---------------------------------+
| id | select_type  | table                             | Extra                           |
+----+--------------+-----------------------------------+---------------------------------+
+ Truncated...
| 13 | UNION RESULT | <union1,2,3,4,5,6,7,8,9,10,11,12> | Using temporary; Using filesort |
+----+--------------+-----------------------------------+---------------------------------+
25 rows in set, 1 warning (0.10 sec)

Oh, that doesn’t look too great (also, sorry if you’re on mobile). I sure hope no one runs that query in a loop, or things could go wrong, right? Right?

I kind of warned you about making sounds with the Preonic keyboard, because 80s beeps will never go out of fashion, and these clicky switches aren’t noisier enough anyway. Check the previous post if you need to, this is a follow up.

Playing sounds is very simple with QMK, thanks to the cute PLAY_SONG macro, to make the board play very geeky boops.

Since we started from the default keymap and config for the v3 preonic, we already have audio enabled on the board. There’s nothing else to configure: as far as I know we can already beep to our heart’s content.

In QMK, we have a C macro SONG(list of notes). It gives us back an array, that we can feed to PLAY_SONG to actually play sounds. Of course these are C macros, so we can’t really inline things like this: PLAY_SONG(SONG(TADAAA)). NOPE!

No big deal. We’ll use an intermediate variable to store the “song”, and feed that to PLAY_SONG. For our purpose, let’s reuse a song from song_list.h (this file). Writing your own stuff is fun, but it’ll keep this post short.

I’ll use the CAPS_LOCK_ON_SOUND (and off) sounds, which seem half-appropriate.

Let’s declare these two songs in keymap.c:

#ifdef AUDIO_ENABLE
static float caps_word_on_song[][2] = SONG(CAPS_LOCK_ON_SOUND);
static float caps_word_off_song[][2] = SONG(CAPS_LOCK_OFF_SOUND);
#endif

Now, we could update the previous post’s tap-dance callback to play sounds, but the caps-word feature is more ergonomic than that. Just use the caps_word_set_user callback. It gets a “bool” to indicate the current state. Again, in keymap.c:

#ifdef AUDIO_ENABLE
void caps_word_set_user(bool active) {
	if (active) {
		PLAY_SONG(caps_word_on_song);
	} else {
		PLAY_SONG(caps_word_off_song);
	}
}
#endif

The whole #ifdef wrapping isn’t essential here, but the function only matters when audio is enabled in my case. YMMV.

That’s it. See, THAT WAS NOT THAT HARD. Sorry, just testing. :)

If you’re into mechanical keyboards, you probably know about Tap dances and Caps words already. If not, well these are two features of QMK:

• a tap dance allows you to remap rapid successive taps on a key to a different function (ref), and
• caps word is a function that turns on CAPS but only for the next word you’ll type (ref).

As the Tap dance docs clearly state, a Tap dance can trigger basic key codes only: meaning letters, modifiers, etc., but not Caps words. I briefly experimented on my Preonic, trying to get the backtick key to turn on Caps word, using the basic Tap dance macros, but it failed to trigger anything. Frustrating.

Either I’m holding it wrong, or the docs are correct. Both things are equally possible. :)

But fear not dear reader! Of course you can trigger Caps word from a Tap dance. You’ll just need to write a bit of C, to customize the board’s keymap.

Let’s look into this here.

First, change rules.mk to enable both features, adding:

CAPS_WORD_ENABLE = yes
TAP_DANCE_ENABLE = yes

I also like to have an explicit tapping-term, which defines how long the board waits for your taps to decide what to do next… I’m not sure whether that’s a good definition, but that’s how my brain explains it.

Change config.h to define this as needed, for example:

#define TAPPING_TERM 200

This means you have 200ms to double-tap your keys. This tapping term is also shared with Tap-or-hold custom keycodes, so maybe you will need to figure out per-key tap terms later. YMMV.

The meat of this little hack lives inside keymap.c.

Let’s add a cur_dance helper that will tell use where we’re at in our Tap dance:

// Some Tap Dance states
enum {
	SINGLE_TAP = 1,
	SINGLE_HOLD = 2,
	DOUBLE_TAP = 3,
	DOUBLE_HOLD = 4
};

int cur_dance(tap_dance_state_t *state) {
	if ((state->count == 1) && (!state->pressed)) return SINGLE_TAP;
	else if ((state->count == 1) && (state->pressed)) return SINGLE_HOLD;
	else if ((state->count == 2) && (!state->pressed)) return DOUBLE_TAP;
	else if ((state->count == 2) && (state->pressed)) return DOUBLE_HOLD;

	// magic number. At some point this method could expand to work for more presses
	else return 5;
}

The logic is pretty simple. For our use case, we also need 2 functions to hook in the finished and reset callbacks of a Tap dance. Let’s add these next:

// Tap-dance is over.
void dance_cw_finished(tap_dance_state_t *state, void *user_data) {
	int dance = cur_dance(state);

	if (dance == SINGLE_TAP) {
		register_code(KC_GRV); // single tap -> backtick key
		reset_tap_dance(state);
		return;
	}
	if (dance == DOUBLE_TAP) {
		caps_word_on(); // double tap -> caps words
		return;
	}
}

// Clear the tap-dance floor.
void dance_cw_reset(tap_dance_state_t *state, void *user_data) {
	if (state->count == 1) {
		unregister_code(KC_GRV); // Stop sending backtick.
	}
}

What’s left is some boiler-plate to register our handlers as a Tap dance, named TD_GRV_CW, which is a pretty bad name if you try to pronounce it. It did made sense at the time:

// Backtick or Caps-word tap-dance.
enum {
	TD_GRV_CW,
};

tap_dance_action_t tap_dance_actions[] = {
	// Tap once for ` (backtick), twice for Caps-word.
	[TD_GRV_CW] = ACTION_TAP_DANCE_FN_ADVANCED(NULL, dance_cw_finished, dance_cw_reset),
};

On your board configuration, you can replace KC_GRV with TD(TD_GRV_CW), then compile and flash as usual, now double tapping the backtick key will activate Caps-word, rejoice.

Done?

Dancing is nice(r) with music. My rev3 Preonic has a (possibly annoying) speaker. Wouldn’t it be nice to play a few notes as you activate a tap dance? I’m sure everybody around would love it.

Also, what if I want to have a Tap dance on a different key like Left Control or something?

So many possibilities… Maybe I’ll write about that here next, in a couple of years if this site’s post frequency is stable.

Thanks for reading.

MailDir Stat (mdstat) is a small program I wrote some time ago to play with zig: it lists MailDirs, and optionally only those that contain unread emails. Nothing incredibly new, just a small itch to scratch.

It came from a small frustration with (neo)mutt where I sometimes like to have a tree of mailboxes in the sidebar ; but I only want to see the unread ones. I synchronize various IMAP accounts locally with mbsync, so finding maildirs is easily done with a bit of shell scripting.

For instance, this is how I listed all maildirs until recently:

fd . /home/oz/mail/cyprio --type d | \
  rg 'cur/$' | \
  sed 's,^/home/oz/mail/cyprio/\(.*\)/cur,=\"\1\",' | \
  tr '\n' ' '

This works well enough to tell neomutt to check all the maildirs of my account, and you can get more creative to ignore some directories, etc. With mdstat, this is just:

mdstat ~/mail/cyprio

Personally, I find that it’s an improvement ; but also maybe, a senseless micro-optimization, you know just for fun.

Sidebar and unreads

My mail account(s) often have nested folders, and will until the day I’m forced to properly use notmuch or something, and use labels like the rest of the world. I don’t know.

Still. As I use the sidebar plugin, I’ve realized that the “sidebar parser” isn’t (wasn’t?) clever enough to display a tree unless explicitly asked to. Let me try to explain.

Let’s say that I have new emails under ="news/progamming". To display a nice 2-levels tree with both the news, and programming labels correctly nested, the sidebar config expects to receive both entries like so: ="news/" ="news/programming" (order doesn’t actually matter).

If you just list ="news/programming", you’ll get:

  programming/

If you list both entries, you get:

news/
  programming/

No flying folder without context, nicely indented, which makes my brain work better. 😌

It’s possible to do that with more shell scripting, but it quickly gets dirty when you try to deal with deeper trees, avoiding duplicates, and well it also gets slower because it’s a shell script.

This is my excuse for writing a program that builds maildir trees. In that example, mdstat ... -u will list both news and news/programming when there’s unread mail in the bottom folder ; even if news is not a proper maildir (which is a feature).

If you’re curious, the code is on git.cypr.io with other more or less personal projects. Of course it’s way faster than forking 4+ shell commands.

About zig

I’m not sure that I’ll write much more zig code in the future.

It’s not that I don’t enjoy it, quite the contrary! The language seems simple, and fairly easy to pick up if you’ve ever dealt with C, Go, or Rust. It firmly sits somewhere between C and Go in my mind ; which is a space that I don’t often need to work into… but who knows?

It’s interesting that there’s still a post about joining MailPoet on the homepage. It makes this place feel very dusty, full of cobwebs, and memories, and this post kind of ironical.

I stayed at MailPoet for more or less 2 years. I remember feeling exhausted and depressed at the end of it. A nice burnout to use the proper term: yes, you could get burnout before the COVID pandemic. These were the days! That was mostly my fault for not seeing it coming, but not entirely.

Now what’s new? After some freelancing gigs here and there, I’ve joined back MailPoet ; now part of that “little” company that makes a well-known CMS. I still don’t use it to publish these few rants, or become rich with my newsletter, but it’s running somewhere where I share some pictures of the kid, from time to time.

I’m still not a big fan of PHP (and not writing any, which should be reassuring for the knowledgable PHP folks), but I really like working with open-source, and making tools that solve the problems of actual people.

Let’s try to avoid burnout this time because I feel too old to job-hop like a jumpy millenial ; and there’s still work to do.

Don’t you ever tire of benchmarks? Most of those are inaccurate, hardly reproducible, and often misleading. With that out of the way, I present a benchmark of some very fine programs to compare the speeds of Go, Crystal, and Ruby. ;)

I like Ruby a lot, have used Go for some years, and I consider that Crystal is an interesting middle ground: both nice and fast, although not as mature as the two others, yet.

I recently stumbled upon Guirec Corbel’s benchmark (in French) of the three technologies, written in 2015. I though it would be interesting to check the results a year apart, but we could hardly compare our benchmarks since we’re not using the same hardware, and so I just re-benched everything.

I did some guess work to select the same versions that he probably benched in September 2015, and proceeded to rerun these on my ancient (2011) laptop. This way, we should get a more accurate idea of the evolution of our tools.

I ran these benchmarks from Archlinux, on this CPU: Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz. The programs are sourced from @kostya’s benchmark repository: not all implementations are optimized, but they tend to show how a language performs when the code is more or less identical. Furthermore, I had to dig up old Crystal code to run on 2015’s version (0.7).

For each program, I simply measured the running time, taking the best of 5… If you want averages and variance, pull-requests are warmly welcomed.

Results 2015

We compare the execution time of 4 little benchmarks using:

• Go 1.5 (released 08/2015)
• Crystal 0.7.7 (released 09/2015)
• Ruby 2.2.4 (released 12/2015)

Now fear my mad GNUPlot skills:

OK, let’s just remove Ruby here:

Results 2016

We compare again the execution time of the same 4 benchmarks using:

• Go 1.7.4
• Crystal 0.20.1
• Ruby 2.4.0-preview3

Again, without Ruby:

Comparing speed gains

These numbers do not look like much, but let’s put them on a graph to compare the gains. On this graph, 100% means 2015 speed, lower means worse performance, so higher is better:

It’s easier to grasp how these platforms are all¹ getting better.

These benchmarks only solicitate one CPU core, so they are a small part of a bigger story where concurrency and parallelism play a major role. Having said that, I’d really like it if Ruby’s MRI would get faster faster

I wonder if more Rubyists will switch to Crystal: clearly, the platform is just getting warmed up, and will probably get speedier with time. It is already on par with Go’s speed (even slightly faster), has not even reached a 1.0 version, and yet continually improves.

Courting Ruby developers with speed only may not work: after all, it took Rails to make the language successful.

To me, Go has gained the safer-choice award after several releases. It is a stable platform, continually improving, thanks to the backing of Google. Also… channels are <3. :)

The question we Rubyists should ask ourselves is maybe which tool should we use when our next project hits a performance limit, safe of fun?

I have put my startup-y projects aside for a while now. There is only so much time that I would invest in something that won’t pay its server bills, no matter how fun it is. Still, not easy.

Anyway, to move on I joined MailPoet a few months ago, taking over a project that had been entirely developed by one person, using a technology that no one else there knew very well (why?). That one person decided to leave. Rather than let the project die, MailPoet decided to hire a new person: I have a few years of experience doing ruby and devops, so the role fits me quite well.

Furthermore, I am used to 100% remote work, the team is nice, the to-be-announced project is as awesome as it is challenging. New team, new project, new things to learn. So far, so good.

So you know, new job is nice and all, but I will focus on more important things for now, like my new born daughter. :)

Like a failed communist state that equalizes by spreading poverty, Scrum in its purest form puts all of engineering at the same low level: not a clearly spelled-out one, but clearly below all the business people who are given full authority to decide what gets worked on.

Why “Agile” and especially Scrum are terrible