Corinne's 8th Birthday

March 19, 2023 3:19 pm

Sometime last year Heather had asked if she could help with the next birthday adventure. So as Corinne's was coming up I asked if she still wanted to help. She was very excited to plan an entire adventure for Corinne.

Heather was sick the final weekend before Corinne's birthday, so Jess and I ended up assisting with final prep work. But, in general, the entire thing was Heather's design with some consulting from me to smooth out rough spots and clarify tasks.

It all revolved around a story book which Heather wrote and assembled. The book provided the narrative and character interactions and Corinne was instructed to turn to specific pages during the adventure to continue the story.

The overall theme was a My Little Pony adventure. She helped each of the ponies with some task which provided her with clues to the encoded message Pinkie Pie found where Corinne's presents were supposed to be. Once all the clues were collected it became clear that Queen Chrysalis had ordered the presents taken. With the assembled ponies and the collected elements of harmony she was able to defeat Queen Chrysalis and recover her presents.

Reading the introduction in the story book:

Baking a cake with Pinkie Pie revealing the first 2 clues to the coded message:

Helping Fluttershy rescue stranded animals revealed the number 13 which was the next page in the adventure:

Apple Jack needed her baskets, but couldn't remember how to disable the anti-pranking alarm system. Corinne needed to retrieve them without touching the streamers and found a note with more clues.

Rarity couldn't remember the combination to her lock where she kept her notebook and Corinne needed to sew the pattern which Rarity created as a back up. Once sewn, the pattern marked the numbers 3-1-7-8.

She also needed to clear the clouds that Rainbow Dash forgot to take care of, which recovered another clue and Twilight Sparkle knew a book with helpful information in it (not shown).

Queen Chrysalis was confronted at her hive (in the backyard) and the presents were recovered from the bench.

Then it was time to open presents! At the moment she is all about Squishmallows and reading.

We went to dinner at her choice of restaurant, which was Taco Bell. And after dinner, cake!

Jess whipped up the crochet crown for her to wear to school since she didn't have anything super green to wear and wanted something.

Celery & Redis countdown/eta oddities

March 9, 2023 10:13 am

One of my projects at work uses the Python package Celery with Redis to manage executing background tasks. And we ran into some odd behavior that we didn't see explained anywhere else, so I figure I'll capture it here for the next poor soul running into these issues.

First, if you care about this subject, you should read this post over at Instawork which is a good discussion of the risks involved in using countdown and eta. It helps set the stage.

Setup

We're using Celery with a Redis broker as part of a Django application. We apply one of 3 priorities to each of our tasks: Low, Medium, and High. High-priority tasks represent things that a human user is waiting on and need to be completed as soon as possible. Low-priority tasks are things that need to happen eventually, but we don't really care when. And anything else gets configured as medium priority.

This set up worked in our validation testing. We saw the queues get loaded up in Redis and the workers execute tasks in priority order as expected.

The Wrong Queue

After a large-scale data-processing task we noticed that high-priority user tasks were not executing.

When I inspected the queues in Redis I found that the high-priority queue was full of low-priority tasks. So the workers were extremely busy (correctly) processing the queue, but the tasks they were running were low priority. And the human's task was stuck behind them all.

How did this happen?

Countdown/ETA Reservations

The first part of the puzzle is how Celery handles countdown / eta tasks. countdown allows you to say "execute the task 5 minutes from now" while eta allows you to say "execute the task no earlier than March 10, 2023 at 10:08AM."

countdown is purely syntactic sugar for eta so that you don't have to calculate actual times yourself, so when you call apply_async with a countdown parameter Celery converts it to an eta parameter. Since internally Celery only concerns itself with eta values we'll only talk in terms of eta from this point on.

When an eta task is generated it gets put into the appropriate priority queue. But, it doesn't stay in the queue until its eta passes. Instead, any worker checking the queue will reserve the task immediately and hold it internally until the eta passes.

During my investigation, while the workers were idle, I scheduled a few hundred tasks with an eta and, as a result of the above behavior, the priority queues in Redis were empty. Workers will continue reserving eta tasks from queues until they have a task that needs to actually execute now. Once the workers are busy, eta tasks will stay in their appropriate queues until a worker is freed up and comes looking again.

Processing Reserved Tasks

Alright, so our workers have reserved all of our eta tasks with varying priorities and now the tasks are starting to pass their etas and need to be executed. At this point the worker completely ignores the priorities on the tasks. It begins executing whichever reserved task it happens upon first in its internal data structure (this is probably an internal queue, but I don't know for sure).

So once a worker reserves a task its priority is no longer respected. If you schedule a few hundred eta tasks with mixed priorities (as I did) you see them executed in what appears to be an arbitrary order (I suspect they're actually executed in order they were reserved, but I haven't verified that because it's not relevant to my concerns).

This is not good and reason enough to avoid using eta tasks for anything but high-priority tasks. But, it doesn't explain how we ended up with low-priority tasks in our high-priority queue in Redis.

Death of a Celery Worker

We have Celery configured to replace each worker after completing 10 tasks. This was an attempt to work around an issue where workers would stop pulling tasks from the queue and everything would stall out. We had a hypothesis that the issue was unclosed connections to Redis and so replacing the workers would force unclosed connections to get cleaned up. We haven't yet verified what was actually happening or if replacing the workers fixed anything though. It's a very intermittent problem and we haven't identified a sure trigger. (Though we did solidly identify that if Redis isn't ready to serve connection when Celery starts then Celery will not reconnect properly and workers will only execute a single task before hanging forever.)

Anyway, the point is that we replace our workers every so often. Well, what happens to all those eta tasks the worker had reserved? They go back in the queue so another worker can get them. But, they all go into the default queue instead of going back to their appropriate priority queues. It happens that the default queue is the high-priority queue. So each time a worker was getting cycled, all the eta tasks it held were pushed into the high-priority queue.

The perfect storm

So here's the scenario. Our workers are sitting idle waiting for work to do. Our large-scale data-processing task schedules a bunch of low-priority jobs with an eta. The workers eagerly snap up all these tasks and reserve them for future execution. As soon as the earliest etas pass each worker begins executing and stops reserving more eta tasks from the low-priority queue.

Each worker completes 10 tasks and gets replaced. As each worker is replaced it returns the remainder of its eagerly-reserved eta tasks to the high-priority queue. The new workers being spawned now begin processing the high-priority queue since it's full of tasks.

A user comes along and engages in an action backed by a high-priority task. But the user's high-priority task is now stuck behind several thousand low-priority tasks that have been misplaced in the high-priority queue.

Moral of the story

We had run across the countdown parameter to apply_async and thought it would be a good way to avoid some unnecessary work by pairing some high-churn jobs with a flag so they'd only be scheduled again if they weren't already scheduled (this flag was managed outside of the Celery world).

We will be rolling back that change so as to avoid this situation in the future.

My First Book in French

January 19, 2023 7:48 pm

On our trip to Quebec last year I bought a few French books at a bookstore. "La Planète des Singes" (The Planet of the Apes) was the shortest one (only 190 pages) and I started reading it sometime in November I think. I usually only read it while sitting at gymnastics or swim class. Since it was notable mental effort to read it worked well to focus on it in small doses. Instead of reading a few pages a minute I was reading a few minutes per page.

I was pleasantly surprised to find that I could understand probably 50-75% of the material. Enough that I followed the story line all the way through and could provide a high-level synopsis, but I definitely missed details and nuance. And I also learned several words by context. Some were pretty obvious: "gorille," "chimpanzé," "orang-outan." But others were less so, like "chaloupe," meaning "shuttle/skiff" which took them from their spaceship to the planet's surface.

Probably the most satisfying moment was near the beginning of the story when I determined the characters were discussing the time-dilation effects of general relativity. I figure I must be doing okay with my progress if I can identify general relativity being discussed in French.

Christmas 2022 Board Game Roundup

January 8, 2023 7:47 pm

We got a bunch of board games for Christmas this year. Here's my quick rundown and roundup of the games. I'll give each game my own, personal, first-impression rating and an inferred rating from Heather and Corinne based on my interpretation of how much they enjoyed the game.

Exploding Kittens (2015)

Kyle's Rating: 2/5 (simple and highly random)
Girls' Rating: 5/5 (silliness, and EXPLODING KITTENS!)

Fairly simple competitive, player-elimination card game. Every turn you draw a card from the deck, if it's an exploding kitten, you're out. Last player unexploded wins. You have a hand of cards which allows you to modify the game flow and potentially defuse a kitten. It's short, it's easy, the instructions are clear. The girls like it. We played it several times inside the air fort. The girls played it with friends that came over and taught it without assistance.

Happy Little Dinosaurs (2021)

Kyle's Rating: 1/5 (terrible instructions, probably higher rated if taught by someone else so you don't have to deal with the instructions)
Girls' Rating: 3/5 (cute and a bit silly)

You're a stressed out dinosaur trying to survive the ongoing apocalypse around you. You'll face various calamities and do your best to stay alive. This is also a competitive, player-elimination card game, but slightly more involved than Exploding Kittens. You have a hand of cards that may enable you to better survive (or throw your fellow dinosaurs under the proverbial bus). Be the last dinosaur alive to win (or be first to escape the apocalypse by moving to the end of the track).

The game is pretty straightforward, and the artwork is adorable, but the instructions are absolutely abysmal. Some of the worst I've ever seen. I think the clearest way to describe their failure is that they seem to describe the game from a detached observers view--like an anthropologist describing what's happening but not understanding why its happening. So when you read them you can understand the "appearance" of the game, but not any of the motivation for why you're doing things. Once you get past that, the rules are actually quite simple.

Perhaps it was the pain suffered from attempting to decipher the instructions, but we were not particularly impressed with this one after a few play-throughs.

Camel Up (2nd edition, 2018)

Kyle's Rating: 4/5 (clear rules, analyzing probabilities provides some depth)
Girls' Rating: 4/5 (wacky camels and light hearted)

You're in Egypt gambling on the camel races, but this race is a little....different. The camels climb on top of each other and move in stacks and a couple of camels are running the wrong way around the track.

The instructions are well written and easy to follow to get set up and playing. You take turns either making bets or moving the camels (by dropping a die out of the pyramid). Once the race is over the player with the most money wins. I think Jess is a little annoyed playing with me since I'm able to analyze the probabilities fairly readily and made good bets. The girls have liked it and Corinne, after insisting she didn't want to play, loves it.

It's silly, a bit whacky, very much non-serious, and fairly quick to play.

Paint the Roses (2022)

Kyle's Rating: 4/5 (going to especially appeal to logic players)
Girls Rating: n/a (haven't played with the girls yet)

You're gardeners for the capricious and violent Queen of Hearts. She's given each of you different instructions on how she wants her garden arranged. You must work together and use deductive reasoning to figure out what the Queen has commanded your fellow gardeners to do while they do the same for you. Keep up with the ever-changing whims of the Queen or it'll be "Off with your head!"

You place new plants in the garden to communicate to the other players what instructions you've been given and/or learn about what instructions they've been given. We lost on the very last turn when the Queen caught us and chopped our head off.

Very nice artwork with detailed figurines for the Queen, the gardeners, and the White Rabbit.

Jurassic Park: Danger! (2018)

Kyle's Rating: 3/5 (dinosaurs are too smart, should be hobbled somehow)
Girls' Rating: 3/5

Jurassic Park is frightening in the dark
All the dinosaurs are running wild
Someone shut the fence off in the rain

One player controls the dinosaurs, hunting down the humans on Isla Nublar for sport. The other players are those humans desperately trying to get the park operating well enough to call for help and escape.

I played the dinosaurs and I think the biggest flaw in this game is that the dinosaurs are too smart. Since the dinosaurs know the objectives of the humans they can make strategic decisions to deny access to key parts of the board. This seems to be required to keep the game balanced as designed, but it also means the dinosaurs are unnatural and it takes away from the atmosphere.

I'm thinking about playing around with rule modifications to force the dinosaurs to make more "mistakes" and feel more natural to the humans. Even something as simple as "roll a die to determine which of your 3 cards you use this turn" would help.

I like the theme and board design should make for good replayability.

My Little Pony: Adventures in Equestria (2022)

Kyle's Rating: 3/5 (mechanics feel a little clunky)
Girls' Rating: n/a (haven't played through a complete game with them yet)

There's trouble in Ponyville and you need the magic of friendship to set things right. In this cooperative game you'll play as one of Pinkie Pie, Rarity, Twilight Sparkle, Applejack, Rainbow Dash, or Fluttershy and work to acquire the necessary resources to clear some hurdles and complete the final challenge.

I think this game struggles a bit on what it's trying to be. It bills itself as a deck-builder, but you don't have time to do as much deck building as in most other deck-builder games. There's a strict time pressure (not wall-clock time, but per-turn events) pushing you towards defeat. The result is that you don't have a lot of time to build-up your deck--if you try you'll lose. Instead you need to use a fairly aggressive play style to stay on top of things to win, which isn't my preference. I usually play more casually, but you will definitely lose this game if you do that.

There's a somewhat awkward "move" mechanic in the game where you need to expend resources to move your standee from one place to another. I feel like this could have been dropped entirely without losing anything related to the core game mechanic.

The Night Cage (2021)

Kyle's Rating: 5/5 (unique mechanics and well executed theme)
Girls' Rating: 3/5 (felt maybe a little too creepy)

You awake in an endless labyrinth with nothing but a flickering candle to light your way. Working with the other prisoners you must find a key for each person, find a gate, and meet there to use your keys to unlock the gate and escape. But beware, only the light of your candle keeps these walls stable. Any time a passage is not being illuminated by a candle it disappears and will change when next seen.

I heard about this game while looking for good Halloweeny games to play. I bought Horrified last October and put this on the wish list. Horrified is campy and light-hearted. The Night Cage is dark and creepy.

The constantly evolving board is well executed. And the candle theme is effectively integrated throughout the game. A little out of place in early January, but I look forward to playing it in October. The instructions are well written and the turn actions are clear. There's also an "advanced" game mode which we haven't tried.

Wingspan (2019)

Kyle's Rating: 5/5 (super chill, beautiful artwork, clear mechanics)
Girls' Rating: 5/5

Develop an ecosystem to support a variety of birds in your wildlife preserve. Manage food, eggs, and space to grow your population.

Players earn points for the different types of birds, number of eggs, and other specific goals. The player with the most points at the end of 4 stages wins. There's a lot going on, but the core game loop is easy enough. The complexity comes from how the base mechanics interact with each other as you try to expand your preserve.

I always find simple mechanics that combine to produce emergent complexity to be very satisfying. Too many games add complications to make a game seem more complex, but if not executed well it feels clunky (I think that describes My Little Pony, above).

Jess learned how to play and taught this one to the rest of us, so I can't comment on the instructions directly, but it also came with a learning tool in which it tells each player exactly what to do for their first 4 rounds to help you figure out what's happening. That feature was very nice. By the end of your 4th turn you have at least a vague idea of why you'd take each of the actions available to you and you're ready to fly solo.

The artwork is beautiful and the gameplay is very chill. Ostensibly you're playing against each other, but you're mainly just doing your own little thing collecting birds and reaching goals and then you compare scores at the end (though you could be aggressive about monitoring what everyone is doing and work to hinder them).

Splendor (2014)

Kyle's Rating: 4/5 (logical analysis and engine building, but also fairly shallow)
Girls' Rating: 3/5

You're a renaissance-era aristocrat looking to expand your influence and power. Acquire resources and grow your empire while gaining the attention and loyalty of local nobles.

This game feels like what you'd get if you boiled down 7 Wonders to a single core mechanic: Acquire low-level resources which will enable you to acquire higher-level resources until you've accumulated enough points to win. Conversely, 7 Wonders would be what you get if you built on this core mechanic to produce more depth and (somewhat counter-intuitively) speed up the game.

Jess also took on learning this one. It seemed to go well, the rules are straight-forward (since it's built around a single mechanic). Easy to learn, easy to play. I always enjoy a game where you get to watch your power build and the engine-building in this scratched that itch well.

There are several expansions that, presumably, increase the depth and complexity.

Mint 21 + Kodi 19 + Intel NUC i3

2:24 pm

The latest in my series on the subject, this follows the last upgrade in 2018: Mint 19 + Kodi 18 + Intel NUC i3.

Not much has changed in how I got my set up the way I want since 4 years ago. There is a minor change on how to get the TrueHD and DTS-HD MA audio passthrough working.

Update NUC’s UEFI

I grabbed the latest firmware from Intel (version 54) and flashed it onto the NUC. One issue I ran into (which presumably isn't new, just not noted previously), is that the NUC wasn't negotiating a display signal through my receiver and I had to plug it directly into the TV to see anything in order to do the firmware upgrade.

Next I went on a wild goose chase because after upgrading the firmware nothing could see the internal mSATA drive on which the OS is installed. After an hour and a half of messing with it (including downgrading the firmware) I eventually disconnected everything, opened it up, re-seated the drive, and put it back together and everything was fine. No idea what that was about.

Install Linux Mint 21.1

I then installed Mint 21.1.  I won't cover that in this post.  Plenty of better places to find that information.  I'm using the 64-bit distribution.

Install Kodi 19

The Kodi Wiki install guide is pretty clear:

$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:team-xbmc/ppa
$ sudo apt update
$ sudo apt install kodi

ALSA, PulseAudio, TrueHD, DTS-HD MA

Since last time some minor details about how to accomplish forcing Kodi to use ALSA have changed. I still use a custom session launcher, but instead of rather forcefully killing pulseaudio it uses pasuspender and sets an environment variable for Kodi.

Create our custom session launcher:

$ nano ~/kodi_custom_session_launcher

Paste in:

#!/bin/bash
pasuspender -- env KODI_AE_SINK=ALSA kodi

Make the launcher executable:

$ chmod +x ~/kodi_custom_session_launcher

Now create the custom XSession entry:

$ sudo nano /usr/share/xsessions/Kodi_ALSA.desktop

And paste in:

[Desktop Entry]
Name=Kodi with ALSA
Comment=This session will start KODI Media Center
# Note: Change "kyle" to your username
Exec=/home/kyle/kodi_custom_session_launcher
TryExec=/home/kyle/kodi_custom_session_launcher
Type=Application

Now if you log out you should be able to select "Kodi with ALSA" as a session option from your login manager.  In Mint, this is accomplished by clicking the circle to the right of the user name.

After a little fussing and some restarts I eventually saw the correct audio targets in the Kodi settings and everything seems to be happy with the audio passthrough support.

Kodi Configuration

I setup some stuff in Kodi's advanced settings file to disable the splash screen, hide the ext4 file system's "lost+found" directories, and increase the network streaming buffer.

Create the file:

$ nano ~/.kodi/userdata/advancedsettings.xml

And paste in:

<?xml version="1.0" encoding="UTF-8"?>
<advancedsettings>
  <splash>false</splash>
  <video>
    <excludefromlisting>
      <regexp>lost\+found</regexp>
    </excludefromlisting>
    <excludefromscan>
      <regexp>lost\+found</regexp>
    </excludefromscan>
    <excludetvshowsfromscan>
      <regexp>lost\+found</regexp>
    </excludetvshowsfromscan>
  </video>
  <network>
    <cachemembuffersize>15728640</cachemembuffersize>
  </network>
</advancedsettings>

NUC's Remote Control IR Receiver

To use the NUC's built-in IR receiver we need to install the ir-keytable utility:

$ sudo apt install ir-keytable

Then we define the keytable we need for the Onkyo RC-764M remote using the Xbox keycode 33003.  Create the file:

$ sudo mkdir /etc/rc_keymaps
$ sudo nano /etc/rc_keymaps/onkyo_rc-764m_nec_33003

And paste in:

# table onkyo_rc-764m_nec_33003, type: NEC
# Using Onkyo Remote Code 33003
# Code mappings match same buttons when using Original Xbox Dongle Remote Code
# ScanCode LIRC_KEY # Remote button text
# LIRC_KEY is modified to get correct action in XBMC using LIRC-in-kernel fake-keyboard actions

0x2d2d3a KEY_I #Display
0x2d2d59 KEY_LEFT #Left
0x2d2d5a KEY_RIGHT #Right
0x2d2d47 KEY_UP #Up
0x2d2d48 KEY_DOWN #Down
0x2d2d4a KEY_C #Guide/Top Menu -- Context Menu
0x2d2d4b KEY_ESC #Prev CH/Menu
0x2d2d45 KEY_BACK #Return
0x2d2d56 KEY_O #Setup -- Codec Info
0x2d2d58 KEY_ENTER #Enter
# UNUSED 0x2d2d4f #Audio
0x2d2d35 KEY_COMMA #Skip Left
0x2d2d34 KEY_PERIOD #Skip Right
0x2d2d32 KEY_R #Rewind
0x2d2d33 KEY_F #Fastforward
0x2d2d31 KEY_P #Play
0x2d2d38 KEY_SPACE #Pause
0x2d2d39 KEY_X #Stop
0x2d2d3b KEY_1 #1
0x2d2d3c KEY_2 #2
0x2d2d3d KEY_3 #3
0x2d2d3e KEY_4 #4
0x2d2d3f KEY_5 #5
0x2d2d40 KEY_6 #6
0x2d2d41 KEY_7 #7
0x2d2d42 KEY_8 #8
0x2d2d43 KEY_9 #9
0x2d2d44 KEY_0 #0
# UNUSED 0x2d2d4e #+10
# UNUSED 0x2d2d46 #CLR
# UNUSED 0x2d2d7c #Search
# UNUSED 0x2d2d7d #Repeat
# UNUSED 0x2d2d7f #Random
# UNUSED 0x2d2d7e #Play Mode

Now configure a systemd service to load the keytable at boot.  Create the service file:

$ sudo nano /etc/systemd/system/ir_receiver.service

And paste in:

[Unit]
Description=Configure the IR Receiver for the desired keytable

[Service]
Type=oneshot
ExecStart=/usr/bin/ir-keytable -c -p NEC -w /etc/rc_keymaps/onkyo_rc-764m_nec_33003

[Install]
WantedBy=multi-user.target

ExecStart needs the absolute path to the ir-keytable executable.  Here's what the command options do:  "-c" clears the existing keytable.  "-p NEC" puts the receiver in NEC mode. "-w /etc/rc_keymaps/onkyo_rc-764m_nec_33003" loads our custom keytable.

And enable the new service:

$ sudo systemctl enable ir_receiver.service

Bonus: Logitech Media Server

I also run Logitech Media Server for the fleet of Squeezebox Radios in the house.  I

The information for LMS is a mess.  Lots of out-of-date information and lack of clarity on recommended approaches.  The wiki linked to in the previous post is now out of date.

However, the package repository still seems to be the right place. This XML file contains the details on the latest release: http://downloads.slimdevices.com/releases/latest.xml

I grabbed the 8.3.0 DEB release for amd64 from: http://downloads.slimdevices.com/LogitechMediaServer_v8.3.0/

$ sudo apt install ./logitechmediaserver_8.3.0_amd64.deb

Misc. Errata

My external USB3 hard-drive enclosure (a Mediasonic ProBox HF2-SU3S2) had been working rock solid under Mint 19. Unfortunately now it's randomly being disconnected. It immediately reconnects and I can re-mount the drives, but it's a major annoyance. The transfer speeds are about 50% faster than before (~120MB/s vs ~80MB/s, so we're hitting the very max of my gigabit networking--which is cool--but, I'd rather have it be stable).

After trying a few things (disabling auto-suspend, checking for the UAS driver [not in use]) and making no progress I wrote a cronjob to check if the disks had become unmounted and, if so, to remount them. The job runs every minute.

Ugly, but so far it's been effective. It could, obviously, be done with an array and a loop instead, but I just wanted it done and not have to try to remember the exact syntax for that.

#!/usr/bin/bash
needed_to_remount=false
if ! mountpoint -q /mnt/TV; then
  needed_to_remount=true
  mount /mnt/TV
fi

if ! mountpoint -q /mnt/General; then
  needed_to_remount=true
  mount /mnt/General
fi

if ! mountpoint -q /mnt/Movies; then
  needed_to_remount=true
  mount /mnt/Movies
fi

if ! mountpoint -q /mnt/4TB_Storage; then
  needed_to_remount=true
  mount /mnt/4TB_Storage
fi

if [ "$needed_to_remount" == "true" ]; then
  echo "$(date) Needed to remount disks"
fi

And the entry in /etc/crontab to run it:

# Workaround external drives being unmounted randomly
* * * * * root /home/kyle/Scripts/remount_disks.sh >> /var/log/remount_disks.log

Which logs when it detects the drives have been unmounted so I can at least monitor how often this is happening. So far it's been happening 3-7 times a day with what-appears-to-be entirely random intervals. Maybe the log file will reveal some pattern I can work with.