Thursday, 2022-07-07

A simple update queue for HN&&LO

HN&&LO is chugging along fine, almost three years after release.

One thing that’s bothered me for a while is that I don’t keep the score and comments for Hackernews entries up to date in a timely manner. So far, I’ve been going back and re-reading last month’s entries but this means that the daily view generally isn’t up-to-date.

The HN API is hosted on Firebase and thus has the ability to get real-time data, but there’s no Perl interface to it, and I really don’t have time to tinker with an entire async setup for a page that’s updated every hour at most.

Instead I’ve implemented a simple queue to keep items up to date.

Every time I read new entries from HN, I add them to the queue. I set a time about an hour in the future.

Every ten minutes or so, I check for items that are older than the current time. If there are items, I compare the data stored in my database to the current status in HN. If there’s a change, I update my datastore and re-submit the item into the queue about 2 hours in the future.

If an item is unchanged, I re-submit it 2 times in case it gets some attention.

If an item is “low score” (currently max 2 upvotes and no comments) it’s given on more chance before being removed from the queue entirely.

I’ve been working on a crude visualization of the queue, it can be seen here:

I log items that are added to the DB here:

Hopefully this will make the main page a bit more informative than before, although just as.