One thing that’s bothered me for a while is that I don’t keep the score and comments for Hackernews entries up to date in a timely manner. So far, I’ve been going back and re-reading last month’s entries but this means that the daily view generally isn’t up-to-date.
The HN API is hosted on Firebase and thus has the ability to get real-time data, but there’s no Perl interface to it, and I really don’t have time to tinker with an entire async setup for a page that’s updated every hour at most.
Instead I’ve implemented a simple queue to keep items up to date.
Every time I read new entries from HN, I add them to the queue. I set a time about an hour in the future.
Every ten minutes or so, I check for items that are older than the current time. If there are items, I compare the data stored in my database to the current status in HN. If there’s a change, I update my datastore and re-submit the item into the queue about 2 hours in the future.
If an item is unchanged, I re-submit it 2 times in case it gets some attention.
If an item is “low score” (currently max 2 upvotes and no comments) it’s given on more chance before being removed from the queue entirely.
I’ve been working on a crude visualization of the queue, it can be seen here:
I log items that are added to the DB here:
Hopefully this will make the main page a bit more informative than before, although just as slow.