Category: Costs to Expect

Posted on

Jobs and tasks gamble paid off

I have spent the last 12 months developing a product for one of my long-term clients, I’m going to refer to the product as EWAQO for the rest of this post.

I initially thought EWAQO would be three to four months of work, lockdowns, homeschooling, the pandemic in general and feature creep put paid to that.

EWAQO is far from the largest product I’ve worked on, I’ve been a professional developer for over twenty years and have worked at scale, it is however the largest product I’ve created single-handled. The Costs to Expect API, Website and App are big when combined, EWAQO is in a world of its own, it has so many different systems and is immense.

I decided early on to go deep with queues and scheduled tasks, I wanted to keep the App fast for the user and do as much of the heavy lifting as possible behind the scenes. Typically, I’m using queues for complex user-initiated tasks/actions and scheduled tasks are for processing.

It is easy to handle the complex stuff behind the scenes, but there is a problem, you need some visibility. Before I wrote the first job I created the interface, that way the client can see what is happening, is a job running? Did it fail? What was the output? Who initiated it? etc.

This was the masterstroke and made so much of the complexity possible, the client could see what is happening, job classes and tasks are simpler to create because you don’t need to worry about the front-end, it all came together brilliantly.

I’m not done with EWAQO, as of writing, there are 27 jobs and 26 scheduled tasks and I have visibility of each one, I can see when they are running, what is due, when and why they fail, it all just works.

We are working toward the soft-release of Costs to Expect this year, part of that is developing two smallish apps that act as companions. Both apps will borrow heavily from what I learned developing EWAQO.

Posted on

Website caching, simple now, later, dependant caches.

Two weeks ago, I quickly added caching to the Costs to Expect Website. My goal was to reduce the number of unnecessary API requests the Website made to the Costs to Expect API, mission accomplished.

The problem, I did a quick job, improvements needed.

The Costs to Expect Website is a long term social experiment, my wife and I are tracking the total cost of raising humans; we want to know how far off the £250,000 per child figure we will be when our sons reach the age of 18.

Back to the caching, issue one, I added caching to the top four or five pages; the rest of the Website needs some love. Issue two, for some unknown reason, I decided 15 mins was a sensible cache lifetime.

Solution one

I hacked in the caching; I added a simplified version of the much more featureful caching in the Costs to Expect App. I need to refactor the ‘request’ code and add caching for all requests.

Solution two

I set the cache lifetime at 15 minutes, why I don’t know. The content on the Website changes at most daily and additionally there is no need for the data to be live; people are not going to ‘throw a fit’ if they can’t see the latest expense we have added for Jack or Niall.

I am going to set the cache lifetime to four hours.

Four hours you say, why not 24? Well, I figured it is a sensible limit to ensure there isn’t too much of a mismatch between cached data while still dramatically reducing API requests.

Imagine a scenario whereby a user browses to the site and visits the summary page; the results are cached; they never, however, make it to the lists for that summary. If a second user comes along three hours later and views the listings, there is a good chance the data will mostly match the cached summary. If I set the cache lifetime at 24 hours, a value that initially seems reasonable, I am increasing the chance of the summaries and data mismatching.

There is a solution to the inconsistent data problem, dependant caches.

I need to add support for linking cached data, for example, a summary and the related lists, and more importantly, controlling the maximum period allowable between dependant cache item creation.

With the current implementation, there can be a difference of up to four hours between summary and list cache creation, realistically, the limit for dependant data should be closer to five minutes.

I will eventually update the caching system for the Costs to Expect App and at some point, trickle the implementation down to the Costs to Expect Website.