System Design Series: Event Courier — Create Events dynamically

“Events are things that happen in the system you are programming, which the system tells you about so your code can react to them”

I was consulting with an organization that had a unique problem to solve. Here is the problem statement —

“We have some internal websites where content teams create content pertaining to the various teams. Content comprises of internal newsletter, articles, etc. that the various teams publish from time to time. The content teams curate these advancements, innovations, etc. and maintain these internal websites for the organization to keep a track of the latest achievements, innovations, etc.

The marketing team then collates content from these various internal websites to put onto public facing sites to highlight new product innovations, service offerings etc. for our B2B clients to see.

This effort is manual and it takes a lot of effort to sift through all these internal websites and pick content pertaining to topics such as Tech, leadership, new products and so on.

We want to automate the content selection for the marketing team so that the effort to find the right articles to put onto public sites is reduced.”

My first instinct was to push them to go for a Data lake but I sensed a reluctance to expend effort on creating pipelines to push data onto a central location. So I decided to go a different route.

After spending a couple of days pondering on a solution, I spoke to their Principal Data Scientist and asked him if it would be possible to create a recommendation engine to decide which content needs to be filtered.

We brainstormed for a bit and realized that it was definitely doable. According to the solution I had in mind, I just needed the recommendation engine to recommend certain keywords that could help narrow down the content.

Then there was the matter of pulling content from the internal websites and collating it.

It was quite obvious we needed some sort of templating engine that could generate HTML on the fly.

Handlebar.js was my first thought

Handlebars

And therefore, this is the solution I proposed

1️⃣The recommendation engine/Admin would push keywords to a Generator Service.

2️⃣The Generator service uses the Command pattern to encapsulate a request to a Lambda function to invoke another Crawler Lambda with parameters for crawling content from internal websites.

3️⃣The Crawler Lambda could take on several incarnations based on the Command with which it was being invoked

(And this is the beauty of the command pattern)

What is the command Pattern?

In object-oriented programming, the command pattern is a behavioral design pattern in which an object is used to encapsulate all information needed to perform an action or trigger an event at a later time. This information includes the method name, the object that owns the method and values for the method parameters.

Thus, the crawler had the power to decide for itself what sort of content it would crawl for.

4️⃣The result would be returned to the Payload Factory which would then dump the results into Dynamo DB.

Why AWS?

Because most of the client infra was already on AWS.

5️⃣Another Service would then periodically pick the dumped content from Dynamo DB based on a freshness status and process it into an HTML template using Handlebar.

6️⃣This service would push the Template to a Kafka Queue to be receieved at the other end by a notification service.

7️⃣The Notification service would then push the created templates to the marketing team on Email, Slack, Microsoft teams or whatever.

So now, the marketing team just had to focus on content that was pushed to them instead of having to sift through myriad internal websites.

I decided to call this pattern — Event Courier because the events were deciding what sort of payload would be delivered to the end state.

Follow me Ritesh Shergill for more articles on

👨‍💻 Tech

👩‍🎓 Career advice

📲 User Experience

🏆 Leadership

I also do

✅ Career Guidance counselling — https://topmate.io/ritesh_shergill/149890

✅ Mentor Startups as a Fractional CTO — https://topmate.io/ritesh_shergill/193786