We developed the solution for US-based Software Innovation Labs - a consortium organization who serve software building clients with a number of specialized software development houses and agencies which provide a broad spectrum of technical and domain expertise for even the most complex software development projects. The client to be serviced on this project was a startup company created to help a professional gamer community monetize their gaming activities. The client’s aim was to enable the gamers operating in the community to gain insights into which of their gaming activities resulted in an increased interest from their followers as well as to equip the gamers with actionable analytics to optimize their gaming style with a view to monetizing their activities more effectively.

Technically, the solution needed to ensure the monitoring of the activity of the thousands of users operating in the community. Historical user-related data needed to be stored and made available for fast data querying - the end user should not experience delays while viewing their personal data records, irrespective of the volume of traffic on the platform. The system was also expected to be robust enough to process a large number of concurrent social media data streams and generate real-time analytics based on the latter. Most importantly, the processes deployed needed to be scalable, i.e., the system needed to be fit enough to efficiently handle a rich spectrum of data sources as well as a large and growing number of users.

At its core, the solution built collects data streams generated from gaming and social media sites, such as Twitch, Facebook, YouTube, and the like. The data collected are processed to display statistics related to the activities undertaken by the gamers as well as those of their followers. The architecture of the system is based on Apache Kafka - an open source distributed streaming platform which lets you subscribe to streams of records, as well as store, process and publish the streams of records as they occur in real-time. As such Kafka lends itself particularly well for building real-time streaming data pipelines that efficiently relay data between systems and applications as well as for crafting real-time streaming applications that transform or react to the streams of data.

Thus Kafka Streams was implemented in the system as a key architectural element which enabled to aggregate and transform the data from multiple gaming and social media sources as well as to generate real-time analytics for the users with which to optimize their performance in the community. The analytics displayed enable the gamers to optimize their performance in terms of popularity as well as to adopt behaviour patterns to monetize their gaming activities more effectively. Kafka with its properties is a key success factor behind meeting one of the main challenges faced while building and running the system, i.e. the distributed streaming platform helps to ensure fast data querying and thus supports smooth user experience.

On top of that, we also chose to use the OpenShift platform for managing (micro)services and performing CI/CD. OpenShift - a PaaS type of system and an extension of Kubernetes enables to set up a private open source cloud for monitoring and maintaining multiple services, to create custom deployment pipelines as well as to support rapid application development, efficient deployments and system scaling.