As a data engineer, I’m always looking for ways to improve my API consumer process. Recently, I’ve been working on a project that involves capturing metrics and data from Python-based Cloud Run processes. I’ve been using a dictionary object to store first-level data, which I save as a JSON file in a bucket. This helps me track information like API login success, data availability, download success, and license changes.
However, I want to take it to the next level by sending more information in the JSON file as the process runs. I’m thinking of converting the JSON file to a pub/sub event process that publishes to a database. With GCP, I can use pub/sub and publish to a simple PostgresDB.
I like the simplicity of my current stack, but I know I’m not getting the information I want. That’s why I’m looking for incremental improvements that won’t require a complete overhaul of my stack.
One idea is to add a messaging queue like RabbitMQ or Apache Kafka to handle the pub/sub event process. This would allow me to decouple my processes and make it easier to add new features in the future.
Another idea is to use a data warehousing tool like Amazon Redshift or Google BigQuery to store and analyze my data. This would give me more flexibility in terms of data analysis and reporting.
I’m also considering adding a monitoring tool like Prometheus or Grafana to keep an eye on my processes and identify potential issues before they become major problems.
What do you think? Have any suggestions on how I can improve my API consumer process?