Hey there, fellow data engineers! I’ve been grappling with a common dilemma in data engineering: whether to use an AWS Lambda or a continuously running ECS service to consume Kinesis streams and write data to a Postgres database. My initial plan was to use a Lambda function with Event Source Mapping to automatically invoke the function when new events are written to the Kinesis stream. However, I soon realized that this could lead to performance issues, with over 120 connections to Postgres (1 invocation per shard, with 30+ streams each having 4 shards). To mitigate this, I considered using RDS Proxy or PgBouncer, but then I thought, why not take a simpler approach? What if I wrote a continuously running script that creates a psycopg ConnectionPool and runs it as an ECS service? This way, I can avoid the connection overload and still achieve my goal. So, how would you solve this problem? Have you faced a similar dilemma? Share your thoughts!
