The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:7143599745649762305?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7143599745649762305%2C7143647126135250944%29&replyUrn=urn%3Ali%3Acomment%3A%28activity%3A7143599745649762305%2C7144314175865896961%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287143647126135250944%2Curn%3Ali%3Aactivity%3A7143599745649762305%29&dashReplyUrn=urn%3Ali%3Afsd_comment%3A%287144314175865896961%2Curn%3Ali%3Aactivity%3A7143599745649762305%29
Head of Product at Data Lakehouse Product Firm:
🤷♂️ GitHub PRs opened per month for some of the top open source data lake compute engines. The first trend I randomly stumbled on was #ApacheFlink and it surprised me so I looked across a few more projects. Opened PRs as a metric alone is far from a comprehensive view of a projects true success and community growth. I'm personally very bullish on Flink and I've seen a rise of success stories of organizations leveraging Flink alongside some of these other engines especially when paired with a low-latency lakehouse table format like Apache Hudi.
What are your thoughts on these trends?
Lead Software Engineer at German Security as a Service Product Firm:
why all java? (except ray)
Head of Product at Data Lakehouse Product Firm:
Lead Software Engineer at German Security as a Service Product Firm:
[Head of Product at Data Lakehouse Product Firm] nice meme, but seriously why?
Head of Product at Data Lakehouse Product Firm:
Why each of these projects chose to build on Java good question I'm not sure. But if your question is why I choose to look at these particular projects, I just grabbed some of the ones top of mind that I see most used around data lakes.
Gfesser: Because the Java language has the JVM, and historically speaking, Java ruled the enterprise for many years. Ray is very new relative to all of the others listed here, with its APIs taking advantage of the subsequent heightened popularity of Python. It's also worth pointing out that Presto and Trino share some history, so listing both of these is a bit redundant. All of this said, while it's clear that Spark shows the most consistency over time with respect to PRs, demonstrating stability, I share the sentiment of some of the other comments here in the sense that what arguably matters more are the changes actually being made to the code base. I always consider activity as one of the criteria to take into account for adoption, but activity shouldn't be considered in isolation (as is the case when determining the productivity of engineers).