New ask Hacker News story: Ask HN: Any pragmatic guides to building large data pipelines?

Ask HN: Any pragmatic guides to building large data pipelines?
3 by elt | 0 comments on Hacker News.
I am rebuilding a data pipeline that processes billions of records. An overview of what I built is as follows: collect data from (n*k) sources-> derive new data -> generate a unified/merged collection of data (n) data. The current solution is all hand crafted code. I know this is a 10,000 foot view of the problem, but are there any guides or books on how to better design and implement this type of solution?