A federal satellite data program ingests files continuously from ground stations and international partner servers. The pipeline runs on Lambda, SQS, and EventBridge, pulling files that reach 400 MB and routing them by product type into storage for downstream processing.
The work covered both the architecture and the code. Large files were timing out because the chunked download path silently never engaged in production, so transfers ran single stream at a fraction of the available bandwidth. A local disk chunked download path fixed it. Range request support had to be forced for an upstream server that honored range requests without advertising them. Download concurrency was benchmarked rather than guessed: throughput climbed steadily through 80 parallel connections and fell off sharply above that, which set the safe ceiling.
Separately, an audit of the product routing configuration raised the match rate from 21 percent to 41 percent across a dataset of more than a million records, by correcting timestamp handling, instrument grouping, and pattern ordering across hundreds of product types.
400 MBLargest files moved reliably through the pipeline
14 MB/sSustained throughput after concurrency tuning
21% to 41%Routing match rate after configuration audit
Client and program not named at their preference. Further detail available under NDA on a call.