Data Engineering Architecture Tutorial 8: API and SaaS Ingestion | Pagination, Rate Limits

Part 8 of the Data Engineering Architecture Tutorial, in the sources and ingestion module. Getting data out of APIs and software as a service applications. Different SaaS applications provide different mechanisms for extracting data, so a single approach does not always work. We follow Databricks' page on API ingestion: pulling data over HTTP, usually as paginated JSON, and the three things you handle yourself, which are authentication, pagination and rate limits. The 429 status code and retry with backoff. Incremental sync in Airbyte's words, with a cursor value such as an updated at timestamp, and why some APIs do not fit. Three patterns Databricks describes for building the pull: a periodic pull, an incremental custom data source, and a scheduled job that lands raw responses as files. Why to check for a managed connector first. Then webhooks, which GitHub describes as receiving data as it happens, as opposed to polling. We finish with where this sits in the picture from part 1. ⏱️ Timestamps 0:00 Intro 0:19 Why SaaS data is hard to ingest 1:02 What API ingestion is 1:34 Authentication 1:51 Network access 2:04 Pagination 2:29 Rate limits and the 429 status code 2:56 Retry with backoff 3:18 Incremental sync 3:59 The cursor 4:41 When an API does not fit 4:55 Pattern one: periodic pull 5:17 Pattern two: incremental custom data source 5:36 Pattern three: land raw responses as files 6:02 Check for a managed connector first 6:25 Webhooks 6:55 Where it sits in the big picture 7:12 Recap Sources (checked 6 October 2026): Databricks documentation, "Ingest data from an API in pipelines"; AWS whitepaper, "Patterns for Ingesting SaaS Data into AWS Data Lakes" (March 2022, kept by Amazon for historical reference only); Microsoft Learn, "Copy and transform data from and to a REST endpoint by using Azure Data Factory"; IETF RFC 6585, "Additional HTTP Status Codes"; Airbyte documentation, Connector Builder, "Incremental Sync"; GitHub Docs, "About webhooks". Each description is that project's or vendor's own, for its own product. The placement in the part 1 picture is this series' own simplification. Next in the Data Engineering Architecture Tutorial: part 9, Event-driven and Internet of Things architecture. Tech in Layers explains tech topics one layer at a time, with diagrams that build up as you listen. #DataEngineering #DataArchitecture #DataEngineeringArchitecture #APIIngestion #SaaS #RESTAPI #Pagination #RateLimiting #Webhooks #LearnDataEngineering #TechInLayers