Stop Using Selenium: Scrape LinkedIn with FastAPI & Voyager API (~800ms)

🐘 Hey, I’m Appu — an elephant on a mission to make Engineering simple and fun! Here's to learning, building, and improving every day. If you're on your engineering journey too, join me! :) 🚀 💬 Let's chat on X/Twitter: https://x.com/AppuExplains 🔗 Let's connect on LinkedIn:   / ketakik17   #SoftwareEngineering #FastAPI #Python #WebScraping #BackendArchitecture Scraping modern web applications with headless browsers like Selenium or Puppeteer is common, but in production, they are heavy, slow, and prone to bot detection. That’s why I built this browser-less LinkedIn Voyager REST API scraper. Instead of spinning up headless Chrome instances that take 5–10 seconds per profile, this project reverse-engineers LinkedIn’s internal REST API—the Voyager API—using Python, FastAPI, and HTTPX. By replicating the exact network protocol used by LinkedIn's desktop frontend, it extracts rich, structured profile graphs in sub-second response times (under 800ms) with zero browser overhead. ✨ Key Architectural & Technical Highlights: 1. Sub-Second Browser-Less Extraction: • Communicates directly with LinkedIn’s internal private endpoints (/voyager/api/graphql targeting voyagerIdentityDashProfiles). • Bypasses client-side DOM rendering entirely to achieve under 800ms extraction latency. 2. Robust Authentication & Anti-CSRF Architecture: • Authenticates requests via li_at session cookies paired with Double-Submit Cookie pattern validation (JSESSIONID cookie + csrf-token header). • Operates over an httpx.AsyncClient session configured with HTTP/2 multiplexing for resilient network I/O. 3. Dual-Strategy Endpoint Fallback & Defensive Routing: • Primary resolution hits the modern Voyager Dash endpoint, with automatic failover to the classic Profile View route (/voyager/api/identity/profiles/{slug}/profileView). • Intercepts authwalls, redirects, and session expiry loops, surfacing structured HTTP 502/404/400 errors instead of unhandled exceptions. 4. Graph Parsing & Normalization: • Consumes LinkedIn's internal normalized JSON entity graph (application/vnd.linkedin.normalized+json+2.1). • Parses flat arrays of graph entities ($type, URN references) into structured, validated Pydantic schemas (Identity, Headline, Experience, Education, Profile Picture). 5. Interactive Testing & Multi-Method Access: • OpenAPI/Swagger interactive UI documentation at /docs. • Programmatic JSON consumption via POST payload, browser-friendly direct URL inspection via GET query string, and command-line execution via cURL. 💻 Check out the code & documentation on GitHub: https://github.com/Kool-K/linkedin-vo... 🌐 Interactive API Docs (Swagger UI): https://linkedin-voyager-api.onrender... Let me know your thoughts or feedback in the comments! #FastAPI #Python #BackendDevelopment #WebScraping #HTTPX #RESTAPI #Pydantic #CloudDeployment #SoftwareEngineering