Senior Software Engineer · Engineering Lead · Fremont, CA

I build systems that move
billions of events a day.

17+ years across distributed systems, AI/ML infrastructure, and FinTech — most recently leading the Instagram Partner Platform team at Meta, and building agentic AI systems used by 70,000+ employees.

See what I've built ↓ GitHub LinkedIn
PIPELINE // A LIFETIME IN ONE DIAGRAM LIVE
200+ Data Sources APIs · events · signals 10K+ Developers partner ecosystem Payment Rails $2B+ daily Event Streams Kafka · async backpressure-bounded Agentic AI / RAG 2M+ queries/day API Platform 1B+ calls/day Compliance Engine 500M+ records Users global
17+
Years engineering
1B+
API calls / day
70K+
AI assistant users
500M+
Records made compliant
Selected Systems

What I've built

Meta · AI Infrastructure

MetaMate — Enterprise Agentic AI Assistant

Multi-agent RAG system serving 70K+ employees and 2M+ daily queries across 200+ live data sources. I personally built and owned the hybrid retrieval pipeline (BM25 + vector search with reciprocal-rank fusion and late-interaction reranking), the fully custom in-house RAG orchestration service, and the model fine-tuning pipeline. Agents execute real workflows — filing tickets, running tools — not just Q&A.

2M+ queries/day 70K+ users 200+ data sources
RAGMulti-agentFAISSLlama fine-tuningReAct
Query Orchestrator Agents BM25 Vector ANN Rerank Llama
Meta · Instagram

Instagram Recommendations Engine — Signal Platform

Owned the partner and merchant signal ingestion layer feeding Instagram's two-tower recommendation system: a Kafka event pipeline into the online/offline feature store, API contracts with ML infrastructure, and the SLO standards for signal freshness. Downstream, candidates are generated by ANN retrieval over 1B+ items and refined through a three-stage ranking cascade.

1B+ item corpus <5ms online features 3-stage ranking cascade
KafkaFeature storeTwo-tower retrievalRedisSpark
Signals Kafka Feat. store ANN L1→L2→L3 Feed
Meta · Instagram Partner Platform (Team Lead)

Partner API Platform

Led the platform serving 1B+ daily API calls from 10K+ external developers. Built sliding-window rate limiting in Redis/Lua, HMAC-signed partner authentication, a Kafka-based observability pipeline that cut mean time to recovery from 4 hours to 35 minutes, and an async Python service layer with bounded-concurrency fan-out. Personally owned the Postgres schema, running zero-downtime expand–contract migrations.

1B+ calls/day 10K+ developers 4h → 35min MTTR
FastAPIRedis/LuaHMAC authPostgresKafka
Partners Rate limit+ HMAC Services Kafka obs. Postgres
Meta · Privacy Engineering

Privacy Compliance Platform

A 14-month program bringing 500M+ records into GDPR, CCPA, and LGPD compliance across 12 teams. Durable workflow orchestration for deletion pipelines, HMAC-SHA256 signed deletion webhooks to partners, and an append-only audit log providing cryptographic proof of compliance.

500M+ records 12 teams aligned 3 regulations
TemporalGDPR/CCPA/LGPDSigned webhooksAudit logs
DSR intake Workflows Deletion Audit log Webhooks
Capgemini · FinTech

Payment Orchestration & Ledger Systems

Regulated payment infrastructure processing $2B+ in daily transactions: saga-based payment orchestration with durable workflows, a double-entry ledger core, and BSA/AML and PCI-DSS compliance built into the transaction path.

$2B+ daily volume Double-entry ledger PCI-DSS compliant
Saga patternTemporalBSA/AMLLedger design
Payment Saga Ledger AML scan
Trajectory

Where I've worked

Present
Senior Software Engineer & Engineering Lead
Meta — Instagram Partner Platform
Led the Instagram Partner Platform team. Built MetaMate's retrieval and fine-tuning pipelines, the Partner API Platform, Instagram's partner signal ingestion for recommendations, and the Privacy Compliance Platform.
Earlier
Engineering roles across consulting & FinTech
Capgemini America · Accenture · TCS
Built regulated payment orchestration and ledger systems at $2B+ daily scale, and delivered large distributed-systems programs for enterprise clients across a decade of consulting engineering.
2008
B.Tech
LNMIIT, Jaipur
Where it all started.
Stack

What I work with

Distributed Systems

  • Kafka event streaming
  • Kubernetes & AWS
  • Redis, Postgres, sharding
  • Rate limiting & backpressure
  • Zero-downtime migrations

AI / ML Infrastructure

  • RAG pipelines & hybrid retrieval
  • Multi-agent / agentic systems
  • LLM fine-tuning (Llama)
  • Vector search (FAISS, Pinecone)
  • Feature stores & ranking systems

FinTech & Compliance

  • Payment orchestration (Saga)
  • Double-entry ledgers
  • GDPR / CCPA / LGPD programs
  • PCI-DSS · BSA/AML
  • Signed webhooks & audit logs
Contact

Let's build something.

manyu.sh@outlook.com LinkedIn GitHub