I Built a Notification System. Here's What Broke.

At my startup, I built our notification system from the ground up. Like most teams, we started simple: likes, comments, the standard social features. What I didn't expect was how quickly "simple notifications" would evolve into an architectural puzzle involving burst traffic, database throttling, and week-long deployment cycles that frustrated our marketing team.

If you're building a notification system — or about to — here's what I wish someone had told me before I started.

The First Surprise: Notifications Aren't One Thing

When the product asked for "notifications," I thought: push to phones, show in a tray, done. Wrong.

We quickly discovered notifications exist in two distinct worlds:

• Push notifications grab attention immediately. They're transient, urgent, optimized for "open the app right now."
• The notification tray is persistent storage. Users revisit it, take actions multiple times, browse history.

Here's the catch: not everything belongs in both places.

Take our action-based notification. When users complete a specific action in a connected third-party service, we want them to immediately create content about it in our app. That's a push notification. But putting it in the tray? Users would see it every time they opened notifications, creating endless prompts for something they should do once. It doesn't fit the user journey.

Contrast to that with likes and comments — users should see these repeatedly. They want to browse who interacted with their content over time.

This product-level distinction drove our first major technical decision: we needed separate schemas for push and tray, not one unified model.

When "Scaling" Has Nothing to Do with User Numbers

Here's what surprised me most: our biggest scaling challenges had nothing to do with user count.

We built a "follow" feature where users subscribe to others' activity. When someone with followers takes an action, every follower gets notified. A single action → potentially thousands of notifications.

The multiplier effect is instant. If a prominent user has 500 followers, one transaction creates 500 database writes, 500 APNS calls, 500 everything.

We hit DynamoDB write throughput limits not because we had millions of users, but because notification types create fan-out patterns that overwhelm infrastructure differently than simple one-to-one notifications.

The lesson: "scale" in notification systems often means complexity scale, not user scale.

The 4KB Problem That Forced a Redesign

Apple Push Notification Service has a hard 4KB payload limit. Seems generous until you start sending rich metadata.

Our V1 approach: create one comprehensive notification object with everything the client might need — user details, post data, action context, rendering hints. Send it everywhere: to APNS, to the tray API, to individual notification lookups.

We quickly exceeded 4KB.

The fix required rethinking our entire data model:

• Push notifications: Stripped down to identifiers and critical display text only
• Notification tray: Full rich data for rendering
• Client responsibility: Fetch additional details on-demand if needed

This split added latency (clients sometimes make callbacks for more data), but it worked. More importantly, it taught me that one schema serving multiple contexts is an anti-pattern in notification systems.

The Week-Long Deployment Problem

Our marketing team wanted to send engagement notifications. Simple request, right?

Wrong. Our API used strictly-typed notification schemas. Each new marketing notification required:

1. Add new notification type to API
2. Deploy backend
3. Update client to handle new type
4. Deploy client
5. Wait for App Store review
6. Wait for users to update

Minimum timeline: one week. Often longer.

For a team finding product-market fit, this was unacceptable. Marketing campaigns need to launch now, not next sprint.

Our solution: maintain two schema philosophies simultaneously.

• Product notifications (likes, comments, watches): Strictly typed. Compile-time safety. Explicit behavior. Comprehensive testing. These are core features that must work correctly.
• Marketing notifications: Generic schema. Configurable title, body, image, action buttons, deep links. Marketing composes them in an internal tool and sends immediately. No code changes, no deployments.

This hybrid approach felt architecturally impure. But it solved a real business problem. Sometimes the pragmatic solution beats the elegant one.

Architecture: Why Decoupling Creation from Delivery Matters

Early on, we coupled notification creation and delivery in the same service. Seemed efficient — smaller infrastructure footprint.

Bad idea.

Here's what happens when they're coupled: slow APNS calls block the creation pipeline. A single failed notification delays dozens of subsequent ones. When APNS has issues, your entire notification system grinds to halt.

We split them:

• Creation service: Receives events, validates, persists to DynamoDB, publishes to SNS. Optimized for throughput.
• Delivery service: Pulls from SQS, assembles payloads, sends to APNS. Optimized for retries and external service integration.

This separation means:

• APNS problems don't block new notifications
• We scale creation and delivery independently
• Failed deliveries retry without impacting creation
• Monitoring focuses on different metrics for each concern

The architecture: SNS for event broadcasting → SQS queues (separate for creation and delivery) → Lambda consumers (pull-based to avoid cold starts) → DynamoDB → APNS.

Yes, more infrastructure to manage. But operational benefits massively outweigh complexity costs.

The Burst Traffic Problem Nobody Warns You About

Fan-out notifications plus unpredictable traffic equals disaster.

When a followed user takes an action, we fan out notifications to all their followers. With a small user base, traffic is spiky — not consistent load. A popular user's single transaction can trigger hundreds of writes to DynamoDB within seconds.

We exhausted write throughput. Database throttling. Failed notification writes. Bad user experience.

Traditional solutions don't work here:

• Over-provisioning: Wasteful given irregular traffic
• Auto-scaling: Too slow to react to sudden bursts

Our solution: client-side rate limiting.

Instead of overwhelming DynamoDB during bursts, the notification writer:

• Queues writes internally
• Throttles operations to stay within provisioned capacity
• Spreads burst writes over time
• Monitors queue depth to detect active rate limiting

Result: Users might receive follow notifications over 30–60 seconds instead of instantly. But they reliably receive them without throttling errors.

Lesson: Fan-out patterns require write throughput management, especially with unpredictable traffic.

The API Performance Trap

We built a monolith with shared libraries. The same getNotificationById() method served:

• Client requests (needed rich data for rendering)
• Internal notification delivery (needed minimal IDs for APNS)
• Individual notification lookups (needed everything)

This generic approach meant notification delivery fetched tons of unnecessary data from multiple services, decorated it, massaged it, then threw most of it away.

Result: 5–10 second notification delays.

Fix: Purpose-built APIs with narrow scopes.

• getNotificationForPush(): Returns only what APNS needs
• getNotificationForTray(): Returns full rendering data
• getNotificationById(): Full data for individual lookups

Latency dropped dramatically.

What I'd Do Differently

If I started over today:

1. Design for multiple contexts immediately. Don't assume one schema fits push, tray, and APIs. They have different constraints.
2. Architect for fan-out from day one. Even with small user base, build assuming multiplicative patterns.
3. Accept the generic/typed hybrid. Velocity matters. Maintain strict typing for product features, generic schemas for rapid iteration.
4. Decouple creation from delivery first, not later. The refactor is painful. Start decoupled.
5. Plan for burst traffic with rate limiting. Fan-out creates unpredictable write patterns that overwhelm databases.
6. Build purpose-specific APIs, not generic ones. The performance cost of fetching unnecessary data compounds with complexity.
7. Test each notification type individually. Generic tests miss type-specific edge cases.

The Real Lesson

The biggest insight from building notification infrastructure: treat it as a product decision, not just an engineering one.

The split between push and tray? Product requirement that drives technical architecture.

The generic marketing schema? Business velocity need that overrides architectural purity.

The acceptable latency for follow notifications during bursts? User experience trade-off.

These aren't technical decisions — they're product decisions with technical implications.

If you're building notifications, don't just ask "how do I send a message to a user?" Ask:

• How do different notification types fit into user journeys?
• What latency is acceptable for which notification categories?
• Where do we need type safety vs. deployment flexibility?
• What traffic patterns will different notification types create?

Start simple. But architect assuming notification complexity will grow faster than your user base. The like notification that works perfectly today will be joined by complex aggregations, fan-out patterns, and marketing campaigns that each demand different architectural approaches.

Plan for that evolution from the beginning.