Base64 Secret Padding Mismatch Between Environments
Problem
Webhooks or API calls fail authorization even though the secret "looks correct" when
comparing environment variables between systems. The root cause is Base64 padding
characters (=) being silently dropped when copying or displaying secrets.
Context / Trigger Conditions
- •Webhook handler returns "Unauthorized" or generic "Server Error"
- •Error occurs in mutation/database call after webhook validation passes
- •Secret was copy-pasted between systems (e.g., Railway → Convex, Vercel → Supabase)
- •CLI tools show the secret without trailing
=characters - •The secret is Base64-encoded (often ends with
=or==)
Example error pattern:
Webhook error: Error: [Request ID: xxx] Server Error
at mutationInner (...)
Solution
Step 1: Identify Base64 Secrets
Base64-encoded secrets often:
- •Are 32-44 characters long
- •Contain alphanumeric characters plus
+,/, and= - •End with
=or==for padding
Step 2: Compare Full Values
Don't trust CLI display output. Get the raw values:
For Convex:
npx convex env list --prod | grep SECRET_NAME # Output: SECRET_NAME=VbjsOyAxdAuOtBkiSXZiUX2sA4ELpO6FFx3Y8lIDiD4=
For Railway:
railway variables | grep SECRET_NAME # Output may truncate: VbjsOyAxdAuOtBkiSXZiUX2sA4ELpO6FFx3Y8lIDiD4 # Note the missing trailing =
Step 3: Fix the Mismatch
When setting secrets, explicitly include the full value with padding:
# Use single quotes to preserve special characters railway variables set 'INTERNAL_API_SECRET=VbjsOyAxdAuOtBkiSXZiUX2sA4ELpO6FFx3Y8lIDiD4=' # Or for Vercel vercel env add INTERNAL_API_SECRET # Then paste the full value including =
Step 4: Redeploy
After fixing the environment variable, trigger a redeploy:
railway deployment redeploy --yes # or vercel --prod
Verification
- •Check logs for successful webhook processing
- •Verify the operation that was failing now succeeds
- •For subscription webhooks: check that user status updates correctly
Example
Scenario: Polar webhook for subscription activation fails in production.
Symptoms:
- •Payment succeeds in Polar
- •Redirect to app shows
?success=true - •User subscription status remains "FREE"
- •Railway logs show:
Webhook error: Error: Server Error
Investigation:
# Convex shows: npx convex env list --prod | grep INTERNAL_API_SECRET # INTERNAL_API_SECRET=VbjsOyAxdAuOtBkiSXZiUX2sA4ELpO6FFx3Y8lIDiD4= # Railway shows (truncated): railway variables | grep INTERNAL # INTERNAL_API_SECRET │ VbjsOyAxdAuOtBkiSXZiUX2sA4ELpO6FFx3Y8lIDiD4
Fix:
railway variables set 'INTERNAL_API_SECRET=VbjsOyAxdAuOtBkiSXZiUX2sA4ELpO6FFx3Y8lIDiD4=' railway deployment redeploy --yes
Notes
- •This issue is particularly common with secrets generated by
openssl rand -base64 32 - •Some CLIs and web UIs strip or hide trailing
=for display purposes - •When in doubt, regenerate the secret and set it in both places simultaneously
- •Consider using hex encoding (
openssl rand -hex 32) to avoid padding issues entirely - •The
=padding in Base64 ensures the encoded length is a multiple of 4
Prevention
When generating and distributing secrets:
- •Generate in one place and immediately copy the full value
- •Use a password manager or secure note to store the canonical value
- •After setting in each system, verify by echoing back or checking logs
- •Consider using non-Base64 formats (hex, UUID) for simpler handling
Related Scenarios
- •JWT secrets between auth providers and backends
- •Stripe/Polar webhook secrets
- •Internal API authentication between microservices
- •Any secret that's validated via string comparison