Backend Hang Debug

Purpose

• •Detect and resolve event-loop hangs where the FastAPI app stops responding (e.g., curl http://localhost:8000/ times out) due to synchronous executor shutdown in the SSE news stream.
• •Provide a repeatable triage flow using py-spy to capture live stacks and pinpoint blocking code.

Scope

• •Backend: backend/app/api/routes/stream.py (news stream), backend/app/services/rss_ingestion.py (RSS workers), startup processes.
• •Tooling: py-spy for live stack dumps; curl with timeouts for smoke tests.

Quick Triage

1. •Reproduce hang: curl -m 5 http://localhost:8000/ and curl -m 5 http://localhost:8000/health; note timeouts.
2. •Process check: ss -tlnp | grep 8000 to confirm listener; ls /proc/$(pgrep -f "uvicorn app.main")/fd | wc -l to rule out FD leak.
3. •Stack capture (inside backend venv): uv pip install py-spy then sudo /home/bender/classwork/Thesis/backend/.venv/bin/py-spy dump --pid $(pgrep -f "uvicorn app.main") (and worker pid if multiprocess). Look for ThreadPoolExecutor.shutdown in api/routes/stream.py frames.

Fix Pattern (non-blocking executor)

• •Replace synchronous context manager with ThreadPoolExecutor(...): inside event_generator with a long-lived executor plus explicit non-blocking shutdown:
  • •Create executor outside the context manager.
  • •On client disconnect, cancel pending futures instead of awaiting shutdown.
  • •In finally, call executor.shutdown(wait=False, cancel_futures=True).
• •Rationale: context manager calls shutdown(wait=True), blocking the event loop if RSS worker threads hang on network I/O.

Implementation Steps

1. •Update stream executor usage in backend/app/api/routes/stream.py:
   • •Instantiate executor = concurrent.futures.ThreadPoolExecutor(max_workers=5).
   • •Dispatch work via loop.run_in_executor(executor, _process_source_with_debug, ...).
   • •On disconnect, cancel() pending futures.
   • •In finally, executor.shutdown(wait=False, cancel_futures=True).
2. •Keep RSS executor as-is (rss_ingestion.py) since it runs in background threads, but ensure request timeouts remain reasonable (currently 60s per RSS requests.get).
3. •Retest:
   • •Restart uvicorn; curl -m 5 http://localhost:8000/health should respond.
   • •Start a stream request and abort the client; server must stay responsive.
   • •Re-run py-spy dump to verify no ThreadPoolExecutor.shutdown(wait=True) frames in main thread.

Verification Checklist

• • curl -m 5 http://localhost:8000/ returns a response (no hang).
• • curl -m 5 http://localhost:8000/health succeeds.
• • Aborting /news/stream does not freeze subsequent requests.
• • py-spy dump shows event loop not blocked on ThreadPoolExecutor.shutdown.
• • Frontend no longer stalls waiting on root/health while backend is busy with streams.

Notes & Future Hardening

• •Consider adding request timeout middleware to fail fast on slow handlers.
• •Add per-source network timeouts and shorter retries for RSS feeds to reduce long-lived threads.
• •If multi-worker uvicorn is used, run py-spy on each worker pid when diagnosing hangs.