Hey everyone,
hoping to get some advice from anyone who runs automated background scripts for fetching monthly utility statements on a Linux server setup.
I am setting up a small Python script on my homelab server to fetch monthly bill details automatically and notify me when new PDF statements are ready. The script passes a consumer reference ID via HTTP POST/GET requests to query the remote endpoint.
The issue I’m running into is that the server response returns a blank wrapper, or the request hangs because the page relies on heavy client-side JavaScript rendering to generate the actual billing table and download button. I checked the portal parameters on this website to make sure I wasn’t missing mandatory form payload keys, but while the page loads fine in a standard desktop browser, standard headless cURL or basic Python requests fail to render the dynamic elements.
Has anyone dealt with similar dynamic JS execution hurdles when writing server-side automation scripts? Should I spin up a headless Chromium container using Playwright/Selenium in Docker to handle the dynamic evaluation, or is there a lighter way to parse these dynamic payloads without consuming too many system resources?
Appreciate any suggestions or workflow examples!