We recently have a requirement to do a site diff tool so that we can compare our pages in qa and prod to make sure the changes go in is expected.
There used to be a version that works but a bit complicated. The existing stack is user provides a list of url paths and 2 hosts as input for an entry lambda function, which will spawn a bunch of additional lambda functions(via SQS) for each path so that they will take respectively connect to source labs, access the 2 hosts + path, then use selenium take screen shot, then upload to some s3 bucket and eventually write some new sqs msg to another queue which will spawn some other lambda functions to do the image diff then upload the diff image to s3 then write more sqs msg and eventually trigger a
reduce lambda function to generate some summary json data for UI.
To be honest it is a lot of moving parts to reason/troubleshot and maintain. Logs are scattered all around in CloudWatch as there are so many functions created. So I decided to make a slightly easier to use version.
Basic flow of new stack
- get all the paths from CMS with API so that user does not need to collect them manually and copy paste in multiple different places which is error-prone
- use puppeteer in headless mode to access the site and manipulate the dom then take screenshot.
- use JIMP to create diff image based on configured threshold and then create the summary json locally
- upload the diff images and summary to s3 bucket which populates the UI.
With the above flow,
- No complex stack on aws, everything is done in one place, debugging/logging is really easy.
- Also threshold is introduced so we do not need to create diff image for all the paths which are similar enough or identical.
- Previously there are always some diff between prod and qa due to some extra prod-only feedback banner which creates big noice for screenshot comparison as it would sometimes cause pixel shift. Now we can easily remove it via puppeteer’s modern API before taking screenshot so the result is much more accurate and concise.
- Saves a lot of space on s3 as on S3 we only need to have paths that are above threshold. And since we are doing everything in one shot, we also do not have to upload the original screenshot to S3 which was required previously due to the nature of multi-staging process.
- If needed we can integrate to Jenkins and run from there ideally with a slave that has puppeteer installed.