Automate screenshots of web pages with Eyewhitness

Created:
Author: Christoph Stoettner
Read in about 4 min · 715 words

Automated Guided Vehicles carrying containers at Port of Rotterdam

Photo by Bernd Dittrich | Unsplash

This week, I investigated an issue with the advanced profiles search in HCL Connections. I had a system which did not show any results in the advanced search for the department number of Connections profiles.

HCL support tried for three months to reproduce the issue, and the users missed the option to search for their teammates with the advanced profiles search.

TL;DR

I started creating entries in the profiles’ database in the table empinst.employee. First I created the department name, which did not work in the other system. To be sure that it is not dependent on a broken index, I recreated the index in my demo system.

The result was that none of my entries were found in the advanced search.

I used a deptNumber like HR/TEAM-SOUTH1, my first thought was an issue with the characters / or -, so I created entries without them. Still, none of the departments showed up.

Then I thought the length of the used string could be the problem, so I added the deptNumber: test to two of the profiles. After recreating the index one more time, a search for test returned one result. To make it short, I made a typo and created one entry with capital T.

Automate the process

During the tests, I created plenty of strings with different length and spelling. During some CTF, I worked with Eyewhitness to automate screenshots of web URLs.

Just install Eyewhitness (needs Selenium and Python3) in a container or a local machine, like explained in the repository. It’s well documented, just follow the README.

So I created the URLs for all used deptNumbers in my profiles database and wrote them to a text file.

select unique(PROF_DEPARTMENT_NUMBER)) FROM EMPINST.EMPLOYEE e WHERE PROF_DEPARTMENT_NUMBER != '';

The URL used in the advanced search was https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&displayName=&preferredFirstName=&preferredLastName=&profileTags=&jobResp=&description=&experience=&countryDisplayValue=&email=&telephoneNumber=&deptNumber=SEARCHTERM

So I merged the results and the URL to get a list with different URLs and search terms. I tested and found that empty search keywords can be removed. My list of URLs looked like this, and I stored it in urls.txt:

https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=1
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=2
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=3a
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=HR/TEAM-SOUTH1
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=HR/TEAMSOUTH1
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=HRTEAM-SOUTH1
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=HRTEAMSOUTH1
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=PQRSTUV
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=Test
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=XYZ
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=a
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=abcdefgh
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=bc
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=def
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=ghij
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=ijklmno
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=klmno
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=pqrstu
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=test
https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=vwxyzab

Now we can use this list to run Eyewhitness and create the screenshots:

cd Eyewhitness/Python/bin
./EyeWitness.py -f urls.txt --delay 5 --web --no-dns --results 25 --width 1920 --height 1080

We see that Eyewhitness calls the URLs and creates a subfolder with the actual date and time with a collection of URLs and screenshots:

################################################################################
#                                  EyeWitness                                  #
################################################################################
#           Red Siege Information Security - https://www.redsiege.com          #
################################################################################

Starting Web Requests (20 Hosts)
Attempting to screenshot https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=1
[*] Sleeping for 5 seconds before taking the screenshot
Attempting to screenshot https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=2
[*] Sleeping for 5 seconds before taking the screenshot
Attempting to screenshot https://cnx8-db2.stoeps.home/profiles/html/advancedSearch.do?keyword=&deptNumber=3a
[*] Sleeping for 5 seconds before taking the screenshots
...
[*] Completed 15 out of 20 services
Finished in 33.67001438140869 seconds

[*] Done! Report written in the /var/home/stoeps/itsec/EyeWitness/Python/2024-11-03_185026 folder!
Would you like to open the report now? [Y/n]

I started a local webserver and opened the URL:

python3 -m http.server

Opening the reports.html shows a table with the used URL and the actual screenshot:

screenshot of report.html

So we can scroll through all the results and quickly see which searches show results and which do not. I found that all deptNumber entries which contain a capital do not show results, but numbers and lowercase entries work fine.

The next time I would even create the database entries for deptNumber via SQL or scripting, then it is even faster.

Eyewhitness can also be used to check URLs regularly and compare screenshots over time. So run a set of URLs before an update and store the results. Then make your changes and run Eyewhitness again. Put the results side by side and compare the result. Or keep both result sets to have proof that everything looked the same after the update.

Authentication with Eyewhitness

I made it easy in this case and opened profiles for anonymous access, so I could use the URL list and didn’t need any login.

If you require authentication, you can copy the LtpaToken2 from one of your authenticated requests and add it with --cookies LtpaToken2=... to the Eyewhitness call. Or you automate this step and get the cookie via curl on the console.

Author
Suggested Reading
Aaron Burden: Fountain pen and a notebook

For GPN19 I prepared a second talk on Documentation with any Editor . The talk was based on a previous one from Froscon 13, but the pipeline tooling changed.

This time there was a technical issue during the recording and so there are only the slides available, but you can still watch the video of the Froscon talk: Froscon 13: Documentation with any Editor

Created:
Last Update:
Read in about 8 min