-
Notifications
You must be signed in to change notification settings - Fork 372
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GoWitness v3.0.3 Missing Output from Writers in CI/CD #237
Comments
No. Writers don't depend on each other. They are all "in addition to".
No. Drivers populate a result that is then passed off to writers, regardless of which driver generated them.
I can't think of a reason now why the behavior would be different on say a GitHub actions runner vs elsewhere, but to test if there is some pathing related weirdness happening, can you try use more specific paths? Eg |
Appreciate the feedback. To check, are you using the official Kali image? I can take a look at it in that scenario to also maybe narrow down what is happening. |
No prob, yeah, that'd be much appreciated! Here's the Dockerfile I'm using to build the image: FROM kalilinux/kali-rolling:latest
ENV DEBIAN_FRONTEND noninteractive
ENV TERM xterm-256color
RUN apt-get update
RUN apt-get update && apt-get install -y -q apt-utils
RUN apt-get install -y -q \
curl dnsutils ca-certificates python3 virtualenv \
python3-distutils-extra python3-virtualenv python3-pip \
python3-setuptools python3-wheel python3-magic python3-venv \
pipx git sudo whois zip unzip libimage-exiftool-perl recon-ng \
golang chromium tini sqlite3 build-essential libpcap-dev wget \
&& apt-get autoremove -y \
&& apt-get autoclean -y \
&& rm -rf /var/lib/apt/lists/*
# install go v1.23.1 since golang apt package is at v1.22.6 as of 9-10-2024
RUN wget https://go.dev/dl/go1.23.1.linux-amd64.tar.gz -O /tmp/go1.23.1.linux-amd64.tar.gz
RUN rm -rf /usr/local/go && tar -C /usr/local -xzf /tmp/go1.23.1.linux-amd64.tar.gz
RUN rm /tmp/go1.23.1.linux-amd64.tar.gz
# Install Tini
ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
WORKDIR /app
COPY . /app
COPY entrypoint.sh deploy.sh /
RUN chmod +x /entrypoint.sh /deploy.sh
RUN if [ ! -d "${HOME}/go" ]; then mkdir "${HOME}/go"; fi
ENV GO111MODULE=on
RUN go mod download
RUN GOOS=linux GOARCH=amd64 go build -v -trimpath -ldflags="-s -w" -o /usr/bin/redacted .
RUN chmod +x /entrypoint.sh /deploy.sh /usr/bin/redacted
RUN mkdir -p "${HOME}/.config/redacted"
RUN cd "$HOME"
RUN rm -rf /app
ENTRYPOINT ["/tini", "--", "/entrypoint.sh"] |
I wonder if it's due to installing sqlite via apt since using the self-contained github.com/glebarez/sqlite |
Thanks! I don't think the apt install of sqlite will matter for the pure Go implementation no. Last one, how do you install/use gowitness in this container? |
Am running it via this function: //// gowitness and subfinder have a dependency collision: https://github.com/projectdiscovery/subfinder/issues/1374
//// # github.com/projectdiscovery/utils/update
//// ../../../go/pkg/mod/github.com/projectdiscovery/[email protected]/update/update.go:97:40: undefined: glamour.ASCIIStyleConfig
//// My PR https://github.com/projectdiscovery/utils/pull/531 for projectdiscovery/utils got merged in, awaiting new release
//// httpx also has older utils indirect dep v0.2.4
// runGoWitness installs the latest version and runs goWitness
func runGoWitness(urlsFile, outputDir, userAgent string) error {
installedAptPackages, err := localio.NewAptInstalled()
if err != nil {
return err
}
// check if chromium installed and install it via apt-get if not installed.
if err = localio.AptInstall(installedAptPackages, "chromium"); err != nil {
return err
}
// TODO: GoWitness v3.X is suitable to be run natively in Go which will be more optimal. problem is there is currently a dependency collision as of 09-20-2024
localio.InfoLabelWithColorf("GoWitness", "blue", "Installing latest version of GoWitness")
if err = localio.RunCommandPipeOutput("GO111MODULE=on go install github.com/sensepost/gowitness@master", nil, false, false, 20); err != nil {
return err
}
if gowitness, exists := localio.CommandExists("gowitness"); exists {
goWitnessScreenshotDir := fmt.Sprintf("%s/gowitness/screenshots", outputDir)
goWitnessJSONLOutputFile := fmt.Sprintf("%s/gowitness/results.jsonl", outputDir)
goWitnessDB := fmt.Sprintf("sqlite://%s/gowitness/gowitness.sqlite3", outputDir) // must specify sqlite:/// for absolute database path. New in GoWitness 2.5.0 release https://github.com/sensepost/gowitness/releases/tag/2.5.0
goWitnessReport := fmt.Sprintf("%s/gowitness/report.zip", outputDir) // exported report file is a zip that contains all the screenshots and report.html file.
// GoWitness automatically creates the output dir, but just in case later versions change, we create the dir.
if err = os.MkdirAll(goWitnessScreenshotDir, 0750); err != nil {
return err
}
if err = localio.RunCommandPipeOutput(fmt.Sprintf("%s scan file -f %s --screenshot-path %s --write-db --write-db-uri '%s' --write-jsonl --write-jsonl-file %s -t 20 --chrome-user-agent '%s'", gowitness, urlsFile, goWitnessScreenshotDir, goWitnessDB, goWitnessJSONLOutputFile, userAgent), nil, true, true, 120); err != nil {
return err
}
if err = localio.RunCommandPipeOutput(fmt.Sprintf("%s report generate --db-uri '%s' --zip-name %s --screenshot-path %s", gowitness, goWitnessDB, goWitnessReport, goWitnessScreenshotDir), nil, true, true, 60); err != nil {
return err
}
localio.PrintInfo("Gowitness", fmt.Sprintf("%s report server --db-uri '%s' --screenshot-path %s", gowitness, goWitnessDB, goWitnessScreenshotDir), "To view gowitness report, unzip report.zip and open report.html in the browser or, run the gowitness server, via the following command:")
}
return nil
} as I'm looking at this code, I see I'm still using master instead of latest for install. not that it should matter too much, is a remnant of previous 2.5.X release. Some helper funcs in there like localio.RunCommandPipeOutput which is just a wrapper around os.exec |
Doing some testing, I'm not 100% sure I can replicate this issue. An extra thing I'm adding to test is to not create the results directory before hand in case that causes some issue, but even with that case I'm not replicating the issue yet. A minimal Dockerfile im using: FROM kalilinux/kali-rolling:latest
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install \
ca-certificates golang chromium sqlite3 jq vim \
-y --no-install-recommends
RUN go install github.com/sensepost/gowitness@latest
CMD ["bash"] An invocation
Using
Since you're shelling out from a go program, maybe add ~/go/bin/gowitness -D scan file -f /targets.txt --write-db --write-db-uri sqlite:///tmp/results-debug-file/gowitness.sqlite3 --write-jsonl --write-jsonl-file /tmp/results-debug-file/gowitness.jsonl --screenshot-path /tmp/results-debug-file/screenshots
2024/09/24 16:58:15 DEBU <cmd/root.go:28> debug logging enabled
2024/09/24 16:58:15 DEBU <cmd/scan.go:72> scanning driver started driver=chromedp
2024/09/24 16:58:15 DEBU <runner/runner.go:42> final screenshot path screenshot-path=/tmp/results-debug-file/screenshots
2024/09/24 16:58:15 DEBU <cmd/scan_file.go:57> starting file scanning file=/targets.txt
2024/09/24 16:58:15 DEBU <drivers/chromedp.go:122> witnessing 👀 target=https://sensepost.com:80
2024/09/24 16:58:15 DEBU <drivers/chromedp.go:122> witnessing 👀 target=https://google.com:80
2024/09/24 16:58:15 DEBU <drivers/chromedp.go:122> witnessing 👀 target=https://sensepost.com:443
2024/09/24 16:58:15 DEBU <drivers/chromedp.go:122> witnessing 👀 target=https://google.com:443
2024/09/24 16:58:20 INFO <runner/runner.go:146> result 🤖 target=https://google.com:443 status-code=200 title=Google have-screenshot=true
2024/09/24 16:58:21 INFO <runner/runner.go:146> result 🤖 target=https://sensepost.com:443 status-code=200 title=":: Orange Cyberdefense ::" have-screenshot=true
2024/09/24 16:58:21 DEBU <drivers/chromedp.go:462> closing browser allocation context Now I know you logged the issue about CI/CD setups specifically, but I wanted to check if thats still what we think the issue is here? Will keep investigating. |
On the CI/CD train, I created a test GitHub action to run gowitness and that seems to also output results. https://github.com/leonjza/gowitness-cicd-example/actions/runs/11018535188/job/30599032224 And some more complete testing: |
Thank you for testing this!!!
Much appreciated. I will follow up later sometime this week with my findings if I am able to figure out what the culprit was. |
So i also did some testing using my Dockerfile in local environment and it looks like the main culprit was using 20 threads. err="error enabling network tracking: context deadline exceeded" 3 threads seems to be the sweet spot from testing in Docker. It worked as expected with 3 threads. Will let ya know if this fixes it in the CI/CD as well next time a pipeline runs so I can review. 🍻 |
Thanks for the feedback. I think there may be something more subtle at play here. Using this Dockerfile FROM kalilinux/kali-rolling:latest
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install \
ca-certificates golang chromium sqlite3 jq vim \
-y --no-install-recommends
ADD top1mlessporn100.txt /top1mlessporn100.txt
RUN go install github.com/sensepost/gowitness@latest
CMD ["bash"]
I can run through a list of ~100 targets using 20 goroutines and a mediocre internet connection in about 15 minutes, with results: # wc -l /top1mlessporn100.txt
100 /top1mlessporn100.txt
# time ~/go/bin/gowitness scan file -f /top1mlessporn100.txt -t 20 --write-db --write-jsonl
2024/09/25 08:27:54 INFO result 🤖 target=http://twitter.com:80 status-code=200 title="X. It’s what’s happening / X" have-screenshot=true
2024/09/25 08:28:08 INFO result 🤖 target=http://twitter.com:443 status-code=200 title="X. It’s what’s happening / X" have-screenshot=true
2024/09/25 08:28:19 INFO result 🤖 target=https://twitter.com:443 status-code=200 title="X. It’s what’s happening / X" have-screenshot=true
2024/09/25 08:28:24 INFO result 🤖 target=http://cloudflare.com:80 status-code=200 title="Connect, Protect and Build Everywhere | Cloudflare" have-screenshot=true
2024/09/25 08:28:30 INFO result 🤖 target=http://cloudflare.com:443 status-code=200 title="Connect, Protect and Build Everywhere | Cloudflare" have-screenshot=true
2024/09/25 08:28:37 INFO result 🤖 target=https://cloudflare.com:443 status-code=200 title="Connect, Protect and Build Everywhere | Cloudflare" have-screenshot=true
2024/09/25 08:28:44 INFO result 🤖 target=http://instagram.com:80 status-code=200 title=Instagram have-screenshot=true
2024/09/25 08:28:53 INFO result 🤖 target=http://instagram.com:443 status-code=200 title=Instagram have-screenshot=true
2024/09/25 08:28:59 INFO result 🤖 target=http://linkedin.com:443 status-code=200 title="LinkedIn: Log In or Sign Up" have-screenshot=true
2024/09/25 08:29:07 INFO result 🤖 target=https://linkedin.com:443 status-code=200 title="LinkedIn: Log In or Sign Up" have-screenshot=true
2024/09/25 08:29:23 INFO result 🤖 target=http://live.com:80 status-code=200 title="Microsoft Outlook (formerly Hotmail): Free email and calendar | Microsoft 365" have-screenshot=true
2024/09/25 08:29:43 INFO result 🤖 target=http://live.com:443 status-code=200 title="Microsoft Outlook (formerly Hotmail): Free email and calendar | Microsoft 365" have-screenshot=true
2024/09/25 08:30:01 INFO result 🤖 target=https://live.com:443 status-code=200 title="Microsoft Outlook (formerly Hotmail): Free email and calendar | Microsoft 365" have-screenshot=true
2024/09/25 08:30:05 INFO result 🤖 target=https://googletagmanager.com:443 status-code=404 title="Error 404 (Not Found)!!1" have-screenshot=true
2024/09/25 08:30:15 INFO result 🤖 target=http://fbcdn.net:80 status-code=200 title="Facebook – log in or sign up" have-screenshot=true
2024/09/25 08:30:21 INFO result 🤖 target=http://fbcdn.net:443 status-code=200 title="Facebook – log in or sign up" have-screenshot=true
2024/09/25 08:30:27 INFO result 🤖 target=https://fbcdn.net:443 status-code=200 title="Facebook – log in or sign up" have-screenshot=true
2024/09/25 08:30:34 INFO result 🤖 target=http://amazon.com:80 status-code=200 title=Amazon.com have-screenshot=true
2024/09/25 08:30:40 INFO result 🤖 target=http://amazon.com:443 status-code=200 title=Amazon.com have-screenshot=true
2024/09/25 08:30:47 INFO result 🤖 target=https://amazon.com:443 status-code=200 title=Amazon.com have-screenshot=true
2024/09/25 08:31:02 INFO result 🤖 target=http://fastly.net:80 status-code=200 title="Powering the best of the internet | Fastly" have-screenshot=true
2024/09/25 08:31:04 INFO result 🤖 target=http://fastly.net:443 status-code=200 title="Powering the best of the internet | Fastly" have-screenshot=true
2024/09/25 08:31:09 INFO result 🤖 target=http://googleusercontent.com:80 status-code=404 title="Error 404 (Not Found)!!1" have-screenshot=true
2024/09/25 08:31:14 INFO result 🤖 target=http://googleusercontent.com:443 status-code=404 title="Error 404 (Not Found)!!1" have-screenshot=true
2024/09/25 08:31:15 INFO result 🤖 target=https://fastly.net:443 status-code=200 title="Powering the best of the internet | Fastly" have-screenshot=true
2024/09/25 08:31:19 INFO result 🤖 target=https://googleusercontent.com:443 status-code=404 title="Error 404 (Not Found)!!1" have-screenshot=true
2024/09/25 08:31:23 INFO result 🤖 target=http://googlesyndication.com:80 status-code=200 title=Google have-screenshot=true
2024/09/25 08:31:28 INFO result 🤖 target=http://googlesyndication.com:443 status-code=200 title=Google have-screenshot=true
2024/09/25 08:31:33 INFO result 🤖 target=https://googlesyndication.com:443 status-code=200 title=Google have-screenshot=true
2024/09/25 08:31:37 INFO result 🤖 target=http://wordpress.org:80 status-code=200 title="Blog Tool, Publishing Platform, and CMS – WordPress.org" have-screenshot=true
2024/09/25 08:31:42 INFO result 🤖 target=http://wordpress.org:443 status-code=200 title="Blog Tool, Publishing Platform, and CMS – WordPress.org" have-screenshot=true
2024/09/25 08:31:46 INFO result 🤖 target=https://wordpress.org:443 status-code=200 title="Blog Tool, Publishing Platform, and CMS – WordPress.org" have-screenshot=true
2024/09/25 08:31:53 INFO result 🤖 target=http://icloud.com:80 status-code=200 title=iCloud have-screenshot=true
2024/09/25 08:31:57 INFO result 🤖 target=http://sharepoint.com:80 status-code=200 title="Microsoft SharePoint Online - Collaboration Software | Microsoft 365" have-screenshot=true
2024/09/25 08:31:58 INFO result 🤖 target=http://icloud.com:443 status-code=200 title=iCloud have-screenshot=true
2024/09/25 08:32:02 INFO result 🤖 target=https://icloud.com:443 status-code=200 title=iCloud have-screenshot=true
2024/09/25 08:32:05 INFO result 🤖 target=http://pinterest.com:80 status-code=200 title=Pinterest have-screenshot=true
2024/09/25 08:32:10 INFO result 🤖 target=http://pinterest.com:443 status-code=200 title=Pinterest have-screenshot=true
2024/09/25 08:32:14 INFO result 🤖 target=https://pinterest.com:443 status-code=200 title=Pinterest have-screenshot=true
2024/09/25 08:32:20 INFO result 🤖 target=http://yahoo.com:80 status-code=200 title="Yahoo is part of the Yahoo family of brands" have-screenshot=true
2024/09/25 08:32:25 INFO result 🤖 target=http://yahoo.com:443 status-code=200 title="Yahoo is part of the Yahoo family of brands" have-screenshot=true
2024/09/25 08:32:31 INFO result 🤖 target=https://yahoo.com:443 status-code=200 title="Yahoo is part of the Yahoo family of brands" have-screenshot=true
2024/09/25 08:32:32 INFO result 🤖 target=http://whatsapp.net:80 status-code=200 title="WhatsApp | Secure and Reliable Free Private Messaging and Calling" have-screenshot=true
2024/09/25 08:32:37 INFO result 🤖 target=http://whatsapp.net:443 status-code=200 title="WhatsApp | Secure and Reliable Free Private Messaging and Calling" have-screenshot=true
2024/09/25 08:32:40 INFO result 🤖 target=https://whatsapp.net:443 status-code=200 title="WhatsApp | Secure and Reliable Free Private Messaging and Calling" have-screenshot=true
2024/09/25 08:33:04 INFO result 🤖 target=http://mail.ru:80 status-code=200 title="Mail: Почта, Облако, Календарь, Заметки, Покупки — сервисы для работы и жизни" have-screenshot=true
2024/09/25 08:33:05 INFO result 🤖 target=http://mail.ru:443 status-code=200 title="Mail: Почта, Облако, Календарь, Заметки, Покупки — сервисы для работы и жизни" have-screenshot=true
2024/09/25 08:33:26 INFO result 🤖 target=http://digicert.com:443 status-code=200 title="TLS/SSL Certificate Authority | Leader in Digital Trust | DigiCert" have-screenshot=true
2024/09/25 08:33:27 INFO result 🤖 target=http://digicert.com:80 status-code=200 title="TLS/SSL Certificate Authority | Leader in Digital Trust | DigiCert" have-screenshot=true
2024/09/25 08:33:46 INFO result 🤖 target=https://digicert.com:443 status-code=200 title="TLS/SSL Certificate Authority | Leader in Digital Trust | DigiCert" have-screenshot=true
2024/09/25 08:33:46 INFO result 🤖 target=https://digicert.com:80 status-code=200 title="TLS/SSL Certificate Authority | Leader in Digital Trust | DigiCert" have-screenshot=true
2024/09/25 08:33:51 INFO result 🤖 target=https://tiktokv.com:443 status-code=404 title="404 Not Found" have-screenshot=true
2024/09/25 08:34:04 INFO result 🤖 target=http://msn.com:80 status-code=200 title="MSN South Africa | Latest News, Results, Celebrity, Hotmail & Outlook" have-screenshot=true
2024/09/25 08:34:16 INFO result 🤖 target=http://msn.com:443 status-code=200 title="MSN South Africa | Latest News, Results, Celebrity, Hotmail & Outlook" have-screenshot=true
2024/09/25 08:34:39 INFO result 🤖 target=https://msn.com:443 status-code=200 title="MSN South Africa | Latest News, Results, Celebrity, Hotmail & Outlook" have-screenshot=true
2024/09/25 08:34:58 INFO result 🤖 target=http://office365.com:80 status-code=200 title="Microsoft 365 - Subscription for Productivity Apps | Microsoft 365" have-screenshot=true
2024/09/25 08:35:14 INFO result 🤖 target=http://yandex.net:80 status-code=200 title="Are you not a robot?" have-screenshot=true
2024/09/25 08:35:24 INFO result 🤖 target=http://yandex.net:443 status-code=400 title=400 have-screenshot=true
2024/09/25 08:36:31 INFO result 🤖 target=https://wordpress.com:443 status-code=200 title="WordPress.com: Build a Site, Sell Your Stuff, Start a Blog & More" have-screenshot=true
2024/09/25 08:36:42 INFO result 🤖 target=http://zoom.us:80 status-code=200 title="One platform to connect | Zoom" have-screenshot=true
2024/09/25 08:36:53 INFO result 🤖 target=http://zoom.us:443 status-code=200 title="One platform to connect | Zoom" have-screenshot=true
2024/09/25 08:36:54 INFO result 🤖 target=http://whatsapp.com:80 status-code=200 title="WhatsApp | Secure and Reliable Free Private Messaging and Calling" have-screenshot=true
2024/09/25 08:36:57 INFO result 🤖 target=https://zoom.us:443 status-code=200 title="One platform to connect | Zoom" have-screenshot=true
2024/09/25 08:36:58 INFO result 🤖 target=https://cloudflare.net:443 status-code=403 title="Just a moment..." have-screenshot=true
2024/09/25 08:37:19 INFO result 🤖 target=http://qq.com:80 status-code=200 title=腾讯网 have-screenshot=true
2024/09/25 08:37:23 INFO result 🤖 target=http://qq.com:443 status-code=200 title=腾讯网 have-screenshot=true
2024/09/25 08:37:24 INFO result 🤖 target=https://qq.com:443 status-code=200 title=腾讯网 have-screenshot=true
2024/09/25 08:37:31 INFO result 🤖 target=http://google-analytics.com:80 status-code=200 title="Analytics Tools & Solutions for Your Business - Google Analytics" have-screenshot=true
2024/09/25 08:37:34 INFO result 🤖 target=http://google-analytics.com:443 status-code=200 title="Analytics Tools & Solutions for Your Business - Google Analytics" have-screenshot=true
2024/09/25 08:37:38 INFO result 🤖 target=https://google-analytics.com:443 status-code=200 title="Analytics Tools & Solutions for Your Business - Google Analytics" have-screenshot=true
2024/09/25 08:37:43 INFO result 🤖 target=http://tiktok.com:80 status-code=200 title="Explore - Find your favourite videos on TikTok" have-screenshot=true
2024/09/25 08:37:45 INFO result 🤖 target=http://tiktok.com:443 status-code=200 title="Explore - Find your favourite videos on TikTok" have-screenshot=true
2024/09/25 08:37:52 INFO result 🤖 target=https://tiktok.com:443 status-code=200 title="Explore - Find your favourite videos on TikTok" have-screenshot=true
2024/09/25 08:38:03 INFO result 🤖 target=http://blogspot.com:80 status-code=200 title="Blogger.com - Create a unique and beautiful blog easily." have-screenshot=true
2024/09/25 08:38:16 INFO result 🤖 target=https://blogspot.com:443 status-code=200 title="Blogger.com - Create a unique and beautiful blog easily." have-screenshot=true
2024/09/25 08:38:24 INFO result 🤖 target=http://reddit.com:80 status-code=200 title="Reddit - Dive into anything" have-screenshot=true
2024/09/25 08:38:27 INFO result 🤖 target=http://reddit.com:443 status-code=200 title="Reddit - Dive into anything" have-screenshot=true
2024/09/25 08:38:35 INFO result 🤖 target=https://reddit.com:443 status-code=200 title="Reddit - Dive into anything" have-screenshot=true
2024/09/25 08:38:36 INFO result 🤖 target=http://opera.com:80 status-code=200 title="Opera Web Browser | Faster, Safer, Smarter | Opera" have-screenshot=true
2024/09/25 08:38:44 INFO result 🤖 target=http://opera.com:443 status-code=200 title="Opera Web Browser | Faster, Safer, Smarter | Opera" have-screenshot=true
2024/09/25 08:38:47 INFO result 🤖 target=https://opera.com:443 status-code=200 title="Opera Web Browser | Faster, Safer, Smarter | Opera" have-screenshot=true
2024/09/25 08:38:52 INFO result 🤖 target=https://googleadservices.com:443 status-code=404 title="Error 404 (Not Found)!!1" have-screenshot=true
2024/09/25 08:38:52 INFO result 🤖 target=http://unity3d.com:80 status-code=200 title="Unity Real-Time Development Platform | 3D, 2D, VR & AR Engine" have-screenshot=true
2024/09/25 08:38:54 INFO result 🤖 target=http://snapchat.com:80 status-code=200 title="Less social media. More Snapchat." have-screenshot=true
2024/09/25 08:38:57 INFO result 🤖 target=http://snapchat.com:443 status-code=200 title="Less social media. More Snapchat." have-screenshot=true
2024/09/25 08:38:59 INFO result 🤖 target=https://snapchat.com:443 status-code=200 title="Less social media. More Snapchat." have-screenshot=true
2024/09/25 08:39:01 INFO result 🤖 target=http://trbcdn.net:80 status-code=403 title="403 Forbidden" have-screenshot=true
2024/09/25 08:39:02 INFO result 🤖 target=http://trbcdn.net:443 status-code=403 title="403 Forbidden" have-screenshot=true
2024/09/25 08:39:05 INFO result 🤖 target=https://trbcdn.net:443 status-code=403 title="403 Forbidden" have-screenshot=true
real 13m2.123s
user 178m59.817s
sys 2m13.824s
# ls
gowitness.jsonl gowitness.sqlite3 screenshots
# ls -lah
total 92M
drwxr-xr-x 1 root root 86 Sep 25 08:39 .
drwx------ 1 root root 36 Sep 25 08:26 ..
-rw-r--r-- 1 root root 34M Sep 25 08:39 gowitness.jsonl
-rw-r--r-- 1 root root 58M Sep 25 08:39 gowitness.sqlite3
drwxr-xr-x 1 root root 4.2K Sep 25 08:39 screenshots
# sqlite3 gowitness.sqlite3 "select url,response_code from results"
http://twitter.com:80|200
http://twitter.com:443|200
https://twitter.com:443|200
http://cloudflare.com:80|200
http://cloudflare.com:443|200
https://cloudflare.com:443|200
http://instagram.com:80|200
http://instagram.com:443|200
http://linkedin.com:443|200
https://linkedin.com:443|200
http://live.com:80|200
http://live.com:443|200
https://live.com:443|200
https://googletagmanager.com:443|404
http://fbcdn.net:80|200
... The next thing I'll test would be to wrap it all in a simple go program that shells out to invoke gowitness. |
Following up. I got it working in GitLab CI/CD using 3 threads and the --write-stdout option. However, it seems that the [INFO] /root/go/bin/gowitness -D scan file -f /output/httpx/urls.txt --screenshot-path /output/gowitness/screenshots --write-db --write-db-uri sqlite:///output/gowitness/gowitness.sqlite3 --write-jsonl --write-jsonl-file /output/gowitness/results.jsonl --write-stdout -t 3 --log-scan-errors --chrome-user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0'
<snippet/>
took: 19m16.840064119s
[INFO] Command=/root/go/bin/gowitness report generate --db-uri sqlite:///output/gowitness/gowitness.sqlite3 --zip-name /output/gowitness/report.zip --screenshot-path /output/gowitness/screenshots
2024/09/26 02:44:07 INFO generating HTML report for results count=0
2024/09/26 02:44:08 INFO report zip file generated successfully path=/output/gowitness/report.zip
took: 359.786308ms Not sure why the results |
Hi, For instance, this is what happens when I save data in a jsonl file and then I try to generate a report. Similarly, saving a sqlite3 db with a regular or custom name would work, but loading said DB from a directory other than the current one would fail when generating a report.
Note the empty As a workaround, I let gowitness write its default db file within the current directory, then generate a report based on that.
|
Created a new issue to track the report generation issues. Let's use this one for the concurrency related issue. |
Made small bits of progress in testing this. Using hosted gitlab, I wrote a pipeline that downloads a domains list, and then installs and runs gowitness both directly and using a another golang program to wrap gowitness. In both cases it seems to work fine. https://gitlab.com/leonjza/gowitness-cicd/-/pipelines/1477509973 Are there any obvious differences here to what you are doing @mr-pmillz ? |
Very nice! Looks good. In my case am using a self-hosted runner on self-hosted gitlab but in either case I think this is quite sufficient to mark this issue as resolved. The threads constraint for my CI/CD is likely a self-hosted runner resources issue. I believe the main issue now is the report generation using a custom sqlite db or jsonl file path. Thanks again! 🍻 |
Describe the bug
In a CI/CD pipeline, GoWitness v3.0.3 runs, but the screenshots and output from sqlite db / jsonl writers doesn't get written. Also there is no stdout.
I just noticed the
--write-stdout
flag which says: "Write successful results to stdout (usefull in a shell pipeline"Is this flag required for the other writer's output to get written in addition to stdout?
Would using go-rod instead of chrome driver in a pipeline be better in your opinion, or is this issue more related to a bug than a particular web driver?
To Reproduce
Steps to reproduce the behavior:
Run the following commands in a GitLab, GitHub, etc. CICD pipeline:
Expected behavior
I expected the screenshots directory to contain the resulting screenshots, the sqlite db to be populated, and the results.jsonl file to not be empty.
Version Information:
Additional context
When I run these commands in a regular terminal, it works as expected. But the same commands in a CICD pipeline does not write out the results to the screenshots, sqlite DB, or results.jsonl file.
The text was updated successfully, but these errors were encountered: