winall spy

CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## 3.1 (2016/06/07)
+
+* Add Windows 7 and Windows 8.1 hosts and firewall rules (Issue #1)
+* Add Wireshark script to extract log and generate CSV (Issue #6)
+* Bug spy rule blocking Windows update (Issue #14)
+* Add diff script to compare current firewall rules / hosts with generated CSVs
+* New hosts and firewall rules
+
+## 3.0 (2016/06/03)
+
+* Add Sysmon, Proxifier, Wireshark capture method in the [Wiki](../../wiki) (Issue #11)
+* Enhancement for firewall script (Issue #2)
+* Separate rules and scripts in distinct folders
+* New hosts and firewall rules
+* Add capture logs in CSV files
+* Add Sysmon script (install / uninstall / extract event log)
+* Add Proxifier script (extract log)
+
 ## 2.7 (2016/05/27)
 
 * Add NCSI alternative probe (Issue #9)

README.md

@@ -1,74 +1,97 @@
 # Windows Spy Blocker [![Donate Paypal](https://img.shields.io/badge/donate-paypal-blue.svg)](https://www.paypal.me/crazyws)
 
-Rules to block Windows spy / telemetry.
-
 ![](../../wiki/img/logo-20160521.png)
 
 <!-- START doctoc generated TOC please keep comment here to allow auto update -->
 <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 
 
+- [About](#about)
 - [How ?](#how-)
 - [Usage](#usage)
-  - [Hosts](#hosts)
-  - [Firewall](#firewall)
-  - [NCSI (Network Connectivity Status Indicator)](#ncsi-network-connectivity-status-indicator)
-  - [DNSCrypt](#dnscrypt)
-  - [Proxifier](#proxifier)
+  - [Data](#data)
+    - [Hosts](#hosts)
+    - [Firewall](#firewall)
+    - [NCSI (Network Connectivity Status Indicator)](#ncsi-network-connectivity-status-indicator)
+    - [DNSCrypt](#dnscrypt)
+    - [Proxifier](#proxifier)
+  - [Logs](#logs)
+  - [Scripts](#scripts)
 - [Projects using WindowsSpyBlocker](#projects-using-windowsspyblocker)
 - [License](#license)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
+## About
+
+**WindowsSpyBlocker** is a set of rules to block Windows spy / telemetry based on multiple tools to [capture traffic](../../wiki/Capture%20traffic). It is open for everyone and if you want to contribute, take a look at the [Wiki](../../wiki).<br />
+To be notified of new releases you can subscribe to this [Atom feed](https://github.com/crazy-max/WindowsSpyBlocker/releases.atom).
+
 ## How ?
 
-I use a QEMU virtual machine on the server virtualization management platform [Proxmox VE](https://www.proxmox.com/en/) based on Windows 10 Pro 64bits with automatic updates enabled.<br />
-I clean traffic dumps every day and compare results with the current rules to add / remove some hosts or firewall rules (need to automate the process...).
+I use a QEMU virtual machines on the server virtualization management platform [Proxmox VE](https://www.proxmox.com/en/) based on :
+
+* Windows 10 Pro 64bits with automatic updates enabled.
+* Windows 8.1 Pro 64bits with automatic updates enabled.
+* Windows 7 SP1 Pro 64bits with automatic updates enabled.
+
+I clean traffic dumps every day and compare results with the current rules to add / remove some hosts or firewall rules.
 
 Tools used to capture traffic :
-* qemu -net dump
-* Wireshark
+* **qemu -net dump** : capture
+* **[Wireshark](../../wiki/captureWireshark)** : capture + logs
+* **[Sysmon](../../wiki/captureSysmon)** : capture + logs
+* **[Proxifier](../../wiki/captureProxifier)** : logs
+
+All traffic events are available in the [logs](#logs) folder.
 
 ## Usage
 
-### Hosts
+### Data
+
+`data` is the master folder of this project. It contains the blocking rules based on domain names or IPs addresses detected during the capture process.
+* `data/<type>/winX/spy.txt` : Block Windows Spy / Telemetry
+* `data/<type>/winX/update.txt` : Block Windows Update
+* `data/<type>/winX/extra.txt` : Block third party applications
 
-* `windowsX_spy.txt` : Block Windows Spy / Telemetry
-* `windowsX_update.txt` : Block Windows Update
-* `windowsX_extra.txt` : Block third party applications
+#### Hosts
 
-Copy / paste the content of the above files in your Windows hosts file located in `C:\Windows\System32\drivers\etc\hosts`.<br />
+Copy / paste the content of the files in `data/hosts` in your Windows hosts file located in `C:\Windows\System32\drivers\etc\hosts`.<br />
 
 You can use the [HostsMan](http://www.abelhadigital.com/hostsman) freeware to keep update your hosts file.<br />
 I have created a git hook to publish the hosts files to my personal website :
-* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/windows10_spy.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/windows10_spy.txt)
-* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/windows10_update.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/windows10_update.txt)
-* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/windows10_extra.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/windows10_extra.txt)
 
-### Firewall
+##### Windows 7
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win7/spy.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win7/spy.txt)
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win7/update.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win7/update.txt)
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win7/extra.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win7/extra.txt)
 
-Some queries use IP addresses but you can stop them with your Firewall.<br />
-All relative information about these IP addresses are listed in the CSV file [firewallTestIPs.csv](https://github.com/crazy-max/WindowsSpyBlocker/blob/master/firewall/firewallTestIPs.csv).<br />
-[Download](https://github.com/crazy-max/WindowsSpyBlocker/archive/master.zip) or clone the repository, execute `firewall\firewallBlockWindowsSpy.bat` and choose an option :<br />
+##### Windows 8.1
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win81/spy.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win81/spy.txt)
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win81/update.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win81/update.txt)
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win81/extra.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win81/extra.txt)
 
-![](../../wiki/img/firewallMenu-20160516.png)
+##### Windows 10
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win10/spy.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win10/spy.txt)
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win10/update.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win10/update.txt)
+* [http://www.crazyws.fr/WindowsSpyBlocker/hosts/win10/extra.txt](http://www.crazyws.fr/WindowsSpyBlocker/hosts/win10/extra.txt)
 
-IPs are added in the Windows Firewall as outbound rules :<br />
+#### Firewall
 
-![](../../wiki/img/firewallRules-20160516.png)
+Some queries use IP addresses but you can stop them with your Firewall.<br />
+All relative information about these IP addresses are listed in the CSV files `firewall-` in the [logs folder](logs).<br />
+To add / remove firewall rules or test IPs, read the instructions in [scripts/firewall folder](scripts/firewall).
 
-### NCSI (Network Connectivity Status Indicator)
+#### NCSI (Network Connectivity Status Indicator)
 
 Windows check a Microsoft site for connectivity, using the Network Connectivity Status Indicator site.<br />
 NCSI performs a DNS lookup on `www.msftncsi.com` and sends a DNS lookup request for `dns.msftncsi.com`.<br />
-You can block this probe by adding the content of the `windowsX_extra.txt` hosts file.<br />
+You can block this probe by adding the content of the `data/<type>/winX/extra.txt` hosts file.<br />
 
 But you will have a ["No Internet access" warning in your system tray](../../wiki/FAQ#no-internet-access-on-my-network-card).<br />
-To solve this problem you can use the alternative WindowsSpyBlocker NCSI by executing `ncsi\ncsi.bat` :<br />
+To solve this problem you can use the alternative WindowsSpyBlcoker NCSI. Read the instructions in [scripts/ncsi folder](scripts/ncsi).
 
-![](../../wiki/img/ncsiMenu-20160527.png)
-
-### DNSCrypt
+#### DNSCrypt
 
 [DNSCrypt](https://dnscrypt.org/) is a protocol for securing communications between a client and a DNS resolver. With this tool you can blacklist some domains with the plugin [libdcplugin_example_ldns_blocking](https://github.com/jedisct1/dnscrypt-proxy#plugins) and add domains with leading and trailing wildcards.<br />
 To install DNSCrypt on Windows, read the [README-WINDOWS](https://github.com/jedisct1/dnscrypt-proxy/blob/master/README-WINDOWS.markdown) on the official GitHub repository.<br />
@@ -80,18 +103,36 @@ dnscrypt-proxy -R <name> --plugin=libdcplugin_example_ldns_blocking.dll,--domain
 
 Replace `<name>` with a [public DNS resolvers supporting DNSCrypt](https://github.com/jedisct1/dnscrypt-proxy/blob/master/dnscrypt-resolvers.csv) you want to use. Note its name, in the first column (for example: `dnscrypt.org-fr`).
 
-### Proxifier
+#### Proxifier
 
 Some hosts are not blocked and required a top level application.<br />
 For example you can use [Proxifier](https://www.proxifier.com/) software to block Microsoft spy.<br />
-Copy the content of the proxifier files in the repository in a blocked rule :
+Copy the content of the proxifier files in `data/proxifier` in a blocked rule :
 
 ![](../../wiki/img/proxifierRules-20160516.png)
 
+### Logs
+
+Logs of tools used to capture traffic and resolution of firewall rules in CSV format available in the [logs folder](logs).
+* `*-all.csv` : all events
+* `*-hosts-count.csv` : number of events per host
+* `*-unique.csv` : first trigger of an event per host / process / destination port
+
+### Scripts
+
+Several scripts are used to ease implementation of rules and contribution. To use these scripts you have to download and install the [Visual C++ Redistributable for Visual Studio 2012](https://www.microsoft.com/en-us/download/details.aspx?id=30679) (vcredist_x86.exe).
+* `diff.bat` : Generate a diff log based on CSV logs and data for Sysmon, Proxifier and Wireshark.
+* `firewall.bat` : Add / remove rules and resolve IPs adresses
+* `ncsi.bat` : Apply an alternate NCSI and test your internet connection the Micrososft way. More info on the [FAQ Wiki page](../../wiki/FAQ#what-is-ncsi-).
+* `proxifier.bat` : Extract events from log and generate CSV files. More info on the [Proxifier Wiki page](../../wiki/captureProxifier)
+* `sysmon.bat` : Install / uninstall Sysmon and extract events log then generate CSV files. More info on the [Sysmon Wiki page](../../wiki/captureSysmon)
+* `wireshark.bat` : Extract events log then generate CSV files based on IPv4 hosts. More info on the [Wireshark Wiki page](../../wiki/captureWireshark)
+
 ## Projects using WindowsSpyBlocker
 
 * [pi-hole](https://pi-hole.net/) : A black hole for Internet advertisements (designed for Raspberry Pi).
 * [StopAd](http://stopad.generate.club/) : Service for MikroTik routers made to block "advertising" and more.
+* [OpenWrt adblock package](https://github.com/openwrt/packages/tree/master/net/adblock/files) : DNS based ad/abuse domain blocking
 
 ## License
 

data/.gitignore

@@ -0,0 +1 @@
+*.tmp

data/dnscrypt/win10/extra.txt

@@ -0,0 +1,19 @@
+*.2mdn.net
+*.akamaitechnologies.com
+apps.skype.com
+cdn.content.prod.cms.msn.com
+choice.microsoft.com.nstac.net
+client.wns.windows.com
+dmd.metaservices.microsoft.com
+img-s-msn-com.akamaized.net
+licensing.mp.microsoft.com
+*.messenger.live.com
+*.msftncsi.com
+oneclient.sfx.ms
+pricelist.skype.com
+search.msn.com
+tk2.plt.msn.com
+ui.skype.com
+view.atdmt.com
+*.weather.microsoft.com
+*.xboxlive.com

data/dnscrypt/win10/spy.txt

@@ -0,0 +1,75 @@
+*.a-msedge.net
+*.ads*.msn.com
+*.ads*.msads.net
+ac3.msn.com
+*.adnexus.net
+*.adnxs.com
+*.ams*.msecn.net
+*.atdmt.com
+aka-cdn-ns.adtech.de
+answers.microsoft.com
+bingads.microsoft.com
+bs.serving-sys.com
+c.msn.com
+cache.datamart.windows.com
+choice.microsoft.com
+choice.microsoft.com.nsatc.net
+compatexchange.cloudapp.net
+compatexchange1.trafficmanager.net
+corp.sts.microsoft.com
+cs1.wpc.v0cdn.net
+db3wns2011111.wns.windows.com
+db5sch101101939.wns.windows.com
+db5sch103082111.wns.windows.com
+diagnostics.support.microsoft.com
+displaycatalog.mp.microsoft.com
+eu.vortex.data.microsoft.com
+feedback.microsoft-hohm.com
+feedback.search.microsoft.com
+feedback.windows.com
+flex.msn.com
+g.msn.com
+geover-prod.do.dsp.mp.microsoft.com
+*.glbdns2.microsoft.com
+h1.msn.com
+h2.msn.com
+lb1.www.ms.akadns.net
+*.location.live.net
+m.adnxs.com
+m.hotmail.com
+*.msedge.net
+msntest.serving-sys.com
+officeclient.microsoft.com
+onesettings-*.metron.live.com.nsatc.net
+pre.footprintpredict.com
+preview.msn.com
+*.rad.live.com
+*.rad.msn.com
+*.rads.msn.com
+redir.metaservices.microsoft.com
+schemas.microsoft.akadns.net
+secure.flashtalking.com
+*.services.social.microsoft.com
+settings-sandbox.data.microsoft.com
+settings-win.data.microsoft.com
+settings.data.microsoft.com
+spynet2.microsoft.com
+spynetalt.microsoft.com
+ssw.live.com
+statsfe1.ws.microsoft.com
+statsfe2.ws.microsoft.com
+survey.watson.microsoft.com
+*.telemetry.appex.bing.net
+*.telemetry.microsoft.com
+*.telemetry.microsoft.com.nsatc.net
+*.telemetry.urs.microsoft.com
+tsfe.trafficshaping.dsp.mp.microsoft.com
+version.hybrid.api.here.com
+*.virtualearth.net
+*.vo.msecnd.net
+*.vortex*.data.microsoft.com
+*.vortex*.data.metron.live.com.nsatc.net
+vortex-*.metron.live.com.nsatc.net
+watson.live.com
+watson.microsoft.com
+win10.ipv6.microsoft.com

data/dnscrypt/win10/update.txt

@@ -0,0 +1,5 @@
+*.delivery.dsp.mp.microsoft.com.nsatc.net
+*.delivery.mp.microsoft.com
+*.windowsupdate.com
+*.update.microsoft.com
+*.update.microsoft.com.akadns.net

data/dnscrypt/win7/extra.txt

@@ -0,0 +1,7 @@
+crl.microsoft.com
+dmd.metaservices.microsoft.com
+g.bing.com
+*.msftncsi.com
+time.windows.com
+wscont.apps.microsoft.com
+wscont.apps.microsoft.com.edgesuite.net

data/dnscrypt/win7/spy.txt

@@ -0,0 +1,9 @@
+compatexchange1.trafficmanager.net
+crl.microsoft.com
+eu.vortex.data.microsoft.com
+settings-win.data.microsoft.com
+spynet2.microsoft.com
+*.telemetry.microsoft.com
+teredo.ipv6.microsoft.com
+*.vortex*.data.microsoft.com
+watson.microsoft.com

dnscrypt/windows10_update.txt → data/dnscrypt/win7/update.txt

@@ -1,4 +1,3 @@
-*.windowsupdate.com
-windowsupdate.com
+download.microsoft.com
 *.update.microsoft.com
-update.microsoft.com
+*.windowsupdate.com

data/dnscrypt/win81/extra.txt

@@ -0,0 +1,18 @@
+activation-v2.sls.microsoft.com
+appex-rf.msn.com
+dmd.metaservices.microsoft.com
+g.bing.com
+img.stb.s-msn.com
+login.live.com
+*.msftncsi.com
+next-services.apps.microsoft.com
+*.services.appex.bing.com
+*.smartscreen.microsoft.com
+*.tile.appex.bing.com
+validation-v2.sls.microsoft.com
+watson.telemetry.microsoft.com
+*.weather.microsoft.com
+wscont.apps.microsoft.com
+wscont.apps.microsoft.com.edgesuite.net
+wscont1.apps.microsoft.com
+wscont2.apps.microsoft.com

data/dnscrypt/win81/spy.txt

@@ -0,0 +1,10 @@
+*.a-msedge.net
+settings-win.data.microsoft.com
+spynet2.microsoft.com
+spynetalt.microsoft.com
+statsfe2.update.microsoft.com
+statsfe2.ws.microsoft.com
+*.telemetry.microsoft.com
+*.trafficmanager.net
+*.vortex.data.microsoft.com
+vortex-win.data.microsoft.com

data/dnscrypt/win81/update.txt

@@ -0,0 +1,10 @@
+bg.v4.a.dl.ws.microsoft.com
+bg.v4.emdl.ws.microsoft.com
+bg1.v4.a.dl.ws.microsoft.com
+bg1.v4.emdl.ws.microsoft.com
+bg5.v4.a.dl.ws.microsoft.com
+bg5.v4.emdl.ws.microsoft.com
+fe2.ws.microsoft.com
+fg.v4.download.windowsupdate.com
+*.update.microsoft.com
+*.windowsupdate.com

data/firewall/win10/extra.txt

@@ -0,0 +1,19 @@
+### firewall win10 extra
+### More info: https://github.com/crazy-max/WindowsSpyBlocker
+
+65.52.100.11
+65.52.100.93
+191.232.139.2
+191.232.139.7
+191.232.139.49
+191.232.139.56
+191.232.139.141
+191.232.139.170
+191.232.139.182
+191.232.139.253
+207.46.194.14
+207.46.194.25
+207.46.194.33
+207.46.223.94
+207.68.166.254
+213.198.96.195