Add nap for telemetry #226

Czaki · 2023-08-11T13:41:43Z

Description

Add NAP with the proposition of a telemetry system.

Type of change

Adds new content page(s)

DragaDoncila

@Czaki this is looking great, thank you for all your work on this 🎉 ❤️ ! I've done a pass of suggestions for wording etc. Once those updates are made and I've made sure the formatting is correct, I'll be happy to approve the draft.

In terms of comments on the proposal itself these are my general thoughts:

I would like to see us include in the NAP a process for getting community feedback and concerns before implementation. While the NAP is usually the way to publicize big changes like this, I think they're mostly watched/discussed by core devs. Since telemetry can be a controversial topic, I would like us to include a process here that ensures the community is aware of and on board with the proposal
I think before accepting the NAP, I would like to see a complete list (as complete as possible) of exactly what is collected with each level of telemetry, and potentially also a sample report, and what would show up on the public dashboards
We mention collecting information on what plugin contributions are used, but I think it would be great to collect information on what internal commands are used as well. It would give us great insight into popular features, potentially underused/underpublicized features and "danger zones" for API deprecations. Not sure to what extent this is possible, but once the app-model conversions are done there should be a few things we can learn?
I would like us to expand a little on the motivation for telemetry beyond "gauging user numbers" which imo is the weakest motivation. For me it's really more about learning what plugins and parts of the codebase are seeing heavy usage, predicting the impact of changing APIs and gaining insight into what could be deprecated, changed, better documented, etc.

docs/naps/8-telemetry.md

Co-authored-by: Draga Doncila Pop <[email protected]>

Czaki · 2023-08-18T22:45:43Z

I would like us to expand a little on the motivation for telemetry beyond "gauging user numbers" which imo is the weakest motivation. For me it's really more about learning what plugins and parts of the codebase are seeing heavy usage, predicting the impact of changing APIs and gaining insight into what could be deprecated, changed, better documented, etc.

This is mentioned in the motivation and scope paragraph. Should I rephase it?

We mention collecting information on what plugin contributions are used, but I think it would be great to collect information on what internal commands are used as well

great idea. I will add this in the following days.

I also agree with polishing the list of collected data. Opening this PR is way to open it on the community. I think, that after some polishing we should make a public announcement.

Carreau · 2023-09-11T14:09:51Z

docs/naps/8-telemetry.md

+1. Disable in settings
+2. uninstall `napari-telemetry` package
+3. Environment variable `NAPARI_TELEMETRY=0`
+4. Full list of endpoints used for collecting telemetry, that could be filtered on the firewall level.


Suggested change

4. Full list of endpoints used for collecting telemetry, that could be filtered on the firewall level.

5 . Disable at system-wide level in a way that is not user-controlable (Hpc/ other deployments).

You mean to add new point?

Possibly we should merge the two sentences:

System-wide disablement e.g. via firewall filtering for hpc or other environments

docs/naps/8-telemetry.md

Carreau · 2023-09-11T14:12:37Z

docs/naps/8-telemetry.md

+
+Also collecting information about data types and their size will provide valuable information about the typical use cases of napari.
+
+Still, users need to be able to opt out of such monitoring, and adjust the level of detail of the information that is sent to the napari server.


Detail, I would store the date when someone agreed, and maybe reconfirm every few month or so.

You might need a way for the remote to reply with a specific error code/version number that void the user consent, and retrigger asking consent as well.

Why do you prefer longer storage?

I think the proposal is that we reconfirm opt-in every few months or so. I think this is a good idea and worth adding.

I still do not follow.
Did we want to reconfirm every three months (for example), or when we update the telemetry code, so change the amount of collected information?

I think it should be min(3_months, package_update) for re-confirming, personally.

The dialog for whether you want to continue sending telemetry could show an example of a recent report that had been sent, so the user can gauge their participation based on real data. We could always add a Don't ask me again option as well of course.

docs/naps/8-telemetry.md

Carreau · 2023-09-11T14:18:40Z

docs/naps/8-telemetry.md

+
+Another option is to scan public plugins and their dependencies. This is simpler but will require establishing additional communication channels to be able to warn users about the potential problem. 
+
+


You will need to describe the way people can contact napari for their data to be removed.

You have to be careful, login IP in the machine that collect logs may be considered collecting PII in some regions.

My Idea is to at the first phase, not to log IP. I do not see the benefit of such information and only risk.

I ha e added GDPR section

Sorry I was unclear, if for example you run the collection service behind nginx, the nginx logs may store IP. So you have to be careful that not only the service you write but part of the stack does not store data and/or that you can delete it.

Some storage is required (Identifying spamming users), but it may be short with fast removal.
And such things will not be propagated to the aggregator.

docs/naps/8-telemetry.md

DragaDoncila

@Czaki I have done a final pass and once that's done I think this should go in in draft mode. Once we have some sample reports to look at, we should be able to add any other necessary detail. It would be nice to have a basic architecture diagram as well.

docs/naps/8-telemetry.md

DragaDoncila · 2023-10-18T07:05:20Z

docs/naps/8-telemetry.md

+1. Disable in settings
+2. uninstall `napari-telemetry` package
+3. Environment variable `NAPARI_TELEMETRY=0`
+4. Full list of endpoints used for collecting telemetry, that could be filtered on the firewall level.


Possibly we should merge the two sentences:

System-wide disablement e.g. via firewall filtering for hpc or other environments

docs/naps/8-telemetry.md

DragaDoncila · 2023-10-18T07:06:24Z

docs/naps/8-telemetry.md

+
+Also collecting information about data types and their size will provide valuable information about the typical use cases of napari.
+
+Still, users need to be able to opt out of such monitoring, and adjust the level of detail of the information that is sent to the napari server.


I think the proposal is that we reconfirm opt-in every few months or so. I think this is a good idea and worth adding.

docs/naps/8-telemetry.md

Co-authored-by: Draga Doncila Pop <[email protected]>

Czaki · 2023-10-18T14:51:22Z

dependency check is failing because of bug in main repo gregsdennis/dependencies-action#24

DragaDoncila · 2023-10-22T00:23:24Z

docs/naps/8-telemetry.md

+
+ ```{eval-rst} 
+ :Author: Grzegorz Bokota 
+ :Created: <date created on, in yyyy-mm-dd format> 


@Czaki you should update this, I think it's fine to date it as the day you opened the PR.

docs/naps/8-telemetry.md

DragaDoncila

@Czaki I think this is fine to go in whenever in draft mode. Thanks for all your work! Any further updates can be made in future PRs as is standard for NAPs

brisvag

Agreed, let's get this in and iterate in a PR.

add nap for telemetry

f8f3228

github-actions bot added the documentation Improvements or additions to documentation label Aug 11, 2023

github-actions bot assigned Czaki Aug 11, 2023

Czaki added this to the 0.5.0 milestone Aug 11, 2023

DragaDoncila reviewed Aug 18, 2023

View reviewed changes

Czaki and others added 4 commits August 18, 2023 23:54

Apply suggestions from code review

966430e

Co-authored-by: Draga Doncila Pop <[email protected]>

Update docs/naps/8-telemetry.md

33c62e3

Co-authored-by: Draga Doncila Pop <[email protected]>

add disabling telemetry in settings

d30adbf

fix toc

848465c

Carreau reviewed Sep 11, 2023

View reviewed changes

docs/naps/8-telemetry.md Outdated Show resolved Hide resolved

Carreau reviewed Sep 11, 2023

View reviewed changes

docs/naps/8-telemetry.md Outdated Show resolved Hide resolved

Carreau reviewed Sep 11, 2023

View reviewed changes

Czaki added 2 commits September 12, 2023 11:26

clarrify data about which plugin will be collected

2058613

add gdpr information

78e9e1c

soupault reviewed Sep 25, 2023

View reviewed changes

docs/naps/8-telemetry.md Outdated Show resolved Hide resolved

soupault reviewed Sep 25, 2023

View reviewed changes

docs/naps/8-telemetry.md Outdated Show resolved Hide resolved

soupault reviewed Sep 25, 2023

View reviewed changes

docs/naps/8-telemetry.md Show resolved Hide resolved

soupault reviewed Sep 25, 2023

View reviewed changes

docs/naps/8-telemetry.md Outdated Show resolved Hide resolved

soupault reviewed Sep 25, 2023

View reviewed changes

docs/naps/8-telemetry.md Show resolved Hide resolved

soupault approved these changes Sep 25, 2023

View reviewed changes

fixes from review

3e17ac6

Czaki mentioned this pull request Oct 14, 2023

Add NAP for multicanvas #249

Merged

DragaDoncila reviewed Oct 18, 2023

View reviewed changes

Czaki commented Oct 18, 2023

View reviewed changes

docs/naps/8-telemetry.md Outdated Show resolved Hide resolved

Apply suggestions from code review

d4c5d9c

Co-authored-by: Draga Doncila Pop <[email protected]>

DragaDoncila reviewed Oct 22, 2023

View reviewed changes

Czaki added 2 commits October 22, 2023 18:50

fox changes from code review

a5c1cc3

reformat text to reduce need of word wrap

0116f73

Czaki commented Oct 22, 2023

View reviewed changes

docs/naps/8-telemetry.md Outdated Show resolved Hide resolved

DragaDoncila approved these changes Oct 23, 2023

View reviewed changes

brisvag approved these changes Oct 23, 2023

View reviewed changes

brisvag added the ready to merge label Oct 23, 2023

remove obsolete section

8469c16

DragaDoncila merged commit e6cc2b5 into napari:main Oct 23, 2023
3 checks passed

Czaki deleted the nap_telemetry branch October 23, 2023 20:30

psobolewskiPhD removed the ready to merge label Oct 26, 2023

psobolewskiPhD modified the milestones: 0.5.0, 0.4.19 Oct 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nap for telemetry #226

Add nap for telemetry #226

Czaki commented Aug 11, 2023

DragaDoncila left a comment

Czaki commented Aug 18, 2023

Carreau Sep 11, 2023

Czaki Oct 12, 2023

DragaDoncila Oct 18, 2023

Carreau Sep 11, 2023

Czaki Sep 12, 2023

DragaDoncila Oct 18, 2023

Czaki Oct 18, 2023

DragaDoncila Oct 22, 2023

Carreau Sep 11, 2023

Czaki Sep 12, 2023

Carreau Sep 12, 2023

Czaki Sep 12, 2023

DragaDoncila left a comment

DragaDoncila Oct 18, 2023

DragaDoncila Oct 18, 2023

Czaki commented Oct 18, 2023

DragaDoncila Oct 22, 2023

DragaDoncila left a comment

brisvag left a comment

	4. Full list of endpoints used for collecting telemetry, that could be filtered on the firewall level.
	5 . Disable at system-wide level in a way that is not user-controlable (Hpc/ other deployments).


		Also collecting information about data types and their size will provide valuable information about the typical use cases of napari.

		Still, users need to be able to opt out of such monitoring, and adjust the level of detail of the information that is sent to the napari server.


		Another option is to scan public plugins and their dependencies. This is simpler but will require establishing additional communication channels to be able to warn users about the potential problem.

Add nap for telemetry #226

Add nap for telemetry #226

Conversation

Czaki commented Aug 11, 2023

Description

Type of change

DragaDoncila left a comment

Choose a reason for hiding this comment

Czaki commented Aug 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DragaDoncila left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Czaki commented Oct 18, 2023

Choose a reason for hiding this comment

DragaDoncila left a comment

Choose a reason for hiding this comment

brisvag left a comment

Choose a reason for hiding this comment