Questions regarding the implementation

[OnkarRuikar]

1. How many people have tested the feature before its public launch (in beta)? Have subject-matter experts assessed the quality of answers in areas like HTML, CSS, JavaScript, …?

2. Will Mozilla make it more clear which parts of MDN are products with commercial interests (sponsored content, Ads, AI) vs parts that are worked on by the volunteer community?

3. What is the purpose of “AI Explain”? Should reader turn to AI explanation when human explanation fails or is it other way around?
   I think explaining entire example is competing with human written pros/explanations. It is not efficient to read AI explanation first then read human explanation also.

4. I think AI explain button shins more while explaining individual selected lines. Can we limit it to only explain selected stuff and not the entire example?

5. Are you going to keep AI explain button open to all (guest users, logged in users, and paid members)? Or only logged in users?

6. Does the technical knowledge in the model solely comes from mdn/content repo or base model also contributes to it?

7. Are you using the same model for AI Explain and AI help? Or do they have different mechanisms behind them?

8. AI Explain, AI Help: How frequently the model is being trained? How long it takes to train it?

9. Is it possible to train every midnight or when mdn/content gets deployed? For many new APIs it says API(method/property) invalid.

10. Can you give setting to enable and disable the explain button, disabled by default?
    Those who are really optimistic about this will enable and use it.

11. Can you give control to authors about adding AI explain button on code blocks?
    We can have annotation like ai-explain on markdown code blocks like we have js-nolint.

```js ai-explain
// some simple code that AI won’t have trouble explaining
// mostly learning section simple code blocks
```

or

```js no-ai-explain
// complex code that AI don’t understand yet, and giving wrong explanation
```

So that author/maintainers can disable AI explain on the blocks where answers are not correct.
Or start with no explain on all blocks and enable only when the author feels it explains the block fairly.
Note: for this we do need PR companion to have AI explain facility so that maintainer can decide to enable or disable on the code blocks in PR.

12. Apart from thumbs up or down where should AI related bugs be reported? in Yari repo?

13. Is the model just auto correcting/reinforcing itself or there are humans involved to fix wrong answers/information?

14. Suppose content on a page is wrong and a PR corrects it. How does the model handles this change? Do we retrain the model from the start or existing model continuously evolves? How the unlearning wrong info happens?

15. Are you going to support other languages(non English) that are maintained in mdn/translated-content repo?

[caugner]

Hi @OnkarRuikar, we're tracking your questions (left) with the following numbers (right):

1. ➡️ 13
2. ➡️ 18
3. ➡️ 6
4. ➡️ 7
5. ➡️ 5
6. ➡️ 3
7. ➡️ 2
8. ➡️ 1
9. ➡️ 1
10. ➡️ 4
11. ➡️ 8
12. ➡️ 9
13. ➡️ 10
14. ➡️ 11
15. ➡️ 12

[caugner]

Hi @OnkarRuikar,

Let me answer the engineering-focused questions (questions 1-5 and 10-11 will be answered by colleagues):

6. Does the technical knowledge in the model solely comes from mdn/content repo or base model also contributes to it?

• For AI Help beta, we provide relevant MDN content as context to the LLM and prompt it specifically to use only that information to answer the question, which means it will prioritize information from the MDN content provided.
• Technically, it is impossible to prevent the LLM from using other information it was trained on, but our testing showed that it does work as expected in most cases, and that there are cases where it can be helpful when the MDN content doesn’t answer the question.

7. Are you using the same model for AI Explain and AI help? Or do they have different mechanisms behind them?
8. AI Explain, AI Help: How frequently the model is being trained? How long it takes to train it?
9. Is it possible to train every midnight or when mdn/content gets deployed? For many new APIs it says API(method/property) invalid.

• We don’t train our own model.
• Both AI Explain and AI Help beta use OpenAI GPT-3.5, but differently.
• AI Explain simply prompted GPT-3.5 to explain the code example.
• AI Help beta first determines the most relevant content sections on MDN using embedding-similarity search and passes these to GPT-3.5 along with the question and specific prompts on how to answer the question.
• The embedding search index is updated as part of our daily deployments.

12. Apart from thumbs up or down where should AI related bugs be reported? in Yari repo?

• If the bug is related to an answer that AI Help gave, please always report through the “Report a problem with this answer on GitHub” link, so that we have all the context to assess and follow up. (We received 7 issue reports so far.)
• Other bugs can be reported in the yari repository.

13. Is the model just auto correcting/reinforcing itself or there are humans involved to fix wrong answers/information?

• We are looking into ways of improving the helpfulness and correctness of AI Help answers, such as:
  • Improve how we find relevant content sections on MDN to answer the question.
  • Identify and close content gaps.
  • Identify limitations and adjust the prompts to reject unanswerable questions.
  • Experiment how adjustments to the LLM prompt impact helpfulness and correctness.
• We will use a set of reference questions to test that our improvements don’t cause any regressions.

14. Suppose content on a page is wrong and a PR corrects it. How does the model handles this change? Do we retrain the model from the start or existing model continuously evolves? How the unlearning wrong info happens?

• As mentioned above, we don’t train a model, so corrections are picked up every day during the usual builds.

15. Are you going to support other languages(non English) that are maintained in mdn/translated-content repo?

• AI Help beta currently supports answering questions in all languages that GPT-3.5 supports, and should answer in the same language.
• We may remove this, because we don’t have the capacity to review issue reports about questions in foreign languages.
• We don’t have plans to incorporate content from mdn/translated-content for the same reasons.

[LeoMcA]

3. What is the purpose of “AI Explain”? Should reader turn to AI explanation when human explanation fails or is it other way around?
   I think explaining entire example is competing with human written pros/explanations. It is not efficient to read AI explanation first then read human explanation also.

• In the "ideal" user flow - one where "AI Explain" or a feature like it wouldn't be necessary - a user would read the prose surrounding the example and understand the example itself
• However, sometimes, especially in the case of more junior developers, specific parts of the example might not make sense - those might be parts relating to the content on the page, or (IMO even more interestingly) parts not relating to the content on the page (e.g. an example on a page about, e.g. fetch, where the user doesn't understand async/await or arrow function syntax)
• In those latter cases it's not practical to explain every bit of an example on every page - our reference documentation would become very very long, we can't explain what a css selector is on every page about css for instance.
• The intent with AI explain was to help users get a useful explanation about those things, without having to open lots of tabs and jump across them to understand what just one example is doing/how it works

4. I think AI explain button shins more while explaining individual selected lines. Can we limit it to only explain selected stuff and not the entire example?

• Indeed, this was the bit I was very interested/excited/mind blown about - the level of context GPT could gather from the surrounding contextual clues to narrow in on what "this bit" was doing in the example
• As we've said in the postmortem this feature won't come back until 2024 at the earliest, so there's ample room for discussion about what form that might be in
• But for some very rough engineering thoughts (from me individually): the programming problem we'd have to solve to give GPT better context for explaining examples or selected lines would be very similar to the problem required to simply link statically from e.g. a particular css property in an example to its reference documentation (as that's the context we'd want to be feeding in). Perhaps that link itself is sufficient for making these examples easier to understand, without a "personalised" response, though it wouldn't solve the "many tabs" problem. Perhaps something like wikipedia-style document previews on hover would (which we've long talked about). Perhaps there's the possibility of a more "choose your own adventure" style experience where you can just rely on the reference documentation if you're comfortable with that, and then opt-in to a LLM generated one if you're comfortable with the possibility for inaccurate information within that.

5. Are you going to keep AI explain button open to all (guest users, logged in users, and paid members)? Or only logged in users?

• As acknowledged in the post-mortem, launching it open to all (i.e. not behind a login) was a mistake

10. Can you give setting to enable and disable the explain button, disabled by default? Those who are really optimistic about this will enable and use it.

• Perhaps: we've long discussed (internally and with various community members) the possibilities around crafting a "customised" MDN experience, everything from the UI (as it would be in this case) to the actual structure/layout/navigation of documentation ("e.g. don't show things in search results which can't be used in X browser") - none of this is concrete, just thoughts we've thrown around
• In the explicit context of "AI Explain", as mentioned we won't be prioritising work on it before the end of the year at the earliest

11. Can you give control to authors about adding AI explain button on code blocks? We can have annotation like ai-explain on markdown code blocks like we have js-nolint.

• My own rough (personal) thoughts again: there's a certain irony here that the feature was intended to be supplemental to the huge amounts of work authors already do, but this would increase the amount of work they'd need to do, and make contributing a more complex process
• But it's certainly something we can consider and discuss with content contributors when revisiting the feature