Thoughts on next steps?

[NunoSempere]

In my mind, some good next steps are:

• Add history to the graphql API (maybe [RFC] New event-based history storage #50)?
• Add charts functionality, both as an endpoint and in dashboards/frontpage, maybe as a toggle
• Integrate into LessWrong/EA Forum. As an ideal goal, maybe a newspaper through Global Guessing. (Incorporate dashboards and forecasts into lesswrong/EA Forum. #27)
• Add resolutions Save resolutions #10
• ...

Thoughts about what you want to work on next? There are also a few hanging threads:

• Improve the twitter integration (Automatically post new questions on twitter? #19). For this, having an endpoint with new questions added in the last 24h would be great
• General infrastructure improvements
• Get image previews (Suggestion: Generate image previews using Vercel #58)
• Better documentation (Documentation for grapqhl usage #65)

[berekuk]

Ok, a few various thoughts first.

• on UI & user-facing features
• on the data layer
• on infra & internals

UI:

Adding a separate page for each question is an obvious low-hanging fruit. I'd like to do this soon just to have a place to add other stuff to them, e.g. charts.

If Tools for forecasters is still in scope, then we need to start some ground work for it, e.g. adding logins. Maybe come up with a minimal useful feature which requires logins, e.g. editing dashboards or question notifications and implement such feature together with infra for logins? I'm not sure how to prioritize this compared to history and charts, user accounts can be pretty complex to implement well, and "personal forecasting tools" is a big project by itself, though it's not hard to MVP some small part of it.

Charts are cool! I'm less sure if we should use an external lib or just implement them from scratch, btw, maybe with something like d3 (which is much lower-level than Victory or Nivo). But I'll maybe try to argue in favor of this on #63 in more detail, later.

---

On the data layer, I'd like to spend some time figuring out how our data (graphql objects and fields, shape of data in general) should look like. I've been thinking about metaforecast as a project for unifying forecast formats from different platforms, so maybe we sketch the common ontology a bit more with an eye towards future extensibility and unification.

• We could scrape other stuff from forecasting platforms, e.g. comments, user profiles, historical data when we haven't got it in time ourselves, etc.; how do we store it and how do we expose it in graphql?
• Or I can imagine a scenario where for some kind of platform we fetch all individual forecasts for a given question and do the aggregation ourselves; how would this look? question(id: "whatever") { id title aggregateForecast(algo: "average") { ... } }; are there different types of aggregate forecasts, and should we consider the "official" aggregate forecast by a platform to be just of of many?
• How do we deal with various inconsistencies? Can we fit all probabilities formats from various platforms into our database, e.g. distributions from metaculus?
• Can we come up with a common API for questions & forecasts and promote them upstream, a la RSS? :) (if metaforecast & forecasting field in general becomes popular enough, I can imagine something like this happening eventually)

I'm not saying we should spend much time on these now, and we can just evolve our schema depending on the current needs. This very much depends on a longer-term vision for metaforecast.

If you like, I can spend a few hours sketching a proposal for a clearer API, though. But maybe I should become more familiar with various platforms specifics first.

Re: history in graphql, I don't have a clear understanding if there are going to be any demand for API and how important it is. But if there is a demand then designing a robust graphql schema upfront might be worthwhile too.

---

On infra:

• I really want to finish typescript refactoring since it makes code changes much much safer
• we need to normalize the DB further to make it strictly typed (and [RFC] New event-based history storage #50 is important too)

I also think #35 is pretty important. The main reason I'd use another platform instead of metaforecast is that I can't be sure if metaforecast has the last version of data. I think we should strive for something like "all data is less than 5 minute old", and probably work with platforms re: push webhooks.

---

My own preference on the order of doing things is:

1. finish prisma transition since there are only a few parts of code left which use pg-wrapper, shouldn't take long, a few hours, maybe
2. spend a day or two on the frontend in general, e.g. add separate question pages and do some minor UI polishing
3. add charts (on question pages, at least, I'm less sure about fitting charts into cards on the frontpage, but that depends on our charting lib)
4. improve dashboards (add a proper form instead of json textarea; integrate into LW)
5. then go back to infra, finalize the typescript transition and do some DB refactorings (get rid of json fields in favor of columns)
6. (4) makes sense together with something else related to fetchers code, since I'll have to familiarize myself with those anyway for rewriting with proper typescript; this'll be a good time to touch on Save resolutions #10 and on Independent update schedules for different platforms #35
7. Doing (4) and (5) will be a good point to discuss our data layer and how we want to develop it further.

I'm less sure about doing #50 now. I feel like I won't be able to figure out the proper approach to it on the first iteration, and it might turn out to be pretty complex. I still think it should be done eventually, but I don't want it to cause any delays for other obviously needed features.

[OAGr]

A few quick thoughts:

1. [On charts] It would be nice if we could have one set of charts between metaforecast and squiggle, so we don't re-implement similar functionality on the team. With Squiggle we currently use Vega and it's working decently, but there's definitely more improvement to be done. https://develop--squiggle-components.netlify.app/
2. Unifying forecast formats seems pretty good to me, but hard to do super well. I'd like it if people could treat that Metaforecast API a bit like some of this.
3. Supporting login + users kind of scares me (More maintenance, GDPR issues). I suggest having this be one of the later ones.
4. Your prioritization list sounds pretty good to me.

[akrolsmir]

At the point where you'd want to roll your own auth and start taking comments, I'd suggest also thinking about deeper integrations with Manifold~ (e.g. we'd be happy to provide auth as a service)

Already, we view ourselves as kind of a meta-platform for prediction markets (aka, one frame is that "On Manifold, anyone can run their own PredictIt/Kalshi"). We already mirror some Metaculus questions manually, and plan on doing so for even more - perhaps programmatically, it's one thing we're chatting with Daniel at Metaculus about.

I know Metaforecast integrates predictions across different forecasting platforms too (eg non-markets like Metaculus) but I don't think that's a problem; in my ideal world Manifold would actually just have a Metaforecast-like-interface where you can see all the predictions across all the sites, and bet on any that you want using the Manifold market maker. I was already thinking of doing like a 1-click integration for any Metaforecast listing, but now it occurs to me that maybe we should just be a single product.

Very ambitious, I know, and maybe not in line with QURI's mission - no worries if so. But would love to start the conversation on this!

[OAGr]

(Sorry for the late reply!)
Thanks for the offer! I think it's pretty unlikely that we'd want to go with this; in general I'm very reluctant to use another web platform for our primary auth. It would raise a bunch of dependency issues, and if anything happened to Manifold (you get acquired, or make a big pivot), I assume it could be a big pain for us. Will keep it in mind though.