Bridge request for Indiegogo.com

[nikiie8]

Bridge request

General information

• Host URI for the bridge (i.e. https://github.com): https://www.indiegogo.com

• Which information would you like to see?

I would like to see the feed of new projects from the following page
https://www.indiegogo.com/explore/all?project_type=campaign&project_timing=just_launched&sort=trending

• How should the information be displayed/formatted?

Title, short description and picture

• Which of the following parameters do you expect?

  • Title
  • URI (link to the original article)
  • Author
  • [ x] Timestamp
  • Content (the content of the article)
  • Enclosures (pictures, videos, etc...)
  • [ x] Categories (categories, tags, etc...)

It would be nice to have the opportunity to choose categories

Options

• Limit number of returned items
  • Default limit: 5
• Load full articles
  • Cache articles (articles are stored in a local cache on first request): yes
  • Cache timeout (max = 24 hours): 24 hours
• Balance requests (RSS-Bridge uses cached versions to reduce bandwith usage)
  • Timeout (default = 5 minutes, max = 24 hours): 5 minutes

Thank you in advance

[Bockiii]

I've worked on this a little yesterday and ran into an issue. Maybe @em92 or another skilled bridge maker can help:

The site is build dynamically with javascript. So if you use the simplehtmldom, you dont get the html content, just a bunch of js information. I researched a bit and it seems like, in order to get the html, the tool would have to actually render the content. Other people/applications use headless browsers in order to do this and there are actually services online that provide this feature. ( like https://proxybot.io/ or https://proxiesapi.com/ ).

So these are the options at this point (from my pov):

• Bridge not possible
• We ship a headless browser with the rss-bridge container that we can use. It looks like https://github.com/symfony/panther could be a promising solution, but needs testing.
• We make it configurable and a mandatory dependency: If you want to use this bridge, you need a proxy that can provide the html.

I would be in favor of option 2. Adding panther doesnt look like a huge thing, the tool itself shouldn't blow up the installation by a lot and if we implement that, other bridges become possible as well.

I would see the implementation of panther as a separate topic though.

[dotter-ak]

@Bockiii,
After some investigation, I've found that you don't need browser to get data from this site. You can get json from this url: https://www.indiegogo.com/private_api/discover

This code returns array with last 12 items:

$url = 'https://www.indiegogo.com/private_api/discover';
$data_array =  array(
	'sort'	=> 'trending',
	'category_main'	=> NULL,
	'category_top_level'	=> NULL,
	'project_timing'	=> 'just_launched',
	'project_type'	=> 'campaign',
	'page_num'	=> 1,
	'per_page'	=> 12,
	'q'	=> '',
	'tags'	=> array()
);
	$curl = curl_init();
	curl_setopt($curl, CURLOPT_POSTFIELDS, json_encode($data_array));
	curl_setopt($curl, CURLOPT_URL, $url);
	curl_setopt($curl, CURLOPT_HTTPHEADER, array(
		'Content-Type: application/json',
	));
	$result = curl_exec($curl);
	curl_close($curl);
$response = json_decode($result, true);
var_dump($response);

[Bockiii]

I actually found that in my browser but was too stupid to build the request headers :D Thanks for your working version, I will see what I can do.