feat(connector-sdk): extensionDomScrape helper; use it in LinkedIn home_feed#1155
Conversation
…me_feed The SDK had extensionNetworkSync for the passive network-capture path but the content-script DOM path (cs_scrape) was hand-wired inline in the LinkedIn connector. Add extensionDomScrape as the DOM-path companion so the home feed goes through the SDK like the network feeds. The helper stays generic (names no site); the selectors and allowed origins are supplied by the caller. No behavior change: same single navigate+cs_scrape dispatch, just wrapped.
📝 WalkthroughWalkthroughThis PR introduces ChangesextensionDomScrape Helper and LinkedIn Integration
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
bug_free 87, simplicity 82, slop 12, bugs 0, 0 blockers Deterministic typecheck/unit/integration logs all exited 0; narrow rerun of extension-dom-scrape + linkedin tests passed (10/10). Skipped server boot because diff is connector SDK/connectors only. Main unverified path is a live Owletto/LinkedIn browser scrape. Suggested fixes
Full verdict JSON{
"bug_free_confidence": 87,
"bugs": 0,
"slop": 12,
"simplicity": 82,
"blockers": [],
"change_type": "feat",
"behavior_change_risk": "low",
"tests_adequate": true,
"suggested_fixes": [
{
"file": "packages/connectors/src/__tests__/linkedin.test.ts",
"line": 17,
"change": "Replace the inline extensionDomScrape reimplementation with the real helper or a shared test mock helper so this test does not duplicate production logic."
},
{
"file": "packages/connectors/src/__tests__/browser-scraper-utils.test.ts",
"line": 26,
"change": "Reuse the same extensionDomScrape test mock/helper used by linkedin.test.ts instead of copying the full implementation here."
}
],
"notes": "Deterministic typecheck/unit/integration logs all exited 0; narrow rerun of extension-dom-scrape + linkedin tests passed (10/10). Skipped server boot because diff is connector SDK/connectors only. Main unverified path is a live Owletto/LinkedIn browser scrape.",
"categories": {
"src": 127,
"tests": 180,
"docs": 0,
"config": 0,
"deps": 0,
"migrations": 0,
"ci": 0,
"generated": 0
}
}Local review gate — branch protection can require the |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/connector-sdk/src/extension-dom-scrape.ts (1)
36-42: 💤 Low valuePrefer
interfacefor the observation shape.
ExtensionScrapeObservationdefines an object shape as atypealias. An interface with an index signature is equivalent here and keeps it consistent with the other declarations in this file:♻️ Proposed refactor
-/** Dispatcher observation envelope; the index signature satisfies `ChromeActionOutput`. */ -export type ExtensionScrapeObservation = Record<string, unknown> & { - tab_id?: number; - cs_scrape?: boolean; - persistent_reused?: boolean; - result?: ExtensionScrapeResult; -}; +/** Dispatcher observation envelope; the index signature satisfies `ChromeActionOutput`. */ +export interface ExtensionScrapeObservation { + tab_id?: number; + cs_scrape?: boolean; + persistent_reused?: boolean; + result?: ExtensionScrapeResult; + [k: string]: unknown; +}As per coding guidelines: "Use
interfacefor defining object shapes in TypeScript files".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/connector-sdk/src/extension-dom-scrape.ts` around lines 36 - 42, Replace the type alias ExtensionScrapeObservation with an equivalent interface to match the file's conventions: declare interface ExtensionScrapeObservation that includes the index signature (to satisfy ChromeActionOutput) plus the optional properties tab_id, cs_scrape, persistent_reused, and result (referencing ExtensionScrapeResult); keep the shape identical but use the interface keyword so it aligns with other declarations in this module.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@packages/connector-sdk/src/extension-dom-scrape.ts`:
- Around line 36-42: Replace the type alias ExtensionScrapeObservation with an
equivalent interface to match the file's conventions: declare interface
ExtensionScrapeObservation that includes the index signature (to satisfy
ChromeActionOutput) plus the optional properties tab_id, cs_scrape,
persistent_reused, and result (referencing ExtensionScrapeResult); keep the
shape identical but use the interface keyword so it aligns with other
declarations in this module.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 43e55991-a293-471a-bdb0-96aed9d897d1
📒 Files selected for processing (7)
packages/connector-sdk/src/__tests__/extension-dom-scrape.test.tspackages/connector-sdk/src/extension-dom-scrape.tspackages/connector-sdk/src/index.tspackages/connectors/src/__tests__/browser-scraper-utils.test.tspackages/connectors/src/__tests__/connector-sdk.mock.tspackages/connectors/src/__tests__/linkedin.test.tspackages/connectors/src/linkedin.ts
2 participants
What
Adds the missing SDK helper for the content-script DOM path so the LinkedIn home feed goes through the SDK like the network feeds, instead of a raw inline dispatch.
The connector SDK already had
extensionNetworkSyncfor passive network capture (company updates, jobs, Revolut, X). But the LinkedIn home feed can't use network capture — attaching the CDP debugger that the Network-domain intercept needs stops the personalized feed from rendering, so the Voyager XHRs never fire. Instead it reads the live DOM via the extension's content-script opcs_scrape. That path was hand-wired inline inlinkedin.ts(dispatcher.dispatch('navigate', { cs_scrape: true, ... })+ readingobservation.result.{rows,loggedIn}), leaving the SDK asymmetric: a helper for the network path, a raw dispatch for the DOM path.This PR closes that gap.
Changes
packages/connector-sdk/src/extension-dom-scrape.ts(new):extensionDomScrape<TItem>({ dispatcher, url, config, parseRows, allowedOrigins, persistent?, focus? }). Does the singlenavigate+cs_scrapedispatch (defaultingpersistent/focusto true), readsobservation.result, returns{ items, loggedIn, count, host?, landedUrl? }(loggedInistrueunless the extension reports an explicitfalse). Reuses the existingChromeActionDispatchertype fromextension-network.ts— no second definition. ExportsextensionDomScrape+ExtensionScrapeConfig/ExtensionScrapeResult/ExtensionScrapeObservation/ExtensionDomScrapeResultfrom the package index.packages/connectors/src/linkedin.ts:syncHomeFeednow callsextensionDomScrapeinstead of the inline dispatch. Deletes the now-dead localCsScrapeObservation/HomeFeedScrapeResulttypes (keepsHomeFeedRow,HOME_FEED_SCRAPE_CONFIG,buildHomeFeedEvents).company_updates/jobsstay onextensionNetworkSync.The SDK helper stays generic and names no site — the selectors (
HOME_FEED_SCRAPE_CONFIG) and allowed origins stay in the connector, where they belong (CSP/site-specific constraint).No behavior change
Same single
navigate+cs_scrapedispatch with the samescrape_config/allowed_origins, same logged-out error message — just wrapped behind the SDK.Tests / typecheck
bun test packages/connectors/src/__tests__/ packages/connector-sdk/src/__tests__/extension-dom-scrape.test.ts packages/connector-sdk/src/__tests__/extension-network.test.ts→ 38 pass / 0 fail. Existing LinkedIn home-feed tests still pass; addedextension-dom-scrape.test.ts(stub dispatcher → asserts dispatch shape, row parsing, andloggedIntrue/false/absent handling).bunx tsc --noEmitonpackages/connector-sdk→ clean. Onpackages/connectors, no new errors in touched files (linkedin.ts clean); remaining errors are the known pre-existing baseline in other connectors +.ts-import-extension config noise.Need help on this PR? Tag
@codesmithwith what you need. Autofix is disabled.Summary by CodeRabbit
New Features
Improvements
Tests