Skip to content
Sadie

Feed

Feeds are the live signal Sadie monitors on your behalf. Each feed is a tracked source of current content: an x_account, an rss feed, or a newsletter. Feeds have a handle or URL, a label you choose, and a status (active, paused, error, or connecting).

Feeds are where the “what’s happening” half of a brief or a discourse opportunity comes from. Your wiki supplies the “what you know” half.

Items, deduplicated across users

Section titled “Items, deduplicated across users”

The same X post or newsletter issue can show up in more than one user’s follow list. Sadie keeps one canonical copy per piece of content, keyed by source and external ID, and links it to each user’s feed separately. You see your feed. Behind the scenes the content lives once. This keeps storage honest and makes theme counts accurate. If a later refresh discovers a better canonical body, existing per-user rows repair from it; longer bodies win by default, and shorter replacements are allowed only for clear boilerplate cleanup.

RSS and newsletter items prefer full feed content when publishers provide it, then try article extraction for new canonical items with URLs. Feed-provided HTML is decoded to clean text before storage, including entity decoding and removal of script/style residue. Article extraction is quality-aware: a shorter clean article can replace a longer feed body only when the feed text is mostly subscription/footer/navigation boilerplate, so Today and briefs reason over the actual body instead of markup or page furniture.

Known RSS/newsletter rows are not scraped forever. Healthy canonical bodies are reused, while a capped, priority-ranked repair set of URL-bearing empty, short, teaser-like, or boilerplate-heavy known items can be rehydrated during ordinary refresh. That lets old weak rows improve without letting arbitrary database order or no-URL rows spend the article-fetch budget, and without turning every refresh into a full article crawl.

Dedup and theme clustering happen in the compile step

Section titled “Dedup and theme clustering happen in the compile step”

When the compile job runs, it does three things to raw items:

  1. Deduplicates near-identical posts or stories
  2. Extracts detected themes from each item
  3. Rolls theme signals up into feedThemeCounts per user, with item count, distinct days, and distinct authors

When a theme accumulates enough signal across those three dimensions, Sadie can automatically promote it into a new wiki entry. The promotion threshold is tunable in Settings. Once promoted, a theme carries a promotedToWikiId marker so Sadie does not duplicate the entry on the next pass.

Read-only by design

Section titled “Read-only by design”

Sadie’s X adapter is strictly read-only. The three operations it exposes are recentSearch, filteredStream, and postCounts. There is no posting path. There will not be one.

This is a product decision, not a gap. The point of a discourse opportunity is to surface a moment you are positioned for. The act of entering the conversation is yours. Sadie shows you the room. You decide if and how to speak.

Feed health

Section titled “Feed health”

Feeds are managed on the Feeds surface. Paste a handle, feed URL, or newsletter address. Sadie validates, fetches the first pull, and moves the feed to active.

Sadie distinguishes transient failures from broken sources. Rate limits, 5xx responses, X network fetch failures, and slow RSS/newsletter hosts stay active with a retry window; client-side failures like a deleted feed or permanently bad URL move the feed to error and wait for your attention rather than silently failing.

When you save a feed item into Sources, Sadie freezes the article/post body as a library source, keeps a pointer back to the origin feed item, and records the save as reference-material learning in Soul. Memory can then show both the origin feed’s health and the fact that you chose this live item as evidence, without rewriting the saved artifact when the feed refreshes.

See also

Section titled “See also”