The Moksha Feed Stream

The mdemos.feeds app provides a moksha Producer_ that will handle automatically fetching, parsing, and caching feeds. Once installed, the Moksha Feed Stream will be run by the MokshaHub_.

It will automatically fetch all feeds used by the Moksha Feed object, as well as all feeds listed in a feeds.txt file, if one exists.

Installing the mdemos.feeds app

Currently the easiest way to install the app is from the git repository

$ git clone git://github.com/ralphbean/mdemos.feeds
$ cd mdemos.feeds

From here, if you are running Moksha from an RPM installation:

$ paver reinstall

Or, if you’re running from a virtualenv installation:

$ paver install

Configuring the Moksha Feed Aggregator

The feed app can be configured with the following options in your config file:

# Enable the feed aggregator in the moksha-hub
moksha.feedaggregator = True

# Number of seconds between polling feeds
feed.poll_frequency = 900

# Where to store the feed caches. Defaults to an in-memory cache.
#feed_cache = sqlite:///%(here)s/feeds.db

# Max age (in seconds) of each feed in the cache
feed.max_age = 300

# Timeout in seconds for the web request
feed.timeout = 60

# The number of simultaneous connections
feed.deferred_groups = 3

Performing post-processing on feed entries

All new feed entries for a given $URL are sent to the moksha.feeds.$URL message topic. By default only the title and url of the entry are sent to the message broker. However, Moksha provides a simple API for writing post-processing plug-ins that can modify the feed before it is sent to any LiveWidgets or Consumers.

def process_feed_entry(entry):
 """ This method is called by the Moksha Feed Streamer with each feed entry.

 Here is where we do post-processing on the entry before it gets serialized
 to JSON and send to our message broker, and then the users.

 :entry: A :mod:`feedparser` feed object
 """
 return dict(author=entry['author'],
             author_link=entry['author_detail']['href'],
             content=entry['content'][0]['value'],
             author_avatar=entry['source']['icon'])

Then you simply plug this method into the moksha.feeds.post_processor entry-point:

setup(...
      entry_points="""
        [moksha.feeds.post_processor]
        myfeedprocessor = myapp.feed_processor:process_feed_entry
      """
)