Effectiveness of RSS feed item duplication detection using word matching

Tan, Ian K. T. and Su, Tze-Wei and Khor, Hao-Ming and Ong, Ee-Chun (2011) Effectiveness of RSS feed item duplication detection using word matching. Sunway Academic Journal, 8. pp. 38-53. ISSN 1823-500X

SAJ_8_2011_38-53.pdf - Published Version
Available under License Creative Commons Attribution No Derivatives.

Download (424kB) | Preview


Users of feed aggregators know that duplicated articles are found occasionally on the feeds they subscribe to. It can be time consuming to read all articles and stumble upon duplicated items they have already read. Our work here is to determine the effectiveness of using basic word matching to remove duplicated items and only show the most relevant item, thus saving readers’ time. The method described in this paper to remove duplicates involves word matching heuristics with an appropriate matching percentage. The duplicated feeds are then ranked to only display the highest ranked article. Ranking is done using the number of search items found on the titles of the news feeds where the highest number returned will be considered the highest ranked article. Using Malaysian online news feeds, our method found that with a matching percentage of 40%, our method will be able to minimize duplicates effectively with minimal errors. We did further empirical studies using 9 technology blog feeds over a longer period to provide us with a better averaging results. The matching percentage obtained is also within the same quantum. The method described here has a low overhead in terms of processing for the duplicates and with careful selection of matching percentage, the system will effectively remove the majority of duplicates

Item Type: Article
Additional Information: Authors are with the Faculty of Information Technology, Multimedia University
Uncontrolled Keywords: duplicate removal; word matching; RSS feed; news; ranking
Subjects: Q Science > QA Mathematics > QA76 Computer software
Divisions: Others > Non Sunway Academics
Depositing User: Ms. Molly Chuah
Date Deposited: 29 Sep 2016 07:23
Last Modified: 03 Oct 2016 10:28
URI: http://eprints.sunway.edu.my/id/eprint/387

Actions (login required)

View Item View Item