Experimental Evaluation of Related Papers Finding Techniques

Abstract

Introduction/Importance of Study: Related-paper recommendation systems generally contain two categories: Content-Based (CB) approaches, which estimate relatedness using semantic similarity between research paper texts, and metadata-based approaches, which infer relatedness from bibliographic information such as citations, references, authorship, and publication venue. Although CB methods, such as Jensen–Shannon Divergence (JSD), computed over TF–IDF representations provide accurate relatedness scores, they are computationally expensive because they require processing of the full text of each paper. Metadata-based methods offer a more efficient substitute, but their relative effectiveness to strong CB measures remains unclear. This study investigates which bibliometric technique correlates most strongly with JSD-based semantic relatedness to identify a low-cost substitute for computationally expensive CB methods.

Novelty Statement: Since no existing dataset contained the required combination of full text, citations, references, and “related papers” lists, we constructed a new dataset of 1,225 papers, selected to statistically represent the population for a target keyword at 95% confidence with ±2.8% margin of error. No prior research has analyzed bibliometric methods using a unified dataset.

Material and Method: JSD-based relatedness scores were computed using full-text TF–IDF representations for all papers. We then calculated bibliographic relatedness using bibliographic coupling (BC), co-citation coupling (CC), and Katz similarity, and also extracted relatedness scores from Semantic Scholar (SS).

Result and Discussion: Correlation analysis revealed the following Pearson correlations with JSD: BC = 0.40, SS = 0.35, Katz = 0.01, CC = –0.11. These results indicate that BC-based relatedness aligns most closely with CB semantic similarity, followed by SS, while Katz and CC show negligible or negative correlation. Notably, the finding that Semantic Scholar’s related-paper measure correlates less strongly with JSD than bibliographic coupling is both surprising and practically important. 

Concluding Remarks: Overall, the results highlight the potential of BC-based methods as an efficient and reliable alternative to traditional full-text similarity computations for estimating relatedness.