Multi-core Sentiment Engine

This rule-based text analysis algorithm is able to utilize multiple cores of a processor and is designed to run on a server as an engine to process hundreds of text files. The algorithm takes URLs and id as input and calculates the below scores and metrics

calc_values = {
      "POSITIVE SCORE": positive_score,
      "NEGATIVE SCORE": negative_score,
      "POLARITY SCORE": polarity_score,
      "SUBJECTIVITY SCORE": subjectivity_score,
      "AVG SENTENCE LENGTH": avg_sentencelen,
      "PERCENTAGE OF COMPLEX WORDS": percent_complexwords,
      "FOG INDEX": fog_index,
      "AVG NUMBER OF WORDS PER SENTENCE": avg_wordper_sentence,
      "COMPLEX WORD COUNT": complexword_count,
      "WORD COUNT": cleanword_count,
      "SYLLABLE PER WORD": syllablecount_perword,
      "PERSONAL PRONOUNS": pronouns_count,
      "AVG WORD LENGTH": avg_wordlength
    }

from the article text and updates the values to the output destination, data from here can be used for other applications. The data extraction and NLP problem is tasked with solving ways to design and implement a sentiment analysis algorithm, which analyzes textual data from the web and assesses its sentiment, positive, negative, or neutral, along with determining more variables such as subjectivity score and readability.

The problem is solved using an extract, transform, load approach with multiprocessing for parallel processing of text files. The program uses the multiprocessing library to create multiple processes equal to the number of logical processors(cores) found in the executing system. The number of files to process is split among the processes almost equally. The processes execute the runengine() function in parallel, processing separate file lists divided for each process.

• Significant Performance Gains: Parallel processing effectively leverages multiple CPU cores, distributing workload and substantially reducing processing time, especially for large datasets.
• Shared Data Handling: The use of a managed list to store output dataframes by each process ensures efficient communication and data collection between parallel processes.
• Error Handling: Error handling is essential to prevent potential issues during multiprocessing, ensuring the integrity of the final output.

Implementation is available at github.