Song data like million song dataset code#
Song data like million song dataset download#

The lyrics dataset consisted of lyrics as a bag of words (the original lyric files are copyright protected) Further cleaning had to be done by removing incomplete records and records with too many empty features. Only the dataset files in the HDF5 files were read and the group and metadata files were ignored. To load the Million Song Dataset, the HDF5 files in the directory structure had to be read into a Spark dataframe. The size of all the datasets is 300GB, too large for conventional processing.

Echonest User Datset : song play history for over 1 million users.

TU Wien Genre Dataset : categorization of the above dataset into 21 different genres.

MusiXmatch Lyrics Dataset : lyrics (where applicable) for the above available as an indexed data.

Song data like million song dataset download#

The dataset is stored in a hierarchal HDF5 format and is available to download from the Open Science Data Cloud and AWS.ДЄdditionally, the following 3 auxiliary datasets were used:. The dataset consists of over 50 features and metadata extracted from the song audio. The main dataset for the project is the Million Song Dataset (MSD), a freely-available collection of audio features and metadata for a million contemporary popular music tracks. analyasis.py : parse output files and select fields to plot word clouds into a single file.combine.py : parse output files and select fields to plot locaitons.parselyrics.py : convert the lyrics to dictionary format.concat.py : parse output files to single file.

Song data like million song dataset code#

clustering.scala : main clustering algorithm and recommender system code.

optimum.scala : find the optimal number of clusters and plot the elbow plot.

process.scala : code to process and join all datasets and save the necessary fields.

makecsv.py : script to cache the dataset into a csv file.The main aim of the project is not to provide precise and groundbreaking results, but to use the concepts of Big Data Analytics on large data that can not be analyzed and processed using conventional methods.ДЇolder consists of the following code files:. This project on the publicly available Million Song Dataset aims to address three separate questions - recommending songs to a user based on his play history, visualizing trends in music across the years and finally predicting the genre of an unknown song based on its lyrics.