How Data Collection Works



Data collection is comprised of two separate, but distinct parts.




Part 1 -

The first part of data collection involves pulling the stock closing prices from an API called Stocklytics. Essentially how this works
is that we created an account with http://stockalytics.com. From here we are given a RESTful URL, the parameters that we pass into the URL are
the dates that we want quotes from, as well as the Stock Ticker for which we want the quotes. Once we invoke this URL we receive a JSON file 
with the Stock information. Once we have this we are now in possession of accurate and up to date stock opening prices, closing prices, and even volume. 
Even though all we need is the closing price, the other information can be useful in the future if we ever wish to implement any sort of graphs.

  


Part 2 - 

The second part of data collection requires us pulling information from news articles from the New York Times website. 
The New York Times offers a RESTful link with which developers can pass certain parameters and receive certain information from news articles. 
Collecting this data is key in regards to the Data Mining aspect of our algorithm. By passing the name of a stock ticker you can get articles in 
which that stock ticker appears. How the data collection algorithm here works is that it pulls at random any 500 articles from the New York Times 
database with the stock ticker name in order of newest first. It then searches these articles to see how many times the stock ticker name appears. 
Then once all of the instances of the Stock Ticker name are counted it is divided by 100. And this is the point value that is added onto the the 
predicted value that is obtained from the neural network.  