Page MenuHomePhabricator
Create Task
Maniphest T45

Backfill Card Data - Web Scraper
Open, NormalPublic
Actions

Description

Maybe a bot or something to check what's missing and go grab that data

Originally created 2019-01-31

Event Timeline

ceec created this task.Mar 29 2019, 10:53 PM
ceec triaged this task as Normal priority.
ceec created this object with visibility "Public (No Login Required)".
ceec added a project: Data.
ceec renamed this task from Backfill Card Data to Backfill Card Data - Web Scraper.Apr 15 2019, 11:31 PM

have a scraper mostly built, need to figure out a way to just rip the image names out of the raw JS

maybe just a generic php file_get_contents and then string search for the variable list

ceec moved this task from To Do to Doing on the enstars.info board.Apr 15 2019, 11:32 PM
ceec added a comment.May 27 2019, 9:54 PM

made a LOT of progress. it's in the final stages right now. It's creating the scout and then creating the cards in the cardsuggestion table

Issues still:

  • stars will have to be manually added
  • to save the images to the server, looks like laravel has a class

https://laracasts.com/discuss/channels/laravel/storing-image-file-retrieved-from-external-url?page=1

ceec moved this task from Doing to Half Done on the enstars.info board.Nov 26 2019, 12:15 AM
ceec added a comment.Apr 1 2020, 1:14 AM

Okay new websites now, with less information! Also LOTS of data i dont have from old game.

Current scraper wont have to create new scouts/events. Just backfill data.

ceec added a comment.May 14 2020, 12:25 AM

Break this into two, old scraper to grab old content

New scraper to just grab basic info?