Last edited by D, 2026-10-15, during class; This article is still in construction!
What better place to start scraping staff comments than... Popper's Lounge!
You might be asking, "Wait, but those aren't staff comments? And there already are multiple backups made of it?" Well, this is actually the first time I've ever scraped things off the internet as part of a project. Since the layouts of Popper's Lounge pages are pretty simple and straightforward, I've decided to use them as a training ground for getting a hold of the tools and libraries to use, the problems I might encounter down the road, how to process and structure the data collected, and so on.
The Popper's Lounge main page features an introduction written by Mary followed by a list of links to the 34 individual pages for each Popper's Lounge interview.
The list itself is represented in HTML via a <table>. Unlike some of the other... strangely designed layouts to be encountered later on throughout the project, it is structured minimally, perhaps due to the fact that hardly any styling is applied to it. (Maybe this is just how they made websites back in the late 90's.)
<tbody>
<tr>
<td width="50%"><a href="guest1.html">第１回『マリィ、ジュディ、アヤ』</a></td>
<td width="50%"><a href="guest2.html">第２回『ショルキー』</a></td>
</tr>
<tr>
<td width="50%"><a href="guest3.html">第３回『めぐみちゃん』</a></td>
<td width="50%"><a href="guest4.html">第４回『アッシュ』</a></td>
</tr>
...
Each <td> element conveniently contains an <a> with the link to its corresponding subpage.
As one could imagine, manually clicking on the 34 links and copy-pasting its contents by hand would be extremely tedious; so I took the approach of 1) Extracting the list of links to the 34 subpages, and then 2) Writing a script to automatically go through that list and save whatever is returned from the links.
What better way to represent a list of links that need to be sequentially accessed, than an array?
Converting the raw <table> to an array is pretty straightforward. One would just have to write a script that reads through the file; whenever the string a href=" is encountered, whatever immediately follows it up until the next " is appended to the array. The array is then written to a JSON file for transportability.
My language of choice for the script was Python: