Google+
4elements web design & consultancy

Shineyrock

web design & consultancy
Shineyrock web design & consultancy

Shineyrock

  1. web design en consultancy

blog

  1. Home
  2. blog
  3. lees
  4. Using the New York Times API to Scrape Metadata
  • dislike -1 06

    Using the New York Times API to Scrape Metadata

    1. February 2017
    2. Daily
    3. gelezen 804
    4. reactie(s) 1

    Final product image
    What You'll Be Creating

    Introduction

    Last week, I wrote an introduction to scraping web pages to collect metadata, mentioning that it's not possible to scrape the New York Times site. The Times paywall blocks your attempts to gather basic metadata. But there is a way around this using the New York Times API.

    Recently I began building a community site on top of the Yii platform, which I wrote about in Programming With Yii2: Building Community with Comments, Sharing and Voting (Envato Tuts+). I wanted to make it easy to add links related to content on the site. While it's easy for people to paste URLs into forms, it becomes time-consuming to also provide title and source information.

    So in today's tutorial, I'm going to expand the scraping code I wrote recently to leverage the New York Times API to gather headlines when Times links are added.

    Remember, I participate in the comment threads below, so tell me what you think! You can also reach me on Twitter @lookahead_io.

    Getting Started

    Sign Up for an API Key

    New York Times API - API Gallery Home Page

    First, let's sign up to request an API Key:

    New York Times API - API Sign Up Page

    After you submit the form, you'll receive your key in an email:

    New York Times API - Email with API Key

    Exploring the New York Times API

    New York Times API - Categories

    The Times offers APIs in the following categories:

    • Archive
    • Article Search
    • Books
    • Community
    • Geographic
    • Most Popular
    • Movie Reviews
    • Semantic
    • Times Newswire
    • TimesTags
    • Top Stories

    It's a lot. And, from the Gallery page, you can click on any topic to see the individual API category documentation:

    New York Times API - Documentation of articlesearch json

    The Times uses LucyBot to power their API docs, and there is a helpful FAQ:

    New York Times API - FAQ

    They even show you how to quickly get your API usage limits (you'll need to plug in your key):

     curl --head
   https://api.nytimes.com/svc/books/v3/lists/overview.json?api-key=<your-api-key>
    2>/dev/null | grep -i "X-RateLimit"
    X-RateLimit-Limit-day: 1000
    X-RateLimit-Limit-second: 5
    X-RateLimit-Remaining-day: 180
    X-RateLimit-Remaining-second: 5

    I initially struggled to make sense of the documentation—it's a parameter-based specification, not a programming guide. However, I posted some questions as issues to the New York Times API GitHub page, and they were quickly and helpfully answered.

    Working With Article Search

    For today's episode, I'm going to focus on using the NY Times Article Search. Basically, we'll extend the Create Link form from the last tutorial:

    New York Times API - Create Link Form with NYT Story URL about Polar Bears

    When the user clicks Lookup, we'll make an ajax request through to Link::grab($url). Here's the jQuery:

    $(document).on("click", '[id=lookup]', function(event) {
  $.ajax({
     url: $('#url_prefix').val()+'/link/grab',
     data: {url:   $('#url').val()},
     success: function(data) {
       $('#title').val(data);
       return true;
     }
  });
});

    Here's the controller and model method:

    // Controller call via AJAX Lookup request
public static function actionGrab($url) {
  Yii::$app->response->format = Response::FORMAT_JSON;
  return Link::grab($url);
}
...
// Link::grab() method
public static function grab($url) {
  //clean up url for hostname
  $source_url = parse_url($url);
  $source_url = $source_url['host'];
  $source_url=str_ireplace('www.','',$source_url);
  $source_url = trim($source_url,' \\');
  // use the NYT API when hostname == nytimes.com
  if ($source_url=='nytimes.com') {
   ...

    Next, let's use our API key to make an article search request:

        $nytKey=Yii::$app->params['nytapi'];
    $curl_dest = 'http://api.nytimes.com
        /svc/search/v2/articlesearch.json?fl=headline&fq=web_url:%22'.
        $url.'%22&api-key='.$nytKey;
    $curl = curl_init();
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_URL,$curl_dest);
    $result = json_decode(curl_exec($curl));
    $title = $result->response->docs[0]->headline->main;
  } else {
    // not NYT, use the standard metatag scraper from last episode
         ...
    }
  }
  return $title;
}

    And it works quite easily—here's the resulting headline (by the way, climate change is killing Polar Bears and we should care):

    New York Times API - Create Link Form with NYT Story URL and Headline from Article Search API

    If you want more details from your API request, just add additional arguments to the ?fl=headline request such as keywords and lead_paragraph:

    Yii::$app->response->format = Response::FORMAT_JSON;
$nytKey=Yii::$app->params['nytapi'];
$curl_dest = 'http://api.nytimes.com/svc/search/v2/articlesearch.json?'.
  'fl=headline,keywords,lead_paragraph&fq=web_url:%22'.$url.'%22&api-key='.$nytKey;
$curl = curl_init();
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_URL,$curl_dest);
$result = json_decode(curl_exec($curl));
var_dump($result);

    Here's the result:

    The response from the API request

    Perhaps I'll write a PHP library to better parse the NYT API in coming episodes, but this code breaks out the keywords and the lead paragraph:

    Yii::$app->response->format = Response::FORMAT_JSON;
$nytKey=Yii::$app->params['nytapi'];
$curl_dest = 'http://api.nytimes.com/svc/search/v2/articlesearch.json?'.
  'fl=headline,keywords,lead_paragraph&fq=web_url:%22'.$url.'%22&api-key='.$nytKey;
$curl = curl_init();
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_URL,$curl_dest);
$result = json_decode(curl_exec($curl));
echo $result->response->docs[0]->headline->main.'<br />'.'<br />';
echo $result->response->docs[0]->lead_paragraph.'<br />'.'<br />';
foreach ($result->response->docs[0]->keywords as $k) {
  echo $k->value.'<br/>';
}

    Here's what it shows for this article:

    Polar Bears’ Path to Decline Runs Through Alaskan Village

The bears that come here are climate refugees, on land because
the sea ice they rely on for hunting seals is receding.

Polar Bears
Greenhouse Gas Emissions
Alaska
Global Warming
Endangered and Extinct Species
International Union for Conservation of Nature
National Snow and Ice Data Center
Polar Bears International
United States Geological Survey

    Hopefully that starts to expand your imagination about how to use these APIs. It's pretty exciting what may now be possible.

    In Closing

    The New York Times API is very useful, and I'm glad to see them offering it to the developer community. It was also refreshing to get such quick API support via GitHub—I just didn't expect this. Keep in mind that it's intended for non-commercial projects. If you have some money-making idea, send them a note to see if they'll work with you. Publishers are eager for new sources of revenue.

    I hope you found these web scraping episodes helpful and put them to use in your projects. If you'd like to see today's episode in action, you can try out some of the web scraping on my site, Active Together.

    Please do share any thoughts and feedback in the comments. You can also always reach me on Twitter @lookahead_io directly. And be sure to check out my instructor page and other series, Building Your Startup With PHP and Programming With Yii2.

    Related Links

    Photo

    martijn broeders

    founder/ strategic creative bij shineyrock web design & consultancy
    e-mail: .(JavaScript must be enabled to view this email address)
    telefoon: 434 210 0245
    Plaats reactie...
    4elements
    Comments
  • 1
    silhouette
    lilian
    11 September, 2018 om

    Excellent blog heгe! Also your web site loads up fast!What web host are yоu using? Can I get yоur affiliate link to your host? I wish my website loaded up as fast as yours lol

    4elements
    Comments

Per - categorie

    Op - datum

      Copyright © 1999 - 2025 Shineyrock web design en consultancy, Alle rechten voorbehouden / Algemene voorwaarden / Sitemap: XML, HTML, Images, Mobile, News