mirror of
https://gitlab.com/news-flash/article_scraper.git
synced 2025-07-07 16:15:32 +02:00
add new test
This commit is contained in:
Jan Lukas Gernert
parent
f7fa696921
commit
0901a37475
4 changed files with 1129 additions and 325 deletions
325
expected.html
325
expected.html
|
|
@ -1,325 +0,0 @@
|
|||
<article><DIV id="readability-page-1"><article role="article"><p>For more than a decade the Web has used XMLHttpRequest (XHR) to achieve
|
||||
asynchronous requests in JavaScript. While very useful, XHR is not a very
|
||||
nice API. It suffers from lack of separation of concerns. The input, output
|
||||
and state are all managed by interacting with one object, and state is
|
||||
tracked using events. Also, the event-based model doesn’t play well with
|
||||
JavaScript’s recent focus on Promise- and generator-based asynchronous
|
||||
programming.</p>
|
||||
<p>The <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API" target="_blank">Fetch API</a> intends
|
||||
to fix most of these problems. It does this by introducing the same primitives
|
||||
to JS that are used in the HTTP protocol. In addition, it introduces a
|
||||
utility function <code>fetch()</code> that succinctly captures the intention
|
||||
of retrieving a resource from the network.</p>
|
||||
<p>The <a href="https://fetch.spec.whatwg.org/" target="_blank">Fetch specification</a>, which
|
||||
defines the API, nails down the semantics of a user agent fetching a resource.
|
||||
This, combined with ServiceWorkers, is an attempt to:</p>
|
||||
<ol>
|
||||
<li>Improve the offline experience.</li>
|
||||
<li>Expose the building blocks of the Web to the platform as part of the
|
||||
<a href="https://extensiblewebmanifesto.org/" target="_blank">extensible web movement</a>.</li>
|
||||
</ol>
|
||||
<p>As of this writing, the Fetch API is available in Firefox 39 (currently
|
||||
Nightly) and Chrome 42 (currently dev). Github has a <a href="https://github.com/github/fetch" target="_blank">Fetch polyfill</a>.</p>
|
||||
<h2>Feature detection</h2>
|
||||
<p>Fetch API support can be detected by checking for <code>Headers</code>,<code>Request</code>, <code>Response</code> or <code>fetch</code> on
|
||||
the <code>window</code> or <code>worker</code> scope.</p>
|
||||
<h2>Simple fetching</h2>
|
||||
<p>The most useful, high-level part of the Fetch API is the <code>fetch()</code> function.
|
||||
In its simplest form it takes a URL and returns a promise that resolves
|
||||
to the response. The response is captured as a <code>Response</code> object.</p>
|
||||
<DIV><pre>fetch<span>(</span><span>"/data.json"</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span><span>{</span><span>// res instanceof Response == true.</span><span>if</span><span>(</span>res.<span>ok</span><span>)</span><span>{</span>
|
||||
res.<span>json</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>data<span>)</span><span>{</span>
|
||||
console.<span>log</span><span>(</span>data.<span>entries</span><span>)</span><span>;</span><span>}</span><span>)</span><span>;</span><span>}</span><span>else</span><span>{</span>
|
||||
console.<span>log</span><span>(</span><span>"Looks like the response wasn't perfect, got status"</span><span>,</span> res.<span>status</span><span>)</span><span>;</span><span>}</span><span>}</span><span>,</span><span>function</span><span>(</span>e<span>)</span><span>{</span>
|
||||
console.<span>log</span><span>(</span><span>"Fetch failed!"</span><span>,</span> e<span>)</span><span>;</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>Submitting some parameters, it would look like this:</p>
|
||||
<DIV><pre>fetch<span>(</span><span>"http://www.example.org/submit.php"</span><span>,</span><span>{</span>
|
||||
method<span>:</span><span>"POST"</span><span>,</span>
|
||||
headers<span>:</span><span>{</span><span>"Content-Type"</span><span>:</span><span>"application/x-www-form-urlencoded"</span><span>}</span><span>,</span>
|
||||
body<span>:</span><span>"firstName=Nikhil&favColor=blue&password=easytoguess"</span><span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span><span>{</span><span>if</span><span>(</span>res.<span>ok</span><span>)</span><span>{</span>
|
||||
alert<span>(</span><span>"Perfect! Your settings are saved."</span><span>)</span><span>;</span><span>}</span><span>else</span><span>if</span><span>(</span>res.<span>status</span><span>==</span><span>401</span><span>)</span><span>{</span>
|
||||
alert<span>(</span><span>"Oops! You are not authorized."</span><span>)</span><span>;</span><span>}</span><span>}</span><span>,</span><span>function</span><span>(</span>e<span>)</span><span>{</span>
|
||||
alert<span>(</span><span>"Error submitting form!"</span><span>)</span><span>;</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>The <code>fetch()</code> function’s arguments are the same as those passed
|
||||
to the
|
||||
<br><code>Request()</code> constructor, so you may directly pass arbitrarily
|
||||
complex requests to <code>fetch()</code> as discussed below.</p>
|
||||
<h2>Headers</h2>
|
||||
<p>Fetch introduces 3 interfaces. These are <code>Headers</code>, <code>Request</code> and
|
||||
<br><code>Response</code>. They map directly to the underlying HTTP concepts,
|
||||
but have
|
||||
<br>certain visibility filters in place for privacy and security reasons,
|
||||
such as
|
||||
<br>supporting CORS rules and ensuring cookies aren’t readable by third parties.</p>
|
||||
<p>The <a href="https://fetch.spec.whatwg.org/#headers-class" target="_blank">Headers interface</a> is
|
||||
a simple multi-map of names to values:</p>
|
||||
<DIV><pre><span>var</span> content <span>=</span><span>"Hello World"</span><span>;</span><span>var</span> reqHeaders <span>=</span><span>new</span> Headers<span>(</span><span>)</span><span>;</span>
|
||||
reqHeaders.<span>append</span><span>(</span><span>"Content-Type"</span><span>,</span><span>"text/plain"</span>
|
||||
reqHeaders.<span>append</span><span>(</span><span>"Content-Length"</span><span>,</span> content.<span>length</span>.<span>toString</span><span>(</span><span>)</span><span>)</span><span>;</span>
|
||||
reqHeaders.<span>append</span><span>(</span><span>"X-Custom-Header"</span><span>,</span><span>"ProcessThisImmediately"</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>The same can be achieved by passing an array of arrays or a JS object
|
||||
literal
|
||||
<br>to the constructor:</p>
|
||||
<DIV><pre>reqHeaders <span>=</span><span>new</span> Headers<span>(</span><span>{</span><span>"Content-Type"</span><span>:</span><span>"text/plain"</span><span>,</span><span>"Content-Length"</span><span>:</span> content.<span>length</span>.<span>toString</span><span>(</span><span>)</span><span>,</span><span>"X-Custom-Header"</span><span>:</span><span>"ProcessThisImmediately"</span><span>,</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>The contents can be queried and retrieved:</p>
|
||||
<DIV><pre>console.<span>log</span><span>(</span>reqHeaders.<span>has</span><span>(</span><span>"Content-Type"</span><span>)</span><span>)</span><span>;</span><span>// true</span>
|
||||
console.<span>log</span><span>(</span>reqHeaders.<span>has</span><span>(</span><span>"Set-Cookie"</span><span>)</span><span>)</span><span>;</span><span>// false</span>
|
||||
reqHeaders.<span>set</span><span>(</span><span>"Content-Type"</span><span>,</span><span>"text/html"</span><span>)</span><span>;</span>
|
||||
reqHeaders.<span>append</span><span>(</span><span>"X-Custom-Header"</span><span>,</span><span>"AnotherValue"</span><span>)</span><span>;</span>
|
||||
|
||||
console.<span>log</span><span>(</span>reqHeaders.<span>get</span><span>(</span><span>"Content-Length"</span><span>)</span><span>)</span><span>;</span><span>// 11</span>
|
||||
console.<span>log</span><span>(</span>reqHeaders.<span>getAll</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>)</span><span>;</span><span>// ["ProcessThisImmediately", "AnotherValue"]</span>
|
||||
|
||||
reqHeaders.<span>delete</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>;</span>
|
||||
console.<span>log</span><span>(</span>reqHeaders.<span>getAll</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>)</span><span>;</span><span>// []</span></pre></DIV>
|
||||
<p>Some of these operations are only useful in ServiceWorkers, but they provide
|
||||
<br>a much nicer API to Headers.</p>
|
||||
<p>Since Headers can be sent in requests, or received in responses, and have
|
||||
various limitations about what information can and should be mutable, <code>Headers</code> objects
|
||||
have a <strong>guard</strong> property. This is not exposed to the Web, but
|
||||
it affects which mutation operations are allowed on the Headers object.
|
||||
<br>Possible values are:</p>
|
||||
<ul>
|
||||
<li>“none”: default.</li>
|
||||
<li>“request”: guard for a Headers object obtained from a Request (<code>Request.headers</code>).</li>
|
||||
<li>“request-no-cors”: guard for a Headers object obtained from a Request
|
||||
created
|
||||
<br>with mode “no-cors”.</li>
|
||||
<li>“response”: naturally, for Headers obtained from Response (<code>Response.headers</code>).</li>
|
||||
<li>“immutable”: Mostly used for ServiceWorkers, renders a Headers object
|
||||
<br>read-only.</li>
|
||||
</ul>
|
||||
<p>The details of how each guard affects the behaviors of the Headers object
|
||||
are
|
||||
<br>in the <a href="https://fetch.spec.whatwg.org/" target="_blank">specification</a>. For example,
|
||||
you may not append or set a “request” guarded Headers’ “Content-Length”
|
||||
header. Similarly, inserting “Set-Cookie” into a Response header is not
|
||||
allowed so that ServiceWorkers may not set cookies via synthesized Responses.</p>
|
||||
<p>All of the Headers methods throw TypeError if <code>name</code> is not a
|
||||
<a href="https://fetch.spec.whatwg.org/#concept-header-name" target="_blank">valid HTTP Header name</a>. The mutation operations will throw TypeError
|
||||
if there is an immutable guard. Otherwise they fail silently. For example:</p>
|
||||
<DIV><pre><span>var</span> res <span>=</span> Response.<span>error</span><span>(</span><span>)</span><span>;</span><span>try</span><span>{</span>
|
||||
res.<span>headers</span>.<span>set</span><span>(</span><span>"Origin"</span><span>,</span><span>"http://mybank.com"</span><span>)</span><span>;</span><span>}</span><span>catch</span><span>(</span>e<span>)</span><span>{</span>
|
||||
console.<span>log</span><span>(</span><span>"Cannot pretend to be a bank!"</span><span>)</span><span>;</span><span>}</span></pre></DIV>
|
||||
<h2>Request</h2>
|
||||
<p>The Request interface defines a request to fetch a resource over HTTP.
|
||||
URL, method and headers are expected, but the Request also allows specifying
|
||||
a body, a request mode, credentials and cache hints.</p>
|
||||
<p>The simplest Request is of course, just a URL, as you may do to GET a
|
||||
resource.</p>
|
||||
<DIV><pre><span>var</span> req <span>=</span><span>new</span> Request<span>(</span><span>"/index.html"</span><span>)</span><span>;</span>
|
||||
console.<span>log</span><span>(</span>req.<span>method</span><span>)</span><span>;</span><span>// "GET"</span>
|
||||
console.<span>log</span><span>(</span>req.<span>url</span><span>)</span><span>;</span><span>// "http://example.com/index.html"</span></pre></DIV>
|
||||
<p>You may also pass a Request to the <code>Request()</code> constructor to
|
||||
create a copy.
|
||||
<br>(This is not the same as calling the <code>clone()</code> method, which
|
||||
is covered in
|
||||
<br>the “Reading bodies” section.).</p>
|
||||
<DIV><pre><span>var</span> copy <span>=</span><span>new</span> Request<span>(</span>req<span>)</span><span>;</span>
|
||||
console.<span>log</span><span>(</span>copy.<span>method</span><span>)</span><span>;</span><span>// "GET"</span>
|
||||
console.<span>log</span><span>(</span>copy.<span>url</span><span>)</span><span>;</span><span>// "http://example.com/index.html"</span></pre></DIV>
|
||||
<p>Again, this form is probably only useful in ServiceWorkers.</p>
|
||||
<p>The non-URL attributes of the <code>Request</code> can only be set by passing
|
||||
initial
|
||||
<br>values as a second argument to the constructor. This argument is a dictionary.</p>
|
||||
<DIV><pre><span>var</span> uploadReq <span>=</span><span>new</span> Request<span>(</span><span>"/uploadImage"</span><span>,</span><span>{</span>
|
||||
method<span>:</span><span>"POST"</span><span>,</span>
|
||||
headers<span>:</span><span>{</span><span>"Content-Type"</span><span>:</span><span>"image/png"</span><span>,</span><span>}</span><span>,</span>
|
||||
body<span>:</span><span>"image data"</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>The Request’s mode is used to determine if cross-origin requests lead
|
||||
to valid responses, and which properties on the response are readable.
|
||||
Legal mode values are <code>"same-origin"</code>, <code>"no-cors"</code> (default)
|
||||
and <code>"cors"</code>.</p>
|
||||
<p>The <code>"same-origin"</code> mode is simple, if a request is made to another
|
||||
origin with this mode set, the result is simply an error. You could use
|
||||
this to ensure that
|
||||
<br>a request is always being made to your origin.</p>
|
||||
<DIV><pre><span>var</span> arbitraryUrl <span>=</span> document.<span>getElementById</span><span>(</span><span>"url-input"</span><span>)</span>.<span>value</span><span>;</span>
|
||||
fetch<span>(</span>arbitraryUrl<span>,</span><span>{</span> mode<span>:</span><span>"same-origin"</span><span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span><span>{</span>
|
||||
console.<span>log</span><span>(</span><span>"Response succeeded?"</span><span>,</span> res.<span>ok</span><span>)</span><span>;</span><span>}</span><span>,</span><span>function</span><span>(</span>e<span>)</span><span>{</span>
|
||||
console.<span>log</span><span>(</span><span>"Please enter a same-origin URL!"</span><span>)</span><span>;</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>The <code>"no-cors"</code> mode captures what the web platform does by default
|
||||
for scripts you import from CDNs, images hosted on other domains, and so
|
||||
on. First, it prevents the method from being anything other than “HEAD”,
|
||||
“GET” or “POST”. Second, if any ServiceWorkers intercept these requests,
|
||||
they may not add or override any headers except for <a href="https://fetch.spec.whatwg.org/#simple-header" target="_blank">these</a>.
|
||||
Third, JavaScript may not access any properties of the resulting Response.
|
||||
This ensures that ServiceWorkers do not affect the semantics of the Web
|
||||
and prevents security and privacy issues that could arise from leaking
|
||||
data across domains.</p>
|
||||
<p><code>"cors"</code> mode is what you’ll usually use to make known cross-origin
|
||||
requests to access various APIs offered by other vendors. These are expected
|
||||
to adhere to
|
||||
<br>the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS" target="_blank">CORS protocol</a>.
|
||||
Only a <a href="https://fetch.spec.whatwg.org/#concept-filtered-response-cors" target="_blank">limited set</a> of
|
||||
headers is exposed in the Response, but the body is readable. For example,
|
||||
you could get a list of Flickr’s <a href="https://www.flickr.com/services/api/flickr.interestingness.getList.html" target="_blank">most interesting</a> photos
|
||||
today like this:</p>
|
||||
<DIV><pre><span>var</span> u <span>=</span><span>new</span> URLSearchParams<span>(</span><span>)</span><span>;</span>
|
||||
u.<span>append</span><span>(</span><span>'method'</span><span>,</span><span>'flickr.interestingness.getList'</span><span>)</span><span>;</span>
|
||||
u.<span>append</span><span>(</span><span>'api_key'</span><span>,</span><span>'<insert api key here>'</span><span>)</span><span>;</span>
|
||||
u.<span>append</span><span>(</span><span>'format'</span><span>,</span><span>'json'</span><span>)</span><span>;</span>
|
||||
u.<span>append</span><span>(</span><span>'nojsoncallback'</span><span>,</span><span>'1'</span><span>)</span><span>;</span><span>var</span> apiCall <span>=</span> fetch<span>(</span><span>'https://api.flickr.com/services/rest?'</span><span>+</span> u<span>)</span><span>;</span>
|
||||
|
||||
apiCall.<span>then</span><span>(</span><span>function</span><span>(</span>response<span>)</span><span>{</span><span>return</span> response.<span>json</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>json<span>)</span><span>{</span><span>// photo is a list of photos.</span><span>return</span> json.<span>photos</span>.<span>photo</span><span>;</span><span>}</span><span>)</span><span>;</span><span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>photos<span>)</span><span>{</span>
|
||||
photos.<span>forEach</span><span>(</span><span>function</span><span>(</span>photo<span>)</span><span>{</span>
|
||||
console.<span>log</span><span>(</span>photo.<span>title</span><span>)</span><span>;</span><span>}</span><span>)</span><span>;</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>You may not read out the “Date” header since Flickr does not allow it
|
||||
via
|
||||
<br><code>Access-Control-Expose-Headers</code>.</p>
|
||||
<DIV><pre>response.<span>headers</span>.<span>get</span><span>(</span><span>"Date"</span><span>)</span><span>;</span><span>// null</span></pre></DIV>
|
||||
<p>The <code>credentials</code> enumeration determines if cookies for the other
|
||||
domain are
|
||||
<br>sent to cross-origin requests. This is similar to XHR’s <code>withCredentials</code><br>flag, but tri-valued as <code>"omit"</code> (default), <code>"same-origin"</code> and <code>"include"</code>.</p>
|
||||
<p>The Request object will also give the ability to offer caching hints to
|
||||
the user-agent. This is currently undergoing some <a href="https://github.com/slightlyoff/ServiceWorker/issues/585" target="_blank">security review</a>.
|
||||
Firefox exposes the attribute, but it has no effect.</p>
|
||||
<p>Requests have two read-only attributes that are relevant to ServiceWorkers
|
||||
<br>intercepting them. There is the string <code>referrer</code>, which is
|
||||
set by the UA to be
|
||||
<br>the referrer of the Request. This may be an empty string. The other is
|
||||
<br><code>context</code> which is a rather <a href="https://fetch.spec.whatwg.org/#requestcredentials" target="_blank">large enumeration</a> defining
|
||||
what sort of resource is being fetched. This could be “image” if the request
|
||||
is from an
|
||||
<img>tag in the controlled document, “worker” if it is an attempt to load a
|
||||
worker script, and so on. When used with the <code>fetch()</code> function,
|
||||
it is “fetch”.</p>
|
||||
<h2>Response</h2>
|
||||
<p><code>Response</code> instances are returned by calls to <code>fetch()</code>.
|
||||
They can also be created by JS, but this is only useful in ServiceWorkers.</p>
|
||||
<p>We have already seen some attributes of Response when we looked at <code>fetch()</code>.
|
||||
The most obvious candidates are <code>status</code>, an integer (default
|
||||
value 200) and <code>statusText</code> (default value “OK”), which correspond
|
||||
to the HTTP status code and reason. The <code>ok</code> attribute is just
|
||||
a shorthand for checking that <code>status</code> is in the range 200-299
|
||||
inclusive.</p>
|
||||
<p><code>headers</code> is the Response’s Headers object, with guard “response”.
|
||||
The <code>url</code> attribute reflects the URL of the corresponding request.</p>
|
||||
<p>Response also has a <code>type</code>, which is “basic”, “cors”, “default”,
|
||||
“error” or
|
||||
<br>“opaque”.</p>
|
||||
<ul>
|
||||
<li>
|
||||
<code>"basic"</code>: normal, same origin response, with all headers exposed
|
||||
except
|
||||
<br>“Set-Cookie” and “Set-Cookie2″.</li>
|
||||
<li>
|
||||
<code>"cors"</code>: response was received from a valid cross-origin request.
|
||||
<a href="https://fetch.spec.whatwg.org/#concept-filtered-response-cors" target="_blank">Certain headers and the body</a>may be accessed.</li>
|
||||
<li>
|
||||
<code>"error"</code>: network error. No useful information describing
|
||||
the error is available. The Response’s status is 0, headers are empty and
|
||||
immutable. This is the type for a Response obtained from <code>Response.error()</code>.</li>
|
||||
<li>
|
||||
<code>"opaque"</code>: response for “no-cors” request to cross-origin
|
||||
resource. <a href="https://fetch.spec.whatwg.org/#concept-filtered-response-opaque" target="_blank">Severely<br>
|
||||
restricted</a>
|
||||
</li>
|
||||
</ul>
|
||||
<p>The “error” type results in the <code>fetch()</code> Promise rejecting with
|
||||
TypeError.</p>
|
||||
<p>There are certain attributes that are useful only in a ServiceWorker scope.
|
||||
The
|
||||
<br>idiomatic way to return a Response to an intercepted request in ServiceWorkers
|
||||
is:</p>
|
||||
<DIV><pre>addEventListener<span>(</span><span>'fetch'</span><span>,</span><span>function</span><span>(</span>event<span>)</span><span>{</span>
|
||||
event.<span>respondWith</span><span>(</span><span>new</span> Response<span>(</span><span>"Response body"</span><span>,</span><span>{</span>
|
||||
headers<span>:</span><span>{</span><span>"Content-Type"</span><span>:</span><span>"text/plain"</span><span>}</span><span>}</span><span>)</span><span>;</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>As you can see, Response has a two argument constructor, where both arguments
|
||||
are optional. The first argument is a body initializer, and the second
|
||||
is a dictionary to set the <code>status</code>, <code>statusText</code> and <code>headers</code>.</p>
|
||||
<p>The static method <code>Response.error()</code> simply returns an error
|
||||
response. Similarly, <code>Response.redirect(url, status)</code> returns
|
||||
a Response resulting in
|
||||
<br>a redirect to <code>url</code>.</p>
|
||||
<h2>Dealing with bodies</h2>
|
||||
<p>Both Requests and Responses may contain body data. We’ve been glossing
|
||||
over it because of the various data types body may contain, but we will
|
||||
cover it in detail now.</p>
|
||||
<p>A body is an instance of any of the following types.</p>
|
||||
<ul>
|
||||
<li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer" target="_blank">ArrayBuffer</a></li>
|
||||
<li>
|
||||
<a href="https://developer.mozilla.org/en-US/docs/Web/API/ArrayBufferView" target="_blank">ArrayBufferView</a> (Uint8Array
|
||||
and friends)</li>
|
||||
<li>
|
||||
<a href="https://developer.mozilla.org/en-US/docs/Web/API/Blob" target="_blank">Blob</a>/
|
||||
<a href="https://developer.mozilla.org/en-US/docs/Web/API/File" target="_blank">File</a>
|
||||
</li>
|
||||
<li>string</li>
|
||||
<li><a href="https://url.spec.whatwg.org/#interface-urlsearchparams" target="_blank">URLSearchParams</a></li>
|
||||
<li>
|
||||
<a href="https://developer.mozilla.org/en-US/docs/Web/API/FormData" target="_blank">FormData</a> –
|
||||
currently not supported by either Gecko or Blink. Firefox expects to ship
|
||||
this in version 39 along with the rest of Fetch.</li>
|
||||
</ul>
|
||||
<p>In addition, Request and Response both offer the following methods to
|
||||
extract their body. These all return a Promise that is eventually resolved
|
||||
with the actual content.</p>
|
||||
<ul>
|
||||
<li><code>arrayBuffer()</code></li>
|
||||
<li><code>blob()</code></li>
|
||||
<li><code>json()</code></li>
|
||||
<li><code>text()</code></li>
|
||||
<li><code>formData()</code></li>
|
||||
</ul>
|
||||
<p>This is a significant improvement over XHR in terms of ease of use of
|
||||
non-text data!</p>
|
||||
<p>Request bodies can be set by passing <code>body</code> parameters:</p>
|
||||
<DIV><pre><span>var</span> form <span>=</span><span>new</span> FormData<span>(</span>document.<span>getElementById</span><span>(</span><span>'login-form'</span><span>)</span><span>)</span><span>;</span>
|
||||
fetch<span>(</span><span>"/login"</span><span>,</span><span>{</span>
|
||||
method<span>:</span><span>"POST"</span><span>,</span>
|
||||
body<span>:</span> form
|
||||
<span>}</span><span>)</span></pre></DIV>
|
||||
<p>Responses take the first argument as the body.</p>
|
||||
<DIV><pre><span>var</span> res <span>=</span><span>new</span> Response<span>(</span><span>new</span> File<span>(</span><span>[</span><span>"chunk"</span><span>,</span><span>"chunk"</span><span>]</span><span>,</span><span>"archive.zip"</span><span>,</span><span>{</span> type<span>:</span><span>"application/zip"</span><span>}</span><span>)</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>Both Request and Response (and by extension the <code>fetch()</code> function),
|
||||
will try to intelligently <a href="https://fetch.spec.whatwg.org/#concept-bodyinit-extract" target="_blank">determine the content type</a>.
|
||||
Request will also automatically set a “Content-Type” header if none is
|
||||
set in the dictionary.</p>
|
||||
<h3>Streams and cloning</h3>
|
||||
<p>It is important to realise that Request and Response bodies can only be
|
||||
read once! Both interfaces have a boolean attribute <code>bodyUsed</code> to
|
||||
determine if it is safe to read or not.</p>
|
||||
<DIV><pre><span>var</span> res <span>=</span><span>new</span> Response<span>(</span><span>"one time use"</span><span>)</span><span>;</span>
|
||||
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span><span>// false</span>
|
||||
res.<span>text</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>v<span>)</span><span>{</span>
|
||||
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span><span>// true</span><span>}</span><span>)</span><span>;</span>
|
||||
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span><span>// true</span>
|
||||
|
||||
res.<span>text</span><span>(</span><span>)</span>.<span>catch</span><span>(</span><span>function</span><span>(</span>e<span>)</span><span>{</span>
|
||||
console.<span>log</span><span>(</span><span>"Tried to read already consumed Response"</span><span>)</span><span>;</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<p>This decision allows easing the transition to an eventual <a href="https://streams.spec.whatwg.org/" target="_blank">stream-based</a> Fetch
|
||||
API. The intention is to let applications consume data as it arrives, allowing
|
||||
for JavaScript to deal with larger files like videos, and perform things
|
||||
like compression and editing on the fly.</p>
|
||||
<p>Often, you’ll want access to the body multiple times. For example, you
|
||||
can use the upcoming <a href="http://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#cache-objects" target="_blank">Cache API</a> to
|
||||
store Requests and Responses for offline use, and Cache requires bodies
|
||||
to be available for reading.</p>
|
||||
<p>So how do you read out the body multiple times within such constraints?
|
||||
The API provides a <code>clone()</code> method on the two interfaces. This
|
||||
will return a clone of the object, with a ‘new’ body. <code>clone()</code> MUST
|
||||
be called before the body of the corresponding object has been used. That
|
||||
is, <code>clone()</code> first, read later.</p>
|
||||
<DIV><pre>addEventListener<span>(</span><span>'fetch'</span><span>,</span><span>function</span><span>(</span>evt<span>)</span><span>{</span><span>var</span> sheep <span>=</span><span>new</span> Response<span>(</span><span>"Dolly"</span><span>)</span><span>;</span>
|
||||
console.<span>log</span><span>(</span>sheep.<span>bodyUsed</span><span>)</span><span>;</span><span>// false</span><span>var</span> clone <span>=</span> sheep.<span>clone</span><span>(</span><span>)</span><span>;</span>
|
||||
console.<span>log</span><span>(</span>clone.<span>bodyUsed</span><span>)</span><span>;</span><span>// false</span>
|
||||
|
||||
clone.<span>text</span><span>(</span><span>)</span><span>;</span>
|
||||
console.<span>log</span><span>(</span>sheep.<span>bodyUsed</span><span>)</span><span>;</span><span>// false</span>
|
||||
console.<span>log</span><span>(</span>clone.<span>bodyUsed</span><span>)</span><span>;</span><span>// true</span>
|
||||
|
||||
evt.<span>respondWith</span><span>(</span>cache.<span>add</span><span>(</span>sheep.<span>clone</span><span>(</span><span>)</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>e<span>)</span><span>{</span><span>return</span> sheep<span>;</span><span>}</span><span>)</span><span>;</span><span>}</span><span>)</span><span>;</span></pre></DIV>
|
||||
<h2>Future improvements</h2>
|
||||
<p>Along with the transition to streams, Fetch will eventually have the ability
|
||||
to abort running <code>fetch()</code>es and some way to report the progress
|
||||
of a fetch. These are provided by XHR, but are a little tricky to fit in
|
||||
the Promise-based nature of the Fetch API.</p>
|
||||
<p>You can contribute to the evolution of this API by participating in discussions
|
||||
on the <a href="https://whatwg.org/mailing-list" target="_blank">WHATWG mailing list</a> and
|
||||
in the issues in the <a href="https://www.w3.org/Bugs/Public/buglist.cgi?product=WHATWG&component=Fetch&resolution=---" target="_blank">Fetch</a> and
|
||||
<a href="https://github.com/slightlyoff/ServiceWorker/issues" target="_blank">ServiceWorker</a>specifications.</p>
|
||||
<p>For a better web!</p>
|
||||
<p><em>The author would like to thank Andrea Marchesini, Anne van Kesteren and Ben<br>
|
||||
Kelly for helping with the specification and implementation.</em></p></article></DIV></article>
|
||||
21
resources/tests/readability/iab-1/expected.html
Normal file
21
resources/tests/readability/iab-1/expected.html
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
<article><div id="readability-page-1">
|
||||
<p>We messed up. As technologists, tasked with delivering content and services to users, we lost track of the user experience.</p>
|
||||
<p>Twenty years ago we saw an explosion of websites, built by developers around the world, providing all forms of content. This was the beginning of an age of enlightenment, the intersection of content and technology. Many of us in the technical field felt compelled, and even empowered, to produce information as the distribution means for mass communication were no longer restricted by a high barrier to entry.</p>
|
||||
<p>In 2000, the dark ages came when the dot-com bubble burst. We were told that our startups were gone or that our divisions sustained by corporate parent companies needed to be in the black. It was a wakeup call that led to a renaissance age. Digital advertising became the foundation of an economic engine that, still now, sustains the free and democratic World Wide Web. In digital publishing, we strived to balance content, commerce, and technology. The content management systems and communication gateways we built to inform and entertain populations around the world disrupted markets and in some cases governments, informed communities of imminent danger, and liberated new forms of art and entertainment—all while creating a digital middle class of small businesses.</p>
|
||||
<p>We engineered not just the technical, but also the social and economic foundation that users around the world came to lean on for access to real time information. And users came to expect this information whenever and wherever they needed it. And more often than not, for anybody with a connected device, it was free.</p>
|
||||
<p>This was choice—powered by digital advertising—and premised on user experience.</p>
|
||||
<p>But we messed up.</p>
|
||||
<p>Through our pursuit of further automation and maximization of margins during the industrial age of media technology, we built advertising technology to optimize publishers’ yield of marketing budgets that had eroded after the last recession. Looking back now, our scraping of dimes may have cost us dollars in consumer loyalty. The fast, scalable systems of targeting users with ever-heftier advertisements have slowed down the public internet and drained more than a few batteries. We were so clever and so good at it that we over-engineered the capabilities of the plumbing laid down by, well, ourselves. This steamrolled the users, depleted their devices, and tried their patience.</p>
|
||||
<p>The rise of ad blocking poses a threat to the internet and could potentially drive users to an enclosed platform world dominated by a few companies. We have let the fine equilibrium of content, commerce, and technology get out of balance in the open web. We had, and still do have, a responsibility to educate the business side, and in some cases to push back. We lost sight of our social and ethical responsibility to provide a safe, usable experience for anyone and everyone wanting to consume the content of their choice.</p>
|
||||
<p>We need to bring that back into alignment, starting right now.</p>
|
||||
<p><a href="http://www.iab.com/wp-content/uploads/2015/10/getting-lean-with-digital-ad-ux.jpg" target="_blank"><img width="300" height="250" alt="Getting LEAN with Digital Ad UX" src="http://www.iab.com/wp-content/uploads/2015/10/getting-lean-with-digital-ad-ux-300x250.jpg"></a>Today, the IAB Tech Lab is launching the L.E.A.N. Ads program. Supported by the Executive Committee of the IAB Tech Lab Board, IABs around the world, and hundreds of member companies, L.E.A.N. stands for Light, Encrypted, Ad choice supported, Non-invasive ads. These are principles that will help guide the next phases of advertising technical standards for the global digital advertising supply chain.</p>
|
||||
<p>As with any other industry, standards should be created by non-profit standards-setting bodies, with many diverse voices providing input. We will invite all parties for public comment, and make sure consumer interest groups have the opportunity to provide input.</p>
|
||||
<p>L.E.A.N. Ads do not replace the current advertising standards many consumers still enjoy and engage with while consuming content on our sites across all IP enabled devices. Rather, these principles will guide an alternative set of standards that provide choice for marketers, content providers, and consumers.</p>
|
||||
<p>Among the many areas of concentration, we must also address frequency capping on retargeting in Ad Tech and make sure a user is targeted appropriately before, but never AFTER they make a purchase. If we are so good at reach and scale, we can be just as good, if not better, at moderation. Additionally, we must address volume of ads per page as well as continue on the path to viewability. The dependencies here are critical to an optimized user experience.</p>
|
||||
<p>The consumer is demanding these actions, challenging us to do better, and we must respond.</p>
|
||||
<p>The IAB Tech Lab will continue to provide the tools for publishers in the digital supply chain to have a dialogue with users about their choices so that content providers can generate revenue while creating value. Publishers should have the opportunity to provide rich advertising experiences, L.E.A.N. advertising experiences, and subscription services. Or publishers can simply deny their service to users who choose to keep on blocking ads. That is all part of elasticity of consumer tolerance and choice.</p>
|
||||
<p>Finally, we must do this in an increasingly fragmented market, across screens. We must do this in environments where entire sites are blocked, purposefully or not. Yes, it is disappointing that our development efforts will have to manage with multiple frameworks while we work to supply the economic engine to sustain an open internet. However, our goal is still to provide diverse content and voices to as many connected users as possible around the world.</p>
|
||||
<p>That is user experience.</p>
|
||||
<P>IAB Tech Lab Members can join the IAB Tech Lab Ad Blocking Working Group, please email <a href="mailto:adblocking@iab.com" target="_blank">adblocking@iab.com</a> for more information.</P>
|
||||
<p>Read <a target="_blank" href="http://www.iab.com/insights/ad-blocking/">more about ad blocking here</a>.</p>
|
||||
</div></article>
|
||||
1103
resources/tests/readability/iab-1/source.html
Normal file
1103
resources/tests/readability/iab-1/source.html
Normal file
File diff suppressed because one or more lines are too long
|
|
@ -239,6 +239,11 @@ async fn hukumusume() {
|
|||
run_test("hukumusume").await
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn iab_1() {
|
||||
run_test("iab-1").await
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn webmd_1() {
|
||||
run_test("webmd-1").await
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue