As a fallback, if no providers reachable from a browser are found, data is retrieved using recursive gateways, e.g. trustless-gateway.link which can be configured.
This is a marked improvement over fetch which offers no such protections and is vulnerable to all sorts of attacks like Content Spoofing, DNS Hijacking, etc.
A verifiedFetch function is exported to get up and running quickly, and a createVerifiedFetch function is also available that allows customizing the underlying Helia node for complete control over how content is retrieved.
Browser-cache-friendly Response objects are returned which should be instantly familiar to web developers.
Out of the box @helia/verified-fetch uses a default set of trustless gateways for fetching blocks and HTTP delegated routers for performing routing tasks - looking up peers, resolving/publishing IPNS names, etc.
It's possible to override these by passing gateways and routers keys to the createVerifiedFetch function:
For full control of how @helia/verified-fetch fetches content from the distributed web you can pass a preconfigured Helia node to createVerifiedFetch.
The helia module is configured with a libp2p node that is suited for decentralized applications, alternatively @helia/http is available which uses HTTP gateways for all network operations.
By default, if the response can be parsed as JSON, @helia/verified-fetch sets the Content-Type header as application/json, otherwise it sets it as application/octet-stream - this is because the .json(), .text(), .blob(), and .arrayBuffer() methods will usually work as expected without a detailed content type.
If you require an accurate content-type you can provide a contentTypeParser function as an option to createVerifiedFetch to handle parsing the content type.
The function you provide will be called with the first chunk of bytes from the file and should return a string or a promise of a string.
constfetch = awaitcreateVerifiedFetch({ gateways: ['https://trustless-gateway.link'], routers: ['http://delegated-ipfs.dev'], dnsResolvers: { // this resolver will only be used for `.com` domains (note - this could // also be an array of resolvers) 'com.':dnsJsonOverHttps('https://my-dns-resolver.example.com/dns-json'), // this resolver will be used for everything else (note - this could // also be an array of resolvers) '.':dnsOverHttps('https://my-dns-resolver.example.com/dns-query') } })
Custom Hashers
By default, @helia/verified-fetch supports sha256, sha512, and identity hashers.
If you need to use a different hasher, you can provide a custom hasher function as an option to createVerifiedFetch.
IPFS supports several data formats (typically referred to as codecs) which are included in the CID. @helia/verified-fetch attempts to abstract away some of the details for easier consumption.
DAG-PB
DAG-PB is the codec we are most likely to encounter, it is what UnixFS uses under the hood.
When the JSON codec is encountered, the Content-Type header of the response will be set to application/json.
DAG-JSON
DAG-JSON expands on the JSON codec, adding the ability to contain CIDs which act as links to other blocks, and byte arrays.
CIDs and byte arrays are represented using special object structures with a single "/" property.
Using DAG-JSON has two important caveats:
Your JSON structure cannot contain an object with only a "/" property, as it will be interpreted as a special type.
Since JSON has no technical limit on number sizes, DAG-JSON also allows numbers larger than Number.MAX_SAFE_INTEGER. JavaScript requires use of BigInts to represent numbers larger than this, and JSON.parse does not support them, so precision will be lost.
Otherwise this codec follows the same rules as the JSON codec.
Alternatively, it can be decoded using the @ipld/dag-json module and the .arrayBuffer() method, in which case you will get CID objects and Uint8Arrays:
This supports more data types in a safer way than JSON and is smaller on the wire to boot so is usually preferable to JSON or DAG-JSON.
Content-Type
Not all data types supported by DAG-CBOR can be successfully turned into JSON and back into the same binary form.
When a decoded block can be round-tripped to JSON, the Content-Type will be set to application/json. In this case the .json() method on the Response object can be used to obtain an object representation of the response.
When it cannot, the Content-Type will be application/octet-stream - in this case the @ipld/dag-json module must be used to deserialize the return value from .arrayBuffer().
Detecting JSON-safe DAG-CBOR
If the Content-Type header of the response is application/json, the .json() method may be used to access the response body in object form, otherwise the .arrayBuffer() method must be used to decode the raw bytes using the @ipld/dag-cbor module.
if (res.headers.get('Content-Type') === 'application/json') { // DAG-CBOR data can be safely decoded as JSON obj = awaitres.json() } else { // response contains non-JSON friendly data types obj = dagCbor.decode(awaitres.arrayBuffer()) }
console.info(obj) // ...
The Accept header
The Accept header can be passed to override certain response processing, or to ensure that the final Content-Type of the response is the one that is expected.
If the final Content-Type does not match the Accept header, or if the content cannot be represented in the format dictated by the Accept header, or you have configured a custom content type parser, and that parser returns a value that isn't in the accept header, a 406: Not Acceptable response will be returned:
console.info(res.headers.get('accept')) // application/octet-stream constbuf = awaitres.arrayBuffer() // raw bytes, not processed as JSON
Redirects
If a requested URL contains a path component, that path component resolves to
a UnixFS directory, but the URL does not have a trailing slash, one will be
added to form a canonical URL for that resource, otherwise the request will
be resolved as normal.
Manual redirects allow the user to process the redirect. A 301
is returned, and the location to redirect to is available as the "location"
response header.
This differs slightly from HTTP fetch which returns an opaque response as the
browser itself is expected to process the redirect and hide all details from
the user.
If you pass a CID instance, it assumes you want the content for that specific CID only, and does not support pathing or params for that CID.
Options argument
This library does not plan to support the exact Fetch API options object, as some of the arguments don't make sense. Instead, it will only support options necessary to meet IPFS specs related to specifying the resultant shape of desired content.
Where possible, options and Helia internals will be automatically configured to the appropriate codec & content type based on the verified-fetch configuration and options argument passed.
Known Fetch API options that will be supported:
signal - An AbortSignal that a user can use to abort the request.
redirect - A string that specifies the redirect type. One of follow, error, or manual. Defaults to follow. Best effort to adhere to the Fetch API redirect parameter.
headers - An object of headers to be sent with the request. Best effort to adhere to the Fetch API headers parameter.
accept - A string that specifies the accept header. Relevant values:
method - A string that specifies the HTTP method to use for the request. Defaults to GET. Best effort to adhere to the Fetch API method parameter.
body - An object that specifies the body of the request. Best effort to adhere to the Fetch API body parameter.
cache - Will basically act as force-cache for the request. Best effort to adhere to the Fetch API cache parameter.
Non-Fetch API options that will be supported:
onProgress - Similar to Helia onProgress options, this will be a function that will be called with a progress event. Supported progress events are:
helia:verified-fetch:error - An error occurred during the request.
helia:verified-fetch:request:start - The request has been sent
helia:verified-fetch:request:complete - The request has been sent
helia:verified-fetch:request:error - An error occurred during the request.
helia:verified-fetch:request:abort - The request was aborted prior to completion.
helia:verified-fetch:response:start - The initial HTTP Response headers have been set, and response stream is started.
helia:verified-fetch:response:complete - The response stream has completed.
helia:verified-fetch:response:error - An error occurred while building the response.
Some in-flight specs (IPIPs) that will affect the options object this library supports in the future can be seen at https://specs.ipfs.tech/ipips, a subset are:
This library's purpose is to return reasonably representable content from IPFS. In other words, fetching content is intended for leaf-node content -- such as images/videos/audio & other assets, or other IPLD content (with link) -- that can be represented by https://developer.mozilla.org/en-US/docs/Web/API/Response#instance_methods. The content type you receive back will depend upon the CID you request as well as the Accept header value you provide.
All content we retrieve from the IPFS network is obtained via an AsyncIterable, and will be set as the body of the HTTP Response via a ReadableStream or other efficient method that avoids loading the entire response into memory or getting the entire response from the network before returning a response to the user.
If your content doesn't have a mime-type or an IPFS spec, this library will not support it, but you can use the helia library directly for those use cases. See Unsupported response types for more information.
Handling response types
For handling responses we want to follow conventions/abstractions from Fetch API where possible:
For JSON, assuming you abstract any differences between dag-json/dag-cbor/json/and json-file-on-unixfs, you would call .json() to get a JSON object.
For images (or other web-relevant asset) you want to add to the DOM, use .blob() or .arrayBuffer() to get the raw bytes.
For plain text in utf-8, you would call .text()
For streaming response data, use something like response.body.getReader() to get a ReadableStream.
Unsupported response types
Returning IPLD nodes or DAGs as JS objects is not supported, as there is no currently well-defined structure for representing this data in an HTTP Response. Instead, users should request aplication/vnd.ipld.car or use the helia library directly for this use case.
Others? Open an issue or PR!
Response headers
This library will set the HTTP Response headers to the appropriate values for the content type according to the appropriate IPFS Specifications.
By default, we do not include Server Timing headers in responses. If you want to include them, you can pass an
withServerTiming option to the createVerifiedFetch function to include them in all future responses. You can
also pass the withServerTiming option to each fetch call to include them only for that specific response.
Attempting to fetch the CID for content that does not make sense
If you request bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze, which points to the root of the en.wikipedia.org mirror, a response object does not make sense.
Errors
Known Errors that can be thrown:
TypeError - If the resource argument is not a string, CID, or CID string.
TypeError - If the options argument is passed and not an object.
TypeError - If the options argument is passed and is malformed.
AbortError - If the content request is aborted due to user aborting provided AbortSignal. Note that this is a AbortError from @libp2p/interface and not the standard AbortError from the Fetch API.
Extensibility
Verified‑fetch can now be extended to alter how it handles requests by using plugins.
Plugins are classes that extend the BasePlugin class and implement the VerifiedFetchPlugin
interface. They are instantiated with PluginOptions when the VerifiedFetch class is created.
Plugin Interface
Each plugin must implement two methods:
canHandle(context: PluginContext): boolean
Inspects the current PluginContext (which includes the CID, path, query, accept header, etc.)
and returns true if the plugin can operate on the current state of the request.
handle(context: PluginContext): Promise<Response | null>
Performs the plugin’s work. It may:
Return a final Response: This stops the pipeline immediately.
Return null: This indicates that the plugin has only partially processed the request
(for example, by performing path walking or decoding) and the pipeline should continue.
Throw a PluginError: This logs a non-fatal error and continues the pipeline.
Throw a PluginFatalError: This logs a fatal error and stops the pipeline immediately.
Plugin Pipeline
Plugins are executed in a chain (a plugin pipeline):
Initialization:
The VerifiedFetch class is instantiated with a list of plugins.
When a request is made via the fetch method, the resource and options are parsed to
create a mutable PluginContext object.
Pipeline Execution:
The pipeline repeatedly checks, up to a maximum number of passes (default = 3), which plugins
are currently able to handle the request by calling each plugin’s canHandle() method.
Plugins that have not yet been called in the current run and return true for canHandle()
are invoked in sequence.
If a plugin returns a final Response or throws a PluginFatalError, the pipeline immediately
stops and that response is returned.
If a plugin returns null, it may have updated the context (for example, by
performing path walking), other plugins that said they canHandle will run.
If no plugin modifies the context (i.e. no change to context.modified) and no final response is
produced after iterating through all plugins, the pipeline exits and a default “Not Supported”
response is returned.
Diagram of the Plugin Pipeline:
%%{init:{"theme":"dark"}}%%
flowchart TD
A[Resource & Options] --> B[Parse into PluginContext]
B --> C[Plugin Pipeline]
subgraph IP[Iterative Passes max 3 passes]
C1[Check canHandle for each plugin]
C2[Call handle on ready plugins]
C3[Update PluginContext if partial work is done]
C1 --> C2
C2 --> C3
end
C --> IP
IP --> D[Final Response]
%%{init:{"theme":"default"}}%%
flowchart TD
A[Resource & Options] --> B[Parse into PluginContext]
B --> C[Plugin Pipeline]
subgraph IP[Iterative Passes max 3 passes]
C1[Check canHandle for each plugin]
C2[Call handle on ready plugins]
C3[Update PluginContext if partial work is done]
C1 --> C2
C2 --> C3
end
C --> IP
IP --> D[Final Response]
flowchart TD
A[Resource & Options] --> B[Parse into PluginContext]
B --> C[Plugin Pipeline]
subgraph IP[Iterative Passes max 3 passes]
C1[Check canHandle for each plugin]
C2[Call handle on ready plugins]
C3[Update PluginContext if partial work is done]
C1 --> C2
C2 --> C3
end
C --> IP
IP --> D[Final Response]
Finalization:
After the pipeline completes, the resulting response & context is processed (e.g. headers such as ETag,
Cache‑Control, and Content‑Disposition are set) and returned.
Please see the original discussion on extensibility in Issue #167.
Non-default plugins provided by this library
dir-index-html-plugin
This plugin is used to serve dag-pb/unixfs without an index.html child as HTML directory listing of the content requested.
dag-cbor-html-preview-plugin
This plugin is used to serve the requested dag-cbor object as HTML when the Accept header includes text/html.
exportclassMyCustomPluginextendsBasePlugin { id = 'my-custom-plugin'// Required: must be unique unless you want to override one of the default plugins.
// Optionally, list any codec codes your plugin supports: codes = [] //
canHandle(context: PluginContext): boolean { // Only handle requests if the Accept header matches your custom type // Or check context for pathDetails, custom values, etc... returncontext.accept === 'application/vnd.my-custom-type' }
asynchandle(context: PluginContext): Promise<Response | null> { // Perform any partial processing here, e.g., modify the context: context.customProcessed = true;
// If you are ready to finalize the response: returnnewResponse('Hello, world!', { status:200, headers: { 'Content-Type':'text/plain' } });
// Or, if further processing is needed by another plugin, simply return null. } } exportconstmyCustomPluginFactory: VerifiedFetchPluginFactory = (opts: PluginOptions) =>newMyCustomPlugin(opts)
Integrate Your Plugin:
Add your custom plugin to Verified‑Fetch’s plugin list when instantiating Verified‑Fetch:
// async handle(context: PluginContext): Promise<Response | null> { constrecoverable = Math.random() > 0.5// Use more sophisticated logic here ;) if (recoverable === true) { thrownewPluginError('MY_CUSTOM_WARNING', 'A non‑fatal issue occurred', { details: { someKey:'Additional details here' } }); }
if (recoverable === false) { thrownewPluginFatalError('MY_CUSTOM_FATAL', 'A critical error occurred', { response:newResponse('Something happened', { status:500 }) // Required: supply your own error response }); }
// Otherwise, continue processing... // }
How the Plugin Pipeline Works
Shared Context:
A mutable PluginContext is created for each request. It includes the parsed CID, path, query parameters,
accept header, and any other metadata. Plugins can update this context as they perform partial work (for example,
by doing path walking or decoding).
Iterative Processing:
The pipeline repeatedly checks which plugins can currently handle the request by calling canHandle(context).
Plugins that perform partial processing update the context and return null, allowing subsequent passes by other plugins.
Once a plugin is ready to finalize the response, it returns a final Response and the pipeline terminates.
No Strict Ordering:
Plugins are invoked based solely on whether they can handle the current state of the context.
This means you do not have to specify a rigid order, each plugin simply checks the context and acts if appropriate.
Error Handling:
A thrown PluginError is considered non‑fatal and is logged, allowing the pipeline to continue.
A thrown PluginFatalError immediately stops the pipeline and returns the error response.
For a detailed explanation of the pipeline, please refer to the discussion in Issue #167.
@helia/verified-fetch
provides a fetch-like API for retrieving content from the IPFS network.All content is retrieved in a trustless manner, and the integrity of all bytes are verified by comparing hashes of the data.
By default, providers for CIDs are found using delegated routing endpoints.
Data is retrieved using the following strategies:
trustless-gateway.link
which can be configured.This is a marked improvement over
fetch
which offers no such protections and is vulnerable to all sorts of attacks like Content Spoofing, DNS Hijacking, etc.A
verifiedFetch
function is exported to get up and running quickly, and acreateVerifiedFetch
function is also available that allows customizing the underlying Helia node for complete control over how content is retrieved.Browser-cache-friendly Response objects are returned which should be instantly familiar to web developers.
Learn more in the announcement blog post and check out the ready-to-run example.
You may use any supported resource argument to fetch content:
Example: Getting started
Example: Using a CID instance to fetch JSON
Example: Using IPFS protocol to fetch an image
Example: Using IPNS protocol to stream a big file
Configuration
Custom HTTP gateways and routers
Out of the box
@helia/verified-fetch
uses a default set of trustless gateways for fetching blocks and HTTP delegated routers for performing routing tasks - looking up peers, resolving/publishing IPNS names, etc.It's possible to override these by passing
gateways
androuters
keys to thecreateVerifiedFetch
function:Example: Configuring gateways and routers
Usage with customized Helia
For full control of how
@helia/verified-fetch
fetches content from the distributed web you can pass a preconfigured Helia node tocreateVerifiedFetch
.The helia module is configured with a libp2p node that is suited for decentralized applications, alternatively @helia/http is available which uses HTTP gateways for all network operations.
You can see variations of Helia and js-libp2p configuration options at https://ipfs.github.io/helia/interfaces/helia.HeliaInit.html.
Custom content-type parsing
By default, if the response can be parsed as JSON,
@helia/verified-fetch
sets theContent-Type
header asapplication/json
, otherwise it sets it asapplication/octet-stream
- this is because the.json()
,.text()
,.blob()
, and.arrayBuffer()
methods will usually work as expected without a detailed content type.If you require an accurate content-type you can provide a
contentTypeParser
function as an option tocreateVerifiedFetch
to handle parsing the content type.The function you provide will be called with the first chunk of bytes from the file and should return a string or a promise of a string.
Example: Customizing content-type parsing
Custom DNS resolvers
If you don't want to leak DNS queries to the default resolvers, you can provide your own list of DNS resolvers to
createVerifiedFetch
.Note that you do not need to provide both a DNS-over-HTTPS and a DNS-over-JSON resolver, and you should prefer
dnsJsonOverHttps
resolvers for usage in the browser for a smaller bundle size. See https://github.com/ipfs/helia/tree/main/packages/ipns#example---using-dns-json-over-https for more information.Example: Customizing DNS resolvers
Example: Customizing DNS per-TLD resolvers
DNS resolvers can be configured to only service DNS queries for specific TLDs:
Custom Hashers
By default,
@helia/verified-fetch
supportssha256
,sha512
, andidentity
hashers.If you need to use a different hasher, you can provide a custom
hasher
function as an option tocreateVerifiedFetch
.Example: Passing a custom hashing function
IPLD codec handling
IPFS supports several data formats (typically referred to as codecs) which are included in the CID.
@helia/verified-fetch
attempts to abstract away some of the details for easier consumption.DAG-PB
DAG-PB is the codec we are most likely to encounter, it is what UnixFS uses under the hood.
Using the DAG-PB codec as a Blob
Using the DAG-PB codec as an ArrayBuffer
Using the DAG-PB codec as a stream
Content-Type
When fetching
DAG-PB
data, the content type will be set toapplication/octet-stream
unless a custom content-type parser is configured.JSON
The JSON codec is a very simple codec, a block parseable with this codec is a JSON string encoded into a
Uint8Array
.Using the JSON codec
Content-Type
When the
JSON
codec is encountered, theContent-Type
header of the response will be set toapplication/json
.DAG-JSON
DAG-JSON expands on the
JSON
codec, adding the ability to contain CIDs which act as links to other blocks, and byte arrays.CID
s and byte arrays are represented using special object structures with a single"/"
property.Using
DAG-JSON
has two important caveats:JSON
structure cannot contain an object with only a"/"
property, as it will be interpreted as a special type.JSON
has no technical limit on number sizes,DAG-JSON
also allows numbers larger thanNumber.MAX_SAFE_INTEGER
. JavaScript requires use ofBigInt
s to represent numbers larger than this, andJSON.parse
does not support them, so precision will be lost.Otherwise this codec follows the same rules as the
JSON
codec.Using the DAG-JSON codec
Content-Type
When the
DAG-JSON
codec is encountered in the requested CID, theContent-Type
header of the response will be set toapplication/json
.DAG-JSON
data can be parsed from the response by using the.json()
function, which will returnCID
s/byte arrays as plain{ "/": ... }
objects:Alternatively, it can be decoded using the
@ipld/dag-json
module and the.arrayBuffer()
method, in which case you will getCID
objects andUint8Array
s:DAG-CBOR
DAG-CBOR uses the Concise Binary Object Representation format for serialization instead of JSON.
This supports more data types in a safer way than JSON and is smaller on the wire to boot so is usually preferable to JSON or DAG-JSON.
Content-Type
Not all data types supported by
DAG-CBOR
can be successfully turned into JSON and back into the same binary form.When a decoded block can be round-tripped to JSON, the
Content-Type
will be set toapplication/json
. In this case the.json()
method on theResponse
object can be used to obtain an object representation of the response.When it cannot, the
Content-Type
will beapplication/octet-stream
- in this case the@ipld/dag-json
module must be used to deserialize the return value from.arrayBuffer()
.Detecting JSON-safe DAG-CBOR
If the
Content-Type
header of the response isapplication/json
, the.json()
method may be used to access the response body in object form, otherwise the.arrayBuffer()
method must be used to decode the raw bytes using the@ipld/dag-cbor
module.The
Accept
headerThe
Accept
header can be passed to override certain response processing, or to ensure that the finalContent-Type
of the response is the one that is expected.If the final
Content-Type
does not match theAccept
header, or if the content cannot be represented in the format dictated by theAccept
header, or you have configured a custom content type parser, and that parser returns a value that isn't in the accept header, a 406: Not Acceptable response will be returned:It can also be used to skip processing the data from some formats such as
DAG-CBOR
if you wish to handle decoding it yourself:Redirects
If a requested URL contains a path component, that path component resolves to a UnixFS directory, but the URL does not have a trailing slash, one will be added to form a canonical URL for that resource, otherwise the request will be resolved as normal.
It's possible to prevent this behavior and/or handle a redirect manually through use of the redirect option.
Example: Redirect: follow
This is the default value and is what happens if no value is specified.
Example: Redirect: error
This causes a
TypeError
to be thrown if a URL would cause a redirect.Example: Redirect: manual
Manual redirects allow the user to process the redirect. A 301 is returned, and the location to redirect to is available as the "location" response header.
This differs slightly from HTTP fetch which returns an opaque response as the browser itself is expected to process the redirect and hide all details from the user.
Comparison to fetch
This module attempts to act as similarly to the
fetch()
API as possible.The
fetch()
API takes two parameters:Resource argument
This library supports the following methods of fetching web3 content from IPFS:
ipfs://<cidv0>
&ipfs://<cidv1>
ipns://<peerId>
&ipns://<publicKey>
&ipns://<hostUri_Supporting_DnsLink_TxtRecords>
CID.parse('bafy...')
As well as support for pathing & params for items 1 & 2 above according to IPFS - Path Gateway Specification & IPFS - Trustless Gateway Specification. Further refinement of those specifications specifically for web-based scenarios can be found in the Web Pathing Specification IPIP.
If you pass a CID instance, it assumes you want the content for that specific CID only, and does not support pathing or params for that CID.
Options argument
This library does not plan to support the exact Fetch API options object, as some of the arguments don't make sense. Instead, it will only support options necessary to meet IPFS specs related to specifying the resultant shape of desired content.
Some of those header specifications are:
Where possible, options and Helia internals will be automatically configured to the appropriate codec & content type based on the
verified-fetch
configuration andoptions
argument passed.Known Fetch API options that will be supported:
signal
- An AbortSignal that a user can use to abort the request.redirect
- A string that specifies the redirect type. One offollow
,error
, ormanual
. Defaults tofollow
. Best effort to adhere to the Fetch API redirect parameter.headers
- An object of headers to be sent with the request. Best effort to adhere to the Fetch API headers parameter.accept
- A string that specifies the accept header. Relevant values:vnd.ipld.raw
. (default)vnd.ipld.car
vnd.ipfs.ipns-record
method
- A string that specifies the HTTP method to use for the request. Defaults toGET
. Best effort to adhere to the Fetch API method parameter.body
- An object that specifies the body of the request. Best effort to adhere to the Fetch API body parameter.cache
- Will basically act asforce-cache
for the request. Best effort to adhere to the Fetch API cache parameter.Non-Fetch API options that will be supported:
onProgress
- Similar to HeliaonProgress
options, this will be a function that will be called with a progress event. Supported progress events are:helia:verified-fetch:error
- An error occurred during the request.helia:verified-fetch:request:start
- The request has been senthelia:verified-fetch:request:complete
- The request has been senthelia:verified-fetch:request:error
- An error occurred during the request.helia:verified-fetch:request:abort
- The request was aborted prior to completion.helia:verified-fetch:response:start
- The initial HTTP Response headers have been set, and response stream is started.helia:verified-fetch:response:complete
- The response stream has completed.helia:verified-fetch:response:error
- An error occurred while building the response.Some in-flight specs (IPIPs) that will affect the options object this library supports in the future can be seen at https://specs.ipfs.tech/ipips, a subset are:
Response types
This library's purpose is to return reasonably representable content from IPFS. In other words, fetching content is intended for leaf-node content -- such as images/videos/audio & other assets, or other IPLD content (with link) -- that can be represented by https://developer.mozilla.org/en-US/docs/Web/API/Response#instance_methods. The content type you receive back will depend upon the CID you request as well as the
Accept
header value you provide.All content we retrieve from the IPFS network is obtained via an AsyncIterable, and will be set as the body of the HTTP Response via a
ReadableStream
or other efficient method that avoids loading the entire response into memory or getting the entire response from the network before returning a response to the user.If your content doesn't have a mime-type or an IPFS spec, this library will not support it, but you can use the
helia
library directly for those use cases. See Unsupported response types for more information.Handling response types
For handling responses we want to follow conventions/abstractions from Fetch API where possible:
.json()
to get a JSON object..blob()
or.arrayBuffer()
to get the raw bytes..text()
response.body.getReader()
to get aReadableStream
.Unsupported response types
aplication/vnd.ipld.car
or use thehelia
library directly for this use case.Response headers
This library will set the HTTP Response headers to the appropriate values for the content type according to the appropriate IPFS Specifications.
Some known header specifications:
Server Timing headers
By default, we do not include Server Timing headers in responses. If you want to include them, you can pass an
withServerTiming
option to thecreateVerifiedFetch
function to include them in all future responses. You can also pass thewithServerTiming
option to each fetch call to include them only for that specific response.See PR where this was added, https://github.com/ipfs/helia-verified-fetch/pull/164, for more information.
Possible Scenarios that could cause confusion
Attempting to fetch the CID for content that does not make sense
If you request
bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze
, which points to the root of the en.wikipedia.org mirror, a response object does not make sense.Errors
Known Errors that can be thrown:
TypeError
- If the resource argument is not a string, CID, or CID string.TypeError
- If the options argument is passed and not an object.TypeError
- If the options argument is passed and is malformed.AbortError
- If the content request is aborted due to user aborting provided AbortSignal. Note that this is aAbortError
from@libp2p/interface
and not the standardAbortError
from the Fetch API.Extensibility
Verified‑fetch can now be extended to alter how it handles requests by using plugins. Plugins are classes that extend the
BasePlugin
class and implement theVerifiedFetchPlugin
interface. They are instantiated withPluginOptions
when theVerifiedFetch
class is created.Plugin Interface
Each plugin must implement two methods:
canHandle(context: PluginContext): boolean
Inspects the currentPluginContext
(which includes the CID, path, query, accept header, etc.) and returnstrue
if the plugin can operate on the current state of the request.handle(context: PluginContext): Promise<Response | null>
Performs the plugin’s work. It may:Response
: This stops the pipeline immediately.null
: This indicates that the plugin has only partially processed the request (for example, by performing path walking or decoding) and the pipeline should continue.PluginError
: This logs a non-fatal error and continues the pipeline.PluginFatalError
: This logs a fatal error and stops the pipeline immediately.Plugin Pipeline
Plugins are executed in a chain (a plugin pipeline):
Initialization:
VerifiedFetch
class is instantiated with a list of plugins.fetch
method, the resource and options are parsed to create a mutablePluginContext
object.Pipeline Execution:
canHandle()
method.true
forcanHandle()
are invoked in sequence.Response
or throws aPluginFatalError
, the pipeline immediately stops and that response is returned.null
, it may have updated the context (for example, by performing path walking), other plugins that said theycanHandle
will run.context.modified
) and no final response is produced after iterating through all plugins, the pipeline exits and a default “Not Supported” response is returned.Diagram of the Plugin Pipeline:
Please see the original discussion on extensibility in Issue #167.
Non-default plugins provided by this library
dir-index-html-plugin
This plugin is used to serve dag-pb/unixfs without an
index.html
child as HTML directory listing of the content requested.dag-cbor-html-preview-plugin
This plugin is used to serve the requested dag-cbor object as HTML when the Accept header includes
text/html
.Example: Using the plugins
Extending Verified‑Fetch with Custom Plugins
To add your own plugin:
Extend the BasePlugin:
Create a new class that extends
BasePlugin
and implements:canHandle(context: PluginContext): boolean
handle(context: PluginContext): Promise<Response | null>
Example: custom plugin
Integrate Your Plugin:
Add your custom plugin to Verified‑Fetch’s plugin list when instantiating Verified‑Fetch:
Example: Integrate custom plugin
Error Handling in the Plugin Pipeline
Verified‑Fetch distinguishes between two types of errors thrown by plugins:
PluginError (Non‑Fatal):
PluginError
, the error is logged and the pipeline continues with the next plugin.PluginFatalError (Fatal):
PluginFatalError
, the pipeline immediately terminates and the provided error response is returned.Example: Plugin error Handling
How the Plugin Pipeline Works
Shared Context: A mutable
PluginContext
is created for each request. It includes the parsed CID, path, query parameters, accept header, and any other metadata. Plugins can update this context as they perform partial work (for example, by doing path walking or decoding).Iterative Processing: The pipeline repeatedly checks which plugins can currently handle the request by calling
canHandle(context)
.null
, allowing subsequent passes by other plugins.Response
and the pipeline terminates.No Strict Ordering: Plugins are invoked based solely on whether they can handle the current state of the context. This means you do not have to specify a rigid order, each plugin simply checks the context and acts if appropriate.
Error Handling:
PluginError
is considered non‑fatal and is logged, allowing the pipeline to continue.PluginFatalError
immediately stops the pipeline and returns the error response.For a detailed explanation of the pipeline, please refer to the discussion in Issue #167.