]> IIT-CNR/Registro.it

Via Moruzzi,1 Pisa IT 56124 mario.loffredo@iit.cnr.it http://www.iit.cnr.it

IIT-CNR/Registro.it

Via Moruzzi,1 Pisa IT 56124 maurizio.martinelli@iit.cnr.it http://www.iit.cnr.it

Verisign Labs

12061 Bluemont Way Reston VA 20190 USA shollenbeck@verisign.com https://www.verisignlabs.com/

Applications and Real-Time Registration Protocols Extensions RDAP Sorting Paging The Registration Data Access Protocol (RDAP) does not include core functionality for clients to provide sorting and paging (subsetting) parameters for control of large result sets. This omission can lead to unpredictable server processing of queries and client processing of responses. This unpredictability can be greatly reduced if clients can provide servers with their preferences for managing response values. This document describes RDAP query extensions that allow clients to specify their preferences for sorting and paging result sets.

The availability of functionality for result sorting and paging provides benefits to both clients and servers in the implementation of RESTful services . These benefits include: Reducing the server response bandwidth requirements; Improving server response time; Improvement in query precision and, consequently, obtaining more reliable results; Decreasing server query processing load. Approaches to implementing features for result sorting and paging can be grouped into two main categories: Sorting and paging are implemented through the introduction of additional parameters in the query string (i.e. ODATA protocol ); Information related to the number of results and the specific portion of the result set to be returned, in addition to a set of ready-made links for the result set scrolling, are inserted in the HTTP header of the request/response. However, there are some drawbacks associated with use of the HTTP header. First, the header properties cannot be set directly from a web browser. Moreover, in an HTTP session, the information on the status (i.e. the session identifier) is usually inserted in the header or in the cookies, while the information on the resource identification or the search type is included in the query string. The second approach is therefore not compliant with the HTTP standard . As a result, this document describes a specification based on use of query parameters. Currently the RDAP protocol defines two query types: lookup: the server returns only one object; search: the server returns a collection of objects. While the lookup query does not produce issues in the management of large result sets, the search query can potentially generate a large result set that could be truncated according to the limits of the server. In addition, it is not possible to obtain the total number of the objects found that might be returned in a search query response . Lastly, there is no mechanism to specify sort criteria to return the most relevant objects at the beginning of the result set. The protocol described in this specification extends RDAP query capabilities to enable result sorting and paging, by adding four new query parameters that can be applied to RDAP search path segments. The service is implemented using the Hypertext Transfer Protocol (HTTP) and the conventions described in RFC 7480 . The implementation of these parameters is technically feasible, as operators for counting, sorting and paging rows are currently supported by the major RDBMSs. Impact on the current state of RDAP implementation is estimated to be quite low.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

These new query parameters are OPTIONAL extensions of path segments defined in RFC 7482 . The query parameters are as follows: "count": a boolean value that allows a client to request the total number of objects found (that due to truncation can be different from the number of returned objects); "sortby": a string value that allows a client to request a specific sort order for the result set; "limit" and "offset": numeric values that allow a client to request a specific portion of the entire result set and to use the new "paging_links" property to provide a ready-made reference to the next page of the result set. Augmented Backus-Naur Form (ABNF) is used in the following sections to describe the formal syntax of these new parameters.

Currently the RDAP protocol does not allow a client to determine the total number of the results in a query response when the result set is truncated. This is rather inefficient because the user cannot evaluate the query precision and, at the same time, cannot receive information that could be relevant. The new parameter "count" provides additional functionality () that allows a client to request information from the server that specifies the number of elements found to match a particular search pattern.

The ABNF syntax is the following: count = "count" EQ ( trueValue / falseValue ) trueValue = ("true" / "yes" / "1") falseValue = ("false" / "no" / "0") EQ = "=" A trueValue means that the server MUST provide the total number of the objects in the paging_count property of the response (). A falseValue means that the server MUST NOT provide this number.

{ "rdapConformance": [ "rdap_level_0", "paging_level_0" ], ... "paging_count":"73", "domainSearchResults": [ ... ] }

The RDAP protocol does not provide any capability to specify response value sort criteria. A server could implement a default sorting scheme according to the object class, but this feature is not mandatory and might not meet user requirements. Sorting can be addressed by the client, but this solution is rather inefficient. Sorting and paging using features provided by the DBMS used by the RDAP server could help avoid truncation of relevant results and allow for scrolling the result set using subsequent queries. The "sortby" parameter allows the client to ask the server to sort the results according to the values of one or more properties and according to the sort direction of each property. The ABNF syntax is the following: sortby = "sortby" EQ sortbyItem *( "," sortbyItem ) sortbyItem = property-ref [":" ( "a" / "d" ) ] "a" means that the ascending sort MUST be applied, "d" means that the descending sort MUST be applied. If the sort direction is absent, an ascending sort MUST be applied (). In the sortby ABNF syntax, property-ref represents a reference to a property of an RDAP object. Such a reference could be expressed by using a JSON Path. The JSON Path in a JSON document is equivalent to the XPath in a XML document. For example, the JSON Path to select the value of the ldhName property inside an RDAP domain object is "$.ldhName", where $ identifies the root of the document (DOM). Another way to select a value inside a JSON document is the JSON Pointer . While JSON Path or JSON Pointer are both standard ways to select any value inside JSON data, neither is particularly easy to use (e.g. "$.events[?(@.eventAction='registration')].eventDate" is the JSON Path expression of the registration date in a RDAP domain object). Therefore, this specification provides a definition of property-ref in terms of RDAP properties. However, not all the RDAP properties are suitable to be used in sort criteria, such as: properties providing service information (e.g. links, notices, remarks, etc.); multivalued properties (e.g. status, roles, variants, etc.); properties modeling relationships to other objects (e.g. entities). On the contrary, some properties expressed as values of other properties (e.g. registration date) could be used in such a context. In the following, a list of the proposed properties for sort criteria is presented. The properties are divided in two groups, object common properties and object specific properties. Object common properties. Object common properties are derived from the merge of the "eventAction" and the "eventDate" properties. The following values of sortby are defined: registrationDate reregistrationDate lastChangedDate expirationDate deletionDate reinstantiationDate transferDate lockedDate unlockedDate Object specific properties. With regard to the specific properties, some of them are already defined among the query paths. In the following the list of the proposed sorting properties, grouped by objects, is shown: Domain: ldhName Nameserver: ldhName, ipV4, ipV6. Entity: fn, handle, org, email, tel, country, countryName, locality. In the following, the correspondence between the values of sortby parameter and the RDAP object properties is shown (): Object class sortby value Object property Reference in RFC 7483 Reference in RFC 6350 Searchable objectsCommon propertieseventAction values suffixed by "Date"4.5. DomainldhNameldhName5.3. NameserverldhNameldhName5.2. ipV4v4 ipAddress5.2. ipV6v6 ipAddress5.2. Entityhandlehandle5.1. fnvcard fn5.1.6.2.1 orgvcard org5.1.6.6.4 telvcard tel with type="voice"5.1.6.4.1 emailvcard email5.1.6.4.2 countrycountry code (as given in ISO.3166.1988 ) of the country name in vcard adr5.1.6.3.1 countryNamecountry name in vcard adr5.1.6.3.1 localitylocality in vcard adr5.1.6.3.1 With regard to the definitions in , some further considerations must be made to disambiguate cases where the RDAP object property is multivalued: Even if a nameserver can have multiple IPv4 and IPv6 addresses, the most common configuration includes one address for each IP version. Therefore, the assumption of having a single IPv4 and/or IPv6 value for a nameserver cannot be considered too stringent. With the exception of handle values, all the sorting properties defined for entity objects can be multivalued according to the definition of vCard as given in RFC6350 . When more than a value is reported, sorting can be applied to the preferred value identified by the parameter pref="1". Each RDAP provider MAY define other sorting properties than those shown in this document. (FOR DISCUSSION: how should those properties be defined? Is an IANA registry appropriate?)

An RDAP query could return a response with hundreds of object return values, especially when partial matching is used. For that reason, two parameters addressing result pagination are defined to make responses easier to handle: "limit": means that the server MUST return the first N objects of the result set in the response; "offset": means that the server MUST skip the first N objects and MUST return objects starting from position N+1. The ABNF syntax is the following: EQ = "=" limit = "limit" EQ positive-number offset = "offset" EQ positive-number positive-number = non-zero-digit *digit non-zero-digit = "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / "9" digit = "0" / non-zero-digit When limit and offset are used together, they allow implementation of result pagination. The following examples illustrate requests to return, respectively, the first 5 objects, the set of objects starting from position 6, and first 5 objects starting from position 11 of the result set ().

An RDAP server MAY use the new "paging_links" property to provide a ready-made reference to the next page of the result set ().Examples of additional "rel" values are "first", "last", "prev".

The value constraints for the parameters are defined by their ABNF syntax. Therefore, each request providing an invalid value for a parameter SHOULD obtain an HTTP 400 (Bad Request) response code. The same response SHOULD be returned if the client provides an unsupported value for the sortby parameter. The server can provide a different response when it supports the limit and/or offset parameters and the client submits values that are out of the valid ranges. The possible cases are: If the client submits a value for the limit parameter that is greater than the number of objects to be processed, it is RECOMMENDED that server returns a response including only the processed objects. If the client submits a value for the offset parameter that is greater than the number of objects to be processed, it is RECOMMENDED that server returns an HTTP 404 (Not Found) response code. Optionally, the response MAY include additional information regarding the negative answer in the HTTP entity body.

Servers returning paging_links and paging_count properties MUST include "paging_level_0" in the rdapConformance array of their responses.

The implementation of the new parameters is technically feasible, as operators for counting, sorting and paging are currently supported by the major RDBMSs. In the following, the match between the new defined parameters and the SQL operators is shown (): New query parameter SQL operator countcount(*) query without offset, limit and order by ,, sortbyorder by ,, limitlimit n (in MySql and Postgres ) FETCH FIRST n ROWS ONLY (in Oracle ) offsetoffset m (in Postgres) OFFSET m ROWS (in Oracle) limit + offsetlimit n offset m (in MySql and Postgres) OFFSET m ROWS FETCH NEXT n ROWS ONLY (in Oracle) With regard to Oracle, reports only one of the three methods that can be used to implement limit and offset parameters. The others are described in and . In addition, similar operators are completely or partially supported by the most known NoSQL databases (MongoDB, CouchDB, HBase, Cassandra, Hadoop) so the implementation of the new parameters seems to be practicable by servers working without the use of an RDBMS. FOR DISCUSSION: How might this all work without the use of an RDBMS? Would a server need to maintain state information across queries? If so, what are the implications?

The use of limit and offset operators represents the most common way to implement results pagination. However, when offset has a high value, scrolling the result set could take some time. An alternative approach to offset pagination is the keyset pagination, a.k.a. seek-method (). This method has been taken as the basis for the implementation of a cursor parameter () by two REST API providers (,). The cursor parameter is an opaque URL-safe string representing a logical pointer to the first result of the next page ().

But keyset pagination raises some drawbacks with respect to offset pagination: it needs at least one key field; it does not allow to sort by any field and paginate the results because sorting has to be made on the key field; it does not allow to skip pages because they have to be scrolled in sequential order starting from the initial page; it makes very hard the navigation of the result set in both directions because all comparison and sort operations have to be reversed. Furthermore, in the RDAP context, some additional considerations can be made: an RDAP object is a conceptual aggregation of information collected from more than one data structure (e.g. table) and this makes even harder for the developers the implementation of the seek-method that is already quite difficult. In fact, for example, the entity object can gather information from different data structures (registrars, registrants, contacts, resellers, and so on), each one with its own key field mapping the RDAP entity handle; depending on the number of the page results as well as the number and the complexity of the properties of each RDAP object in the response, the time required by offset to skip the previous pages could be much faster than the processing time needed to build the current page. In fact, RDAP objects can be formed by information belonging to multiple data structures and containing multivalued properties (e.g. arrays) and, therefore, data selection is a time consuming process. This situation occurs even though data selection makes use of indexes; depending on the access levels defined by each RDAP operator, the increase of complexity and the decrease of flexibility of keyset pagination with respect to the offset pagination could be considered impractical. Finally, the keyset pagination is not fully compliant with the additional RDAP capabilities proposed by this document. In fact, the presence of a possible cursor parameter doesn’t seem to be consistent with both the sorting capability and the possibility to implement additional ready-made links besides the classic "next page" link. But, while the provisioning of more paging links can be superfluous, dropping the sorting capability seems quite unreasonable. FOR DISCUSSION: Should RDAP specification reports both offset and cursor parameters and let operators to implement pagination according to their needs, the user access levels, the submitted queries?

NOTE: Please remove this section and the reference to RFC 7942 prior to publication as an RFC. This section records the status of known implementations of the protocol defined by this specification at the time of posting of this Internet-Draft, and is based on a proposal described in RFC 7942 . The description of implementations in this section is intended to assist the IETF in its decision processes in progressing drafts to RFCs. Please note that the listing of any individual implementation here does not imply endorsement by the IETF. Furthermore, no effort has been spent to verify the information presented here that was supplied by IETF contributors. This is not intended as, and must not be construed to be, a catalog of available implementations or their features. Readers are advised to note that other implementations may exist. According to RFC 7942, "this will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. It is up to the individual working groups to use this information as they see fit".

Responsible Organization: Institute of Informatics and Telematics of National Research Council (IIT-CNR)/Registro.it Location: https://rdap.pubtest.nic.it/ Description: This implementation includes support for RDAP queries using data from the public test environment of .it ccTLD. The RDAP server does not implement any security policy because data returned by this server are only for experimental testing purposes. The RDAP server implements both offset and cursor based pagination (the latter only when sortby and offset parameters are not present in the query string). Level of Maturity: This is a "proof of concept" research implementation. Coverage: This implementation includes all of the features described in this specification. Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it

FOR DISCUSSION: An IANA-maintained registry could be used to document all of the properties that could be used as sort criteria. Such a registry could also be used to identify the properties for RDAP query filtering capabilities.

Entries in this registry could contain the following: Object class name: the class name of the object the property belongs to (as defined in ); Property reference: the reference to the property; JSON Path: the JSON path of the property inside the object (as defined in ); Published specification: RFC number, bibliographical reference, or URL to a permanent and readily available specification; Contact: the names and email addresses of individuals to contact regarding this registry entry; Intended usage: brief reasons for this registry entry (as defined in ). In the following, an example of the entry about the reference "ipV4" is reported: Object class name: Nameserver Property reference: ipV4 JSON Path: $.ipAddresses.v4[0] Published specification: https://www.ietf.org/id/draft-loffredo-regext-rdap-sorting-and-paging-01.txt Contact: mario.loffredo@iit.cnr.it Intended usage: The entry is about the reference to the v4 ip address within a nameserver object.

Security services for the operations specified in this document are described in RFC 7481 . Search query typically requires more server resources (such as memory, CPU cycles, and network bandwidth) when compared to lookup query. This increases the risk of server resource exhaustion and subsequent denial of service due to abuse. This risk can be mitigated by either restricting search functionality and limiting the rate of search requests. Servers can also reduce their load by truncating the results in the response. However, this last security policy can result in a higher inefficiency if the RDAP server does not provide any functionality to return the truncated results. The new parameters presented in this document provide the RDAP operators with a way to implement a secure server without penalizing its efficiency. The "count" parameter gives the user a measure to evaluate the query precision and, at the same time, return a significant information. The "sortby" parameter allows the user to obtain the most relevant information at the beginning of the result set. In both cases, the user doesn't need to submit further unnecessary search requests. Finally, the "limit" and "offset" parameters enable the user to scroll the result set by submitting a sequence of sustainable queries according to the server limits.

The authors would like to acknowledge Brian Mountford for his contribution to the development of this document.

&ISO.3166.1988; &RFC2119; &RFC5234; &RFC5266; &RFC5988; &RFC6350; &RFC7159; &RFC7230; &RFC7480; &RFC7481; &RFC7482; &RFC7483; facebook.com twitter.com mysql.com mysql.com mysql.com Oracle Corporation Oracle Corporation Oracle Corporation Oracle Corporation Oracle Corporation postgresql.org postgresql.org postgresql.org &RFC6901; &RFC7942; EverSQL.com &W3C.CR-xpath-31-20161213;

Initial version. Added the paragraph "Considerations about Paging Implementation" to "Implementation Considerations" section. Added "Implementation Status" section. Added acknowledgements. Renamed the property reporting the paging links