We identified rate limiting as something which is performed on one of two actors. A malicious actor who is trying to cause harm to and even destroy a service in order to cause damage to a business, or the non-malicious user, who may just use the refresh button to frequently, or a back-end which can only support a low amount of load and needs to be protected from over use. With these users in mind, lets explore rate limiting as its possible on ISAM in version 9.0.6.0.

The Web Reverse proxy on ISAM implements a simple token bucket algorithm, by which incoming requests have are processed for interesting information, this information is then extracted and computed into a token. This token used to identify a bucket which keeps track of a count and timestamp. When requests which contain the same information come in, the same token is computed to identify the bucket and the counter of the bucket incremented. When the limit for this bucket is reached, the request will be rate limited. The bucket is only emptied when the timestamp of the bucket is less than the current time by a defined amount.  The maximum value of the buckets counter, and the time between refreshes are identified as capacity, and interval.

In order to perform rate limiting information in the incoming request must be inspected. This includes HTTP Method, URL, Headers, Cookies, query string parameters, the users session(if present) and the IP address of the incoming connection. First, the method and request URL are used in order to identify which rate limiting policies should apply to this request, then the selected rate limiting policy is used to extract the parts of the request which are defined as interesting by the policy and used to compute a hash. This has is the identifier of the bucket. These parts are the matching and tokenizing criteria of a rate limiting mechanism.

We now have the pieces which define identifying requests and allocating them to buckets which we know to have a capacity and an interval so all that is left is to inform the client that they have been rate limited. This can happen in one of three ways, the connection can be abruptly closed, a 429 and a template page returned, or the request URL can be re-written, to instead route the request to a different resource. This final method is different to the other two, but serves a special and specific purpose, this is to mislead a malicious client, rather than rejecting the request, they can instead be routed to another service, which might spoof the service they’re trying to attack, but instead do nothing, or slow down the processing of their request. We define these different behaviours as reactions to being rate limited.

We are now left with three groups of concepts:

• Matching and tokenizing criteria
• Capacity and interval
• Reaction

Now that we’ve established rate limiting concepts. Lets look into the processing flow and policy as it exists in the reverse proxy.

Rate Limiting Processing

When a request is received the following steps are taken:

1. The URL and HTTP method are checked against all of the configured policies. If a match occurs that policy is invoked. All matching policies will be applied with one caveat – if a policy applies a reaction, that will be returned immediately with no further policy evaluated.
2. The matching policy evaluates the tokenizing criteria. This will extract out all the information from the request as configured by the policy, and build the unique token from it.
3. Using the unique token a bucket is retrieved. If no bucket is found it is created.
4. The bucket is checked to see if it needs to be emptied, or if it is full.
5. If the bucket is full, then the reaction as per policy is invoked.

These steps can be easily mapped back to the concepts established above. 1 & 2 are matching and tokenizing. 3 & 4 are capacity and interval, and 5 is the reaction.

Rate limiting occurs early in the reverse proxy processing of a request. Before authentication, authorization, http transformations, and the backend being called.

Rate Limiting Policy

Rate limiting on ISAM is a YAML file, containing the policy necessary to identify, track and react to duplicate requests in need of rate limiting. ISAM ships 3 rules out of the box. These rules are:

• Limiting use of an OAuth Bearer token. This mitigates against OAuth clients who are abusing the access granted to the on behalf of a user
• Tracking attempts to log into the reverse proxy by way of forms. To mitigate against a brute force attack of a users password
• Rate limiting any access to the reverse proxy based on IP. To stop basic request spam attacks.

Lets explore the policy for limiting login to the reverse proxy. The complete policy, without any comments is:

url: /pkmslogin.form
method: 
  - POST

ip: true

capacity: 5
interval: 60

reaction: TEMPLATE

Each key of this policy relates back to the concepts we explored above:

url – This is a pattern to match on for the URL. The matching performed is case insensitive and can include ‘?’ and “*” characters to match on wildcards. Only if the request url matches this pattern will this policy be evaluated on a given request.

method – this is the method necessary to match on this policy. If the method does not match then this policy will not be considered relevant to the incoming request.

Only once method and URL have been matched for a given request will the policy be evaluated any further.

ip – the IP address of the incoming request may or may not be included when identifying a request, in this instance it would be.

The matching criteria (method and URL), are combined with the tokenizing criteria which in this instance is just IP are now computed into a lookup key used to reference the counter and time.

The above policy only includes one of the available tokenizing criteria, IP address. There are 4 other tokenizing criteria

1. Header – Useful for when the client is including identifying information like an authorization header, or if there is a service in front of the reverse proxy which adds information
2. Cookie – Allows you to easily rate limit a single session via the PD-S-SESSION-ID to limit a single session. Also allows a cookie from a load balancer or back-end to be used.
3. Query Parameter – allows identifying resources based on query argument, such as a resource identifier in a GET request.
4. Credential Attribute*

*Credential attributes can only be used when there is an authenticated session. This also has rate limiting occur later in the processing of the request. For example, this moves rate limiting to occur after HTTP transformations.

Rate Limiting on Credential Attribute

Rate limiting on a credential attribute is a very useful capability. Because the authentication method, or back end application is able to influence the contents of the users credential this allows for context aware rate limiting policy to be authored.

A good example of how the authentication method can include useful information to rate limit on is using the oauth-auth capability of a reverse proxy. When a client is authenticated with oauth-auth their credential will contain an attribute oauth_token_client_id, which is the client_id to which the token was issued. This allows you to write a rate limiting policy that stops that specific client from over using a service. Another oauth use case would be to limit the use of a grant via the state-id. All that is necessary to achieve this would be to include the state-id as part of a resource response from the authorization server. This would stop a specific grant from being abused, regardless of if the access_token is refreshed. Other token attributes can also be returned and used in rate limiting policy.

These are just one application specific example of how rate limiting can be applied. Any extended attributes from an LDAP or which an EAI injects to the credential can be used to enforce a limit on resource usage.

Authoring Rate Limiting Policy

ISAM provides a rate limiting template when you create a new policy. This highlights the different sections we’ve explored about. There is a UI which can be used to author rate limiting policy.  See Secure Web Settings -> Global Settings -> Rate Limiting.

Attaching Rate Limiting Policy

Rate limiting policy after authored, must be attached to a reverse proxy. This is done via the [rate-limiting] stanza. This stanza takes any number of entries with the name policy. For example:

[rate-limiting]
policy = LimitLoginPolicy.yaml
policy = MyCustomPolicy.yaml

The available policies will be populated as comments in the reverse proxy configuration file for ease of use.

Rate Limiting in Action

In the prior steps two rate limiting policies were attached. One custom policy to limit access to all resources to 100 times an hour, and a second policy which is provided out of the box, to limit login attempts to 5 times in 60 seconds. This video shows a user attempting to log in 6 times and what happens on the final attempt.

Rate  Limiting Trace

Rate limiting can be traced like any other reverse proxy component. The trace specification for rate limiting is pdweb.http.ratelimit.

Rate limiting trace looks like the following (with the leading information stripped for brevity). Comments have been added after the # character to give a description about what each line is describing.

I’m keen to hear what and how you plan on rate limiting, please don’t hesitate to leave comments below or reach out if you’ve got questions about your scenario.