Instead of blocking scraping, can't we just put a limit on it? #19

New issue

Open

opened 2024-03-12 13:42:41 +13:00 by fiatjaf · 2 comments

fiatjaf commented

2024-03-12 13:42:41 +13:00

(Migrated from github.com)

The allow_scrape_if_limited_to requires a limit to be specified, but we'll never get clients to adhere to such requirements.

Instead why not have an option to impose a hard limit on scraping queries (or on all queries even) So if someone requests all our events we can just return the last 50 or something.

The `allow_scrape_if_limited_to` requires a `limit` to be specified, but we'll never get clients to adhere to such requirements. Instead why not have an option to impose a hard limit on scraping queries (or on all queries even) So if someone requests all our events we can just return the last 50 or something.

mikedilger commented

2024-03-14 13:55:04 +13:00

(Migrated from github.com)

Interesting. I'm torn though.

I don't want to return results that I know are incorrect (incomplete), I'd rather return an error saying we can't fulfill the request.

Interesting. I'm torn though. I don't want to return results that I know are incorrect (incomplete), I'd rather return an error saying we can't fulfill the request.

mikedilger commented

2024-04-10 13:59:37 +12:00

(Migrated from github.com)

Maybe we could have a config that specifies a max age, and then allow scrapes (backwards in time) up to the max age.