ToOverlordMessage::LoadMoreCurrentFeed
  Overlord::load_more_general_feed()
    1. start = GLOBALS.feed.load_more_general_feed()
    		setting 'feed_chunk'
    2. TempSubscribeGeneralFeedChunk { pubkeys, start }

  Overlord::load_more_person_feed(pubkey)
    1. start = GLOBALS.feed.load_more_person_feed()
    		setting 'person_feed_chunk'
    2. TempSubscribePersonFeedChunk { pubkey, start }

  Overlord::load_more_inbox_feed()
    1. start = GLOBALS.feed.load_more_inbox_feed()
    		setting 'replies_chunk'
    2. TempSubscribeInboxFeedChunk(start)



Subscriptions should have:
	until = earliest date of current feed content
    limit = 50 (for example)

	not 'start'


Feed needs:
	Replace
    	general_feed_start
        person_feed_start
        inbox_feed_start
    With
    	general_feed_count
        person_feed_count
        inbox_feed_count

 	Load feed messages by LIMIT not by SINCE
	it uses storage - find_events()

	> It uses find_event_ids() from a since.


STORAGE find_events NEEDS A LIMIT (applied before loading events)

STORAGE find_event_ids NEEDS A LIMIT

STORAGE find_ek_c_events NEEDS A LIMIT (it will be sorted in reverse order)

Notes to myself: ``` ToOverlordMessage::LoadMoreCurrentFeed Overlord::load_more_general_feed() 1. start = GLOBALS.feed.load_more_general_feed() setting 'feed_chunk' 2. TempSubscribeGeneralFeedChunk { pubkeys, start } Overlord::load_more_person_feed(pubkey) 1. start = GLOBALS.feed.load_more_person_feed() setting 'person_feed_chunk' 2. TempSubscribePersonFeedChunk { pubkey, start } Overlord::load_more_inbox_feed() 1. start = GLOBALS.feed.load_more_inbox_feed() setting 'replies_chunk' 2. TempSubscribeInboxFeedChunk(start) Subscriptions should have: until = earliest date of current feed content limit = 50 (for example) not 'start' Feed needs: Replace general_feed_start person_feed_start inbox_feed_start With general_feed_count person_feed_count inbox_feed_count Load feed messages by LIMIT not by SINCE it uses storage - find_events() > It uses find_event_ids() from a since. STORAGE find_events NEEDS A LIMIT (applied before loading events) STORAGE find_event_ids NEEDS A LIMIT STORAGE find_ek_c_events NEEDS A LIMIT (it will be sorted in reverse order) ```

mikedilger commented

2024-03-04 11:48:13 +13:00

(Migrated from github.com)

I may do this:

Add a pk_c index
Replace the ek_pk index with an ek_pk_c index
Support 'since' and 'limit' in all the per-index finds, and then
Rewrite the general find, supporting both 'since' and 'limit'

I may do this: * Add a pk_c index * Replace the ek_pk index with an ek_pk_c index * Support 'since' and 'limit' in all the per-index finds, and then * Rewrite the general find, supporting both 'since' and 'limit'

mikedilger commented

2024-04-06 11:30:07 +13:00

(Migrated from github.com)

I have replaced the old indexes:

ek_pk (keys are kind+author, values are dup ids)
ek_c (keys are kind+created_at values are dup ids)

With new indexes

akci (keys are author+kind+createdat+id, values are null)
kci (keys are kind+createdat+id. values are null)

Based on experience with chorus and analysis of usage in gossip.

This allowed me to write a "find_event_by_filter" function that respects since, until and limit. Because the indexing is incomplete, we cannot search by any tag, and we cannot search by any kind, nor can we search by ids (use events directly for that), but we can search for everything we need and especially with regard to this issue, we can use limits now.

I have replaced the old indexes: - `ek_pk` (keys are kind+author, values are dup ids) - `ek_c` (keys are kind+created_at values are dup ids) With new indexes - `akci` (keys are author+kind+createdat+id, values are null) - `kci` (keys are kind+createdat+id. values are null) Based on experience with chorus and analysis of usage in gossip. This allowed me to write a "find_event_by_filter" function that respects since, until and limit. Because the indexing is incomplete, we cannot search by any tag, and we cannot search by any kind, nor can we search by ids (use events directly for that), but we can search for everything we need and especially with regard to this issue, we can use limits now.

mikedilger commented

2024-06-21 09:36:12 +12:00

(Migrated from github.com)

this was done a week or so ago.