Why not use memmap2?

norcalli commented

2023-10-04 16:17:07 +13:00

(Migrated from github.com)

memmap2 is the defacto cross platform mmap library. I noticed a lot of the code in the crate is duplicating work that it already does, down to the mutable deref [u8]. It seems like it would save a lot of complexity/code.

It would also potentially have caught another issue we have where although this crate says it's for unix, macos doesn't have mremap, as that is linux only.

Emulating mremap here isn't difficult since we pass maymove, so we could try to remap at the old address and create a new mapping otherwise, or we could skip the old address part if it's not that important.

memmap2 is the defacto cross platform mmap library. I noticed a lot of the code in the crate is duplicating work that it already does, down to the mutable deref [u8]. It seems like it would save a lot of complexity/code. It would also potentially have caught another issue we have where although this crate says it's for `unix`, macos doesn't have `mremap`, as that is linux only. Emulating mremap here isn't difficult since we pass maymove, so we could try to remap at the old address and create a new mapping otherwise, or we could skip the old address part if it's not that important.

mikedilger commented

2023-10-04 21:21:39 +13:00

(Migrated from github.com)

That information is useful, about mremap.

I don't think I could use and extend memmap2, I had to cut open it's innards to do what this crate does: let multiple threads Deref while a writer is appending. If I'm wrong about that, I'd appreciate being corrected in code.

All the commits happened over 5 days quite a while ago now, so I don't remember too many specifics.

That information is useful, about mremap. I don't think I could use and extend memmap2, I had to cut open it's innards to do what this crate does: let multiple threads Deref<u8> while a writer is appending. If I'm wrong about that, I'd appreciate being corrected in code. All the commits happened over 5 days quite a while ago now, so I don't remember too many specifics.

norcalli commented

2023-10-04 21:26:39 +13:00

(Migrated from github.com)

I see... I think that makes sense since in append only mode, we don't need &mut self, so that does makes sense. Could probably save some overlap by doing that on top of https://docs.rs/memmap2/latest/memmap2/struct.MmapRaw.html though, with the as_ptr and as_mut_ptr, but I do see how it is less of a boon now.

I see... I think that makes sense since in append only mode, we don't need `&mut self`, so that does makes sense. Could probably save some overlap by doing that on top of `https://docs.rs/memmap2/latest/memmap2/struct.MmapRaw.html` though, with the `as_ptr` and `as_mut_ptr`, but I do see how it is less of a boon now.

mikedilger commented

2023-10-04 21:28:35 +13:00

(Migrated from github.com)

I never got around to using this crate myself. When I do get back around to what it was for, I'll definitely take some time to see if I can't leverage memmap2 better.

norcalli commented

2023-10-04 21:29:51 +13:00

(Migrated from github.com)

Sounds good. I might take a stab at making an MR for it since I'm actually using it in production if you're interested.

At the very least, I'd want to provide an mremap workaround

Sounds good. I might take a stab at making an MR for it since I'm actually using it in production if you're interested. At the very least, I'd want to provide an mremap workaround

mikedilger commented

2023-10-04 21:30:42 +13:00

(Migrated from github.com)

Yeah, I'd like that

Rows
Columns

Why not use memmap2? #1