class VmCowPages

Define the lock retrieval functions differently depending on whether we should be returning a

local lock instance, or the common one in the hierarchy_state_ptr. Due to the TA_RET_CAP

statements we cannot perform |if constexpr| or equivalent indirection in the function body, and

must have two completely different function definitions.

In the absence of a local lock it is assumed, and enforced in vm_object_lock.h, that there is a

shared lock in the hierarchy state. If there is both a local and a shared lock then the local

lock is to be used for the improved lock tracking.

Returns the size in bytes of this cow pages range. This will always be a multiple of the page

size.

Returns whether this cow pages node is ultimately backed by a user pager to fulfill initial

content, and not zero pages. Contiguous VMOs have page_source_ set, but are not pager backed

in this sense.

This should only be used to report to user mode whether a VMO is user-pager backed, not for any

other purpose.

Returns whether the root of the cow pages hierarchy has non-null page_source_.

Helper function for CowPage cloning methods. Returns any options that should be passed down to

the child.

can_borrow() returns true if the VmCowPages is capable of borrowing pages, but whether

the VmCowPages should actually borrow pages also depends on a borrowing-site-specific flag that

the caller is responsible for checking (in addition to checking can_borrow()). Only if

both are true should the caller actually borrow at the caller's specific potential borrowing

site.

Aside from the general borrowing in the PhysicalPageBorrowingConfig being turned on and

off, the ability to borrow is constant over the lifetime of the VmCowPages.

In addition to whether a VmCowPages is allowed, for correctness reasons, to borrow pages there

are other, potentially variable, factors that influence whether it's considered a good idea for

this VmCowPages to borrow pages. In particular it's possible for this to change over the

lifetime of the VmCowPages.

Returns whether this cow pages node is dirty tracked.

If true this node, and all nodes in this hierarchy, are using parent content markers to

indicate when a leaf node may need to walk up the tree to find content.

When parent content markers are in use an empty page list slot in a leaf node means that there

is *no* visible parent content above, and the parent hierarchy does not have to be searched.

For memory efficiency, and because it would be redundant, parent content markers are never

placed in the hidden nodes, only leaf nodes.

The presence of a parent content marker in a leaf node indicates that there *might* be content

in a parent node and that a tree walk *must* be performed to search for it. The reason for

spurious parent content markers is that zero page deduplication could happen on hidden nodes,

which could remove the content, but leave the parent content markers in the leaf nodes. These

parent content markers are redundant and could be cleaned up.

Use of parent content markers is just the inverse of having a page source, since if there is a

page source we always have to go to it for content as the zero page cannot be assumed. Although

some page sources do supply zero content (physical page provider for contiguous VMOs),

optimizing this check for that is redundant since such page sources do not support

copy-on-write, and so never have children to begin with.

Indicates whether this node can have parent content markers placed in it. This is just checking

if it is both a leaf node, and the tree overall can have parent content markers.

Note that even if this is false, if |tree_has_parent_content_markers| is true then reasoning

may need to be done about parent content markers.

The modified state is only supported for root pager-backed VMOs, and will get queried (and

possibly reset) on the next QueryPagerVmoStatsLocked() call. Although the modified state is

only tracked for the root VMO.

See description on |pinned_page_count_| for meaning.

Sets the VmObjectPaged backlink for this copy-on-write node.

Currently it is assumed that all nodes always have backlinks with the 1:1 hierarchy mapping,

unless this is a hidden node.

Query pager VMO |stats|, and reset them too if |reset| is set to true.

Exposed for testing.

Helper to free |pages| to the PMM. This function will also try to invoke FreePages() on the

backing page source if it supports it. Given the allowance of freeing pages from any object in

the hierarchy, but the page source only being on the root, it is a requirement (and checked on

clone creation), that if a page source is handling free then it may not have CoW children.

There is also an equivalent assumption that if the page source is handling free, then the page

source will be supplying all the pages and this object must never allocate directly from the

PMM.

Callers should avoid calling pmm_free() directly from inside VmCowPages, and instead should use

this helper.

Helper to free |pages| to the PMM. This function will also try to invoke FreePages() on the

backing page source if it supports it. Given the allowance of freeing pages from any object in

the hierarchy, but the page source only being on the root, it is a requirement (and checked on

clone creation), that if a page source is handling free then it may not have CoW children.

There is also an equivalent assumption that if the page source is handling free, then the page

source will be supplying all the pages and this object must never allocate directly from the

PMM.

Callers should avoid calling pmm_free_page() directly from inside VmCowPages, and instead

should use this helper.

Returns an appropriate PmmOptDelayReuse value for use when freeing pages from this object.

Creates a copy-on-write clone with the desired parameters. This can fail due to various

internal states not being correct.

VmCowPages are initially created in the Init state and need to be transitioned to Alive prior

to being used. This is exposed for VmObjectPaged to call after ensuring that creation is

successful, i.e. after it can guarantee that it will transition this cow pages to Dead prior to

it being destroyed.

Resizes the range of this cow pages. |size| must be a multiple of the page size.

See VmObject::Lookup

See VmObject::TakePages

May return ZX_ERR_SHOULD_WAIT if the |page_request| is filled out and needs waiting on. In this

case |taken_len| might be populated with a value less than |len|.

|taken_len| is always filled with the amount of |len| that has been processed to allow for

gradual progress of calls. Will always be equal to |len| if ZX_OK is returned. Similarly the

|splice_offset| indicates the base offset in |pages| where the content should be inserted.

Pre-process a VmPageSpliceList for supply.

If there is a page source, references are converted to pages. When allocating pages from

references, ZX_ERR_SHOULD_WAIT can be returned which is managed by this function.

TODO(sagebarred): handle InitializeVmPage from PhysicalPageProvider here.

See VmObject::SupplyPages

Should never return ZX_ERR_SHOULD_WAIT, waiting on page requests is managed by

ProcessPagesForSupply.

See VmObject::FailPageRequests

See VmObject::DirtyPages. |page_request| is required to support delayed PMM allocations; if

ZX_ERR_SHOULD_WAIT is returned the caller should wait on |page_request|. |alloc_list| will hold

any pages that were allocated but not used in case of delayed PMM allocations, so that it can

be reused across multiple successive calls whilst ensuring forward progress.

See VmObject::EnumerateDirtyRanges

See VmObject::WritebackBegin

See VmObject::WritebackEnd

Tries to prepare the range [offset, offset + len) for writing by marking pages dirty or

verifying that they are already dirty. It is possible for only some or none of the pages in the

range to be dirtied at the end of this call. |dirty_len_out| will return the (page-aligned)

length starting at |offset| that contains dirty pages, either already dirty before making the

call or dirtied during the call. In other words, the range [offset, offset + dirty_len_out)

will be dirty when this call returns, i.e. prepared for the write to proceed, where

|dirty_len_out|

<

= |len|.

If the specified range starts with pages that are not already dirty and need to request the

page source before transitioning to dirty, a DIRTY page request will be forwarded to the page

source. In this case |dirty_len_out| will be set to 0, ZX_ERR_SHOULD_WAIT will be returned and

the caller should wait on |page_request|. If no page requests need to be generated, i.e. we

could find some pages that are already dirty at the start of the range, or if the VMO does not

require dirty transitions to be trapped, ZX_OK is returned.

|offset| and |len| should be page-aligned.

See VmObjectPaged::GetLookupCursorLocked

Adds an allocated page to this cow pages at the specified offset, can be optionally zeroed and

any mappings invalidated. If an error is returned the caller retains ownership of |page|.

Offset must be page aligned. Mappings being invalidated is controlled by |deferred| where if

a nullptr is passed then no mappings, in this object or any child, will be invalidated. If

|deferred| is non-null then mappings will be invalidated if necessary, both immediately in this

object, and via |deferred| for the children.

|overwrite| controls how the function handles pre-existing content at |offset|. If |overwrite|

does not permit replacing the content, ZX_ERR_ALREADY_EXISTS will be returned. If a page is

released from the page list as a result of overwriting, it is returned through |released_page|

and the caller takes ownership of this page. If the |overwrite| action is such that a page

cannot be released, it is valid for the caller to pass in nullptr for |released_page|.

Adds a set of pages consecutively starting from the given offset. Regardless of the return

result ownership of the pages is taken. Pages are assumed to be in the ALLOC state and can be

optionally zeroed before inserting. start_offset must be page aligned.

|overwrite| controls how the function handles pre-existing content in the range, however it is

not valid to specify the |CanOverwriteSlot::PageOrRef| option, as any pages or compressed

references that would get released as a consequence cannot be returned.

Attempts to release pages in the pages list causing the range to become copy-on-write again.

For consistency if there is a parent or a backing page source, such that the range would not

explicitly copy-on-write the zero page then this will fail. Use ZeroPagesLocked for an

operation that is guaranteed to succeed, but may not release memory.

After successful completion the range of pages will all read as zeros. The mechanism used to

achieve this is not guaranteed to decommit, but it will try to.

|range| must be page aligned offsets within the range of the object. |dirty_track| specifies

whether the range being zeroed subscribes to dirty tracking, if |true| the range will start out

as dirty. |dirty_track| only has meaning if the VMO supports dirty tracking, otherwise it is

ignored.

Returns a status code, and the number of bytes that were actually zeroed.

This may be nonzero even if the returned status != ZX_OK.

The returned status is one of the following:

* ZX_OK => The whole range was successfully zeroed.

* ZX_ERR_SHOULD_WAIT => The caller needs to wait on the |page_request| and then retry the

operation. The caller may advance the start offset by the number of zeroed bytes before

retrying.

* Any other error code indicates a failure to zero a part of the range or the whole

range.

Attempts to commit a range of pages. This has three kinds of return status

ZX_OK => The whole range was successfully committed and |len| will be written to

|committed_len|

ZX_ERR_SHOULD_WAIT => A partial (potentially 0) range was committed (output in |committed_len|

and the passed in |page_request| should be waited on before retrying

the commit operation. The portion that was successfully committed does

not need to retried.

* => Any other error, the number of pages committed is undefined.

The |offset| and |len| are assumed to be page aligned and within the range of |size_|.

Increases the pin count of the range of pages given by |offset| and |len|. The full range must

already be committed and this either pins all pages in the range, or pins no pages and returns

an error. The caller can assume that on success len / kPageSize pages were pinned.

The |offset| and |len| are assumed to be page aligned and within the range of |size_|.

All pages in the specified range are assumed to be non-loaned pages, so the caller is expected

to replace any loaned pages beforehand if required.

See VmObject::Unpin

An optional |DeferredOps| can be provided for the purposes of performing extra debugging

checks, but otherwise has no functional requirement. The debug checks are optional as call

sites may not be able to satisfy the locking requirements to construct a DeferredOps, and may

know (due to be undoing a pin they themselves had started), that no checks need to be done.

See VmObject::DebugIsRangePinned

Returns true if a page is not currently committed, and if the offset were to be read from, it

would be read as zero. Requested offset must be page aligned and within range.

Asks the VMO to attempt to reclaim the specified page. There are a few possible outcomes:

1. Exactly this page is reclaimed.

2. This page and other pages are reclaimed.

3. Just other pages are reclaimed.

4. No pages are reclaimed.

Pages other than the one requested may get reclaimed due to any internal relationships between

pages that make it meaningless or difficult to reclaim just the single page in question.

In the cases of (3) and (4) there are some guarantees provided:

1. If the |page| was not from this VMO (or not at the specified offset) then nothing about the

|page| or this VMO will be modified.

2. If the |page| is from this VMO and offset (and was not reclaimed) then the page will have

been removed from any candidate reclamation lists (such as the DontNeed pager backed list).

The effect of (2) is that the caller can assume in the case of reclamation failure it will not

keep finding this page as a reclamation candidate and infinitely retry it.

If the |compressor| is non-null then it must have just had |Arm| called on it.

|eviction_action| hints indicates whether the |always_need| eviction hint should be respected

or ignored. Require will force eviction.

The actual number of pages reclaimed is returned if successful, or a failure reason if not.

VmCowReclaimResult is an alias for fit::result

<VmCowReclaimFailure

, VmCowReclaimSuccess>.

Helper for reclamation functions to perform common checks for whether or not reclamation should

proceed. It takes two parameters, one being the original requested page and the other being

the result of a page list Lookup or LookupMutable, allowing it to check if the page is still

up to date and owned by this VMO.

If any pages in the specified range are loaned pages, replaces them with non-loaned pages

(which requires providing a |page_request|). The specified range should be fully committed

before calling this function. If a gap or a marker is encountered, or a loaned page cannot be

replaced, returns early with ZX_ERR_BAD_STATE. If the replacement needs to wait on the PMM for

allocation, returns ZX_ERR_SHOULD_WAIT, and the caller should wait on the |page_request|.

|non_loaned_len| is set to the length (starting at |offset|) that contains only non-loaned

pages. |offset| and |len| must be page-aligned.

If page is still at offset, replace it with a loaned page.

Attempts to dedup the given page at the specified offset with the zero page. The only

correctness requirement for this is that `page` must be *some* valid vm_page_t, meaning that

all race conditions are handled internally. This function returns false if

* page is either not from this VMO, or not found at the specified offset

* page is pinned

* vmo is uncached

* page is not all zeroes

Otherwise 'true' is returned and the page will have been returned to the pmm with a zero page

marker put in its place.

see VmObject::DebugLookupDepth

VMO_VALIDATION

Calls DebugValidatePageSharesLocked on this and every parent in the chain, returning true if

all return true. Also calls DebugValidateBacklinksLocked on every node in the hierarchy.

Walks all the descendants in a preorder traversal. Stops if func returns anything other than

ZX_OK.

VMO_FRUGAL_VALIDATION

Applies the specific operation to all mappings in the given range. The mappings for the current

object are operated on immediately, with any children being operated on using |deferred|. If

the caller knows that no |DeferredOps| is needed (e.g. the VMO has no children and is not pager

backed) then a nullptr can be provided.

The VmObjectPaged is changing its cache mapping policy. Clean / invalidate all existing pages

and update page queues if required.

Promote pages in the specified range for reclamation under memory pressure. |offset| will be

rounded down to the page boundary, and |len| will be rounded up to the page boundary.

Currently used only for pager-backed VMOs to move their pages to the end of the

pager-backed queue, so that they can be evicted first.

Protect pages in the specified range from reclamation under memory pressure. |offset| will be

rounded down to the page boundary, and |len| will be rounded up to the page boundary. Any

absent pages in the range will first be committed and then, if |set_always_need| is true, the

|always_need| flag in the pages will be set.

If the |ignore_errors| flag is set then any per page errors will be ignored and future pages in

the range will still be operated on. If this flag is not set then any kind of error causes an

immediate abort.

Ensures any pages in the specified range are not compressed, but does not otherwise commit any

pages.

Returns the parent of this cow pages, may be null. Generally the parent should never be

directly accessed externally, but this exposed specifically for tests.

Initializes the PageCache instance for COW page allocations.

Unlocked wrapper around ReplacePageLocked, exposed for the physical page provider to cancel

loans with.

Eviction wrapper, unlike ReclaimPage this wrapper can assume it just needs to evict, and has no

requirements on updating any reclamation lists. Exposed for the physical page provider to

reclaim loaned pages.

Is also used as an internal helper by ReclaimPage.

Potentially transitions from Alive->Dead if the cow pages is unreachable (i.e. has no

paged_ref_ and no children). Used by the VmObjectPaged when it unlinks the paged_ref_, but

prior to dropping the RefPtr, giving the VmCowPages a chance to transition.

If a VmCowPages is returned then this is a parent that needs to have MaybeDeadTransition called

on it.

Helper to allocate a new page for the VMO, filling out the page request if necessary.

static

Used to track dirty_state in the vm_page_t.

The transitions between the three states can roughly be summarized as follows:

1. A page starts off as Clean when supplied.

2. A write transitions the page from Clean to Dirty.

3. A writeback_begin moves the Dirty page to AwaitingClean.

4. A writeback_end moves the AwaitingClean page to Clean.

5. A write that comes in while the writeback is in progress (i.e. the page is AwaitingClean)

moves the AwaitingClean page back to Dirty.

Controls the type of VmPageOrMarker slot in |this| VmCowPages' page_list_ that can be

overwritten by the Add[New]Page[s]Locked functions. It is the caller's responsibility to ensure

that the previous content is dealt with correctly (e.g. any pages and compressed

references are freed).

TODO(sagebarreda

@

): consider refactoring eviction out of reclamation so it can be called

instead of using reclamation with `Require`.