Connect with us

Latest

What they don’t tell you about demand paging in school

Mish Boyka

Published

on

s my adventures with the Linux virtual memory subsystem, and my discovery of a creative way to taunt the OOM (out of memory) killer by accumulating memory in the kernel, rather than in userspace.

Keep reading and you’ll learn:

  • Internal details of the Linux kernel’s demand paging implementation
  • How to exploit virtual memory to implement highly efficient sparse data structures
  • What page tables are and how to calculate the memory overhead incurred by them
  • A cute way to get killed by the OOM killer while appearing to consume very little memory (great for parties)

 

Part 1: Demand paging is nuanced

As usual, the story begins with me asking questions about implementation details. This time, about the Linux kernel’s demand paging implementation.

In Operating Systems 101 we learn that operating systems are “lazy” when they allocate memory to processes. When you mmap() an anonymous page, the kernel slyly returns a pointer immediately. It then waits until you trigger a page fault by “touching” that memory before doing the real memory allocation work. This is called “demand paging”.

This is efficient — if the memory is never touched, no physical memory is ever allocated. This also means you can allocate virtual memory in vast excess of what is physically available (“overcommit”), which can be useful.. You just can’t touch it all.

Let’s dive deeper. Barring execution, “touching” memory means reading or writing. Writes to a new mmap’d region require the kernel to perform a full memory allocation. You need memory, you need it now, and the kernel can’t push it off any longer.

Now, the question: What about reads?

Unlike writes, reads to a new mmap’d region do not trigger a memory allocation. The kernel continues to push off the allocation by exploiting how new anonymous mappings must be zero initialized. Instead of allocating memory, the kernel services the page fault using the “zero page”: a pre-allocated page of physical memory, completely filled with zeros. In theory this is “free” — a single physical frame can back all zero-initialized pages.

The point? Demand paging is nuanced — not all ways of accessing a new mapping require the kernel to allocate memory.

Let’s see what this looks like in the source. The core page fault handler, handle_mm_fault is in mm/memory.c. A few calls deep via __handle_mm_fault and handle_pte_fault, we hit this block:

static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
{
	// ...

	if (!vmf->pte) {
		if (vma_is_anonymous(vmf->vma))
			return do_anonymous_page(vmf);
		else
			// ...
	}

	// ...
}

Which leads us to do_anonymous_page — the core page fault handler for anonymous pages.

static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
{
	// ...
	
	if (pte_alloc(vma->vm_mm, vmf->pmd))
		return VM_FAULT_OOM;
	// ...

	/* Use the zero-page for reads */
	if (!(vmf->flags & FAULT_FLAG_WRITE) && // (1)
			!mm_forbids_zeropage(vma->vm_mm)) {
		entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address), // (2)
						vma->vm_page_prot));
		vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
				vmf->address, &vmf->ptl);
		// ...
		goto setpte;
	}
	// ...
setpte:
	set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); // (3)
	// ...
	return ret;

	// ...
}

Bingo. It checks whether a read caused the fault (1), then maps the virtual page to the zero page (2 and 3).

Note that this all happens in the page fault handler, which is an implementation choice. The mmap core logic does not touch the page tables at all, and only records the presence of the new mapping. It leaves the mapping’s page table entry non-present (present bit = 0) which will trigger a page fault on access.

Alternatively, mmap could proactively allocate the page table entry, and initialize it to the zero page. This would avoid a page fault on the first read, but at the cost of initializing (potentially many) page table entries up front. Given that it’s most efficient to be maximally lazy, the current implementation is best.

Part 2: Allocating infinite memory, and touching it too

This led me to another question. Since reads from anonymous mappings are “free”, in addition to allocating excessive virtual memory, can’t you actually touch all of it? As long as that “touch” is a read?

Time for an experiment. Here’s some code that allocates 100 GB of linear memory, and tries to read from the first byte of each page. It allocates 512 MB at a time because you can’t directly ask mmap for 100 GB :). My test system was a x64 Ubuntu 20.04 VPS.

#include <sys/mman.h>
#include <iostream>

const size_t MB = 1024 * 1024;
const size_t GB = MB * 1024;

int main() {
  size_t alloc_size = 512 * MB;
  size_t total_alloc = 100 * GB;
  size_t num_allocs = total_alloc / alloc_size;

  // std::cout << "alloc_size (MB)" << alloc_size / (1024*1024)<< "n";
  // std::cout << "total_alloc " << total_alloc << "n";
  // std::cout << "num_allocs " << num_allocs << "n";

  std::cout << "Allocating mem...n";

  char* base = nullptr;

  // Allocate a ton of memory
  for (size_t i = 0; i < num_allocs; i++) {
    // Unsound alert - assuming allocations are contiguous and grow down.
    base = (char*)mmap(NULL, alloc_size, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (base == MAP_FAILED) {
      perror(NULL);
      throw std::runtime_error("Fail");
    }
    std::cout << (void*)base << " " << i << "n";
  }

  std::cout << "Allocated Virtual Mem (GB): " << total_alloc / GB << "n";
  std::cout << "Base Addr: " << (void*)base << "n";
  std::cout << "Press enter to start reading.n";
  getchar();
  std::cout << "Reading each page...n";

  // Read the first byte of each page
  for (size_t i = 0; i < total_alloc; i += 0x1000) {
    auto x = base[i];
  }

  std::cout << "Done!n";
  getchar();
}

When we run it, here’s what we see:

$ ./demo
Allocating mem...
0x7f3f6d300000 1
0x7f3f4d300000 2
0x7f3f2d300000 3
...
0x7f26cd300000 198
0x7f26ad300000 199
0x7f268d300000 200
Allocated Virtual Mem (GB): 100
Base Addr: 0x7f268d300000
Press enter to start reading.

It successfully allocated 100 GB of linear virtual memory in 512 MB chunks. We can confirm this with pmap, which shows a 100GB region of anonymous virtual memory at the base address printed.

$ pmap `pidof demo`
485209:   ./demo
00005600e1d0c000      4K r---- demo
00005600e1d0d000      4K r-x-- demo
00005600e1d0e000      4K r---- demo
00005600e1d0f000      4K r---- demo
00005600e1d10000      4K rw--- demo
00005600e2a47000    132K rw---   [ anon ]
00007f268d300000 104857600K r----   [ anon ] <<<< 100 GB region
00007f3f8d300000     16K rw---   [ anon ]
00007f3f8d304000     60K r---- libm-2.31.so
00007f3f8d313000    668K r-x-- libm-2.31.so
...

What does htop say?

htop confirms that 100 GB of virtual memory is allocated (VIRT column), but a much more reasonable 1540 KB of resident memory (RES) is actually occupying RAM. Note the MINFLT column — this is the number of “minor” page faults that have occurred. A minor page fault is one that does not require loading from disk. We will be triggering lots of these and should expect to see this number grow dramatically.

Here’s what happens after I press enter to trigger reading.

0x7f26cd300000 198
0x7f26ad300000 199
0x7f268d300000 200
Allocated Virtual Mem (GB): 100
Base Addr: 0x7f268d300000
Press enter to start reading.

Reading each page...
Done!

The process hits the “Done!” print. This means it successfully touched every page of the 100 GB allocation!

htop confirms that many minor faults have occurred. We would expect it to cause exactly 26214400 faults (100 GB / 4 KB), and indeed, 26214552 – 26214400 = 152, the number we started with. Intriguingly, the resident memory appears to have also increased, which should not have happened. See the Appendix A for discussion of this.

Here’s a video to see it all in action:

So the theory is confirmed! You can apparently “Allocate your memory, and touch it too” (as long as that touch is a read).

If your application, for some reason, benefits from the ability to have a 100 GB array of zeros then this is perfect for you. What about the rest of us?

A closer-to-real-life application is a sparse array. A sparse array is a (typically very large) array, whose elements are mostly zero. By exploiting demand paging, you can implement a memory efficient sparse array, where the majority of the array is backed by the zero page (or not even mapped). You get the fast indexing benefits of an array while avoiding the memory overhead.

Part 3: Playing with a killer

I have a confession. Remember when I said not all ways of accessing memory require the kernel to allocate memory? Yeah, that’s a lie.

Even though a read can be serviced by the shared zero page, that doesn’t mean no memory is allocated. Even the process of mapping a zero page requires allocating memory. And here’s where we get into the nitty gritty.

The hidden overhead of page tables

The overhead comes from the virtual memory infrastructure itself — the page tables. Page tables are data structures that power the virtual memory subsystem. Like regular data structures, they occupy memory, only their overhead is easy to overlook, since it’s hidden from userspace.

This is what page tables on x86_64 look like:

x86_64 Page Tables: Intel Manual Vol 3A Chapter 4.5

They’re a tree data structure that is 4 levels deep, with each node (table) being an array of 512 8-byte entries. Together these tables offer an efficient way to represent a mapping between every virtual page in the address space and a physical frame.

Here’s where the overhead comes from. Each page touched requires 1 Page Table Entry (PTE) to be allocated. However, PTEs aren’t allocated individually. They’re allocated in blocks of 512 called Page Tables. Each Page Table requires 1 Page Directory Entry to be allocated. But Page Directory Entries aren’t allocated individually either, they’re allocated in blocks of 512 called Page Directories. This propagates all the way up to the top level of the tree: the PML4 Table (“Page Map Level 4” Table). There is only one PML4 Table and it’s pointed to by the CPU’s CR3 register.

Note that while “Page Table” refers to a specific type of table, these tables are all colloquially referred to as “page tables”.

Since page table entries are 8 bytes, and all page tables contain 512 entries, it follows that page tables are 4096 bytes. This doesn’t seem like much, but to map even one page, you need one of each table. That’s already 16 KB of overhead. If you’re mapping gigabytes of virtual address space, this overhead will add up.

Intentionally trying to get killed

This leads us to the final question(s). Specifically, how much can page table overhead add up to? Is it possible for page tables to occupy some non-negligible portion of memory? Is it possible to exhaust memory, purely from page tables? How much virtual memory would we need to map to do so?

Here’s pseudocode to calculate the page table overhead from a virtual memory allocation.

Input: virtual_pages_allocated

page_table_entries = virtual_pages_allocated
page_tables        = CEILING(page_table_entries, 512) / 512
page_tables_bytes  = page_tables * 4096
page_dirs          = CEILING(page_tables, 512) / 512
page_dirs_bytes    = page_dirs * 4096
pdp_tables         = CEILING(page_dirs, 512) / 512
pdp_tables_bytes   = pdp_tables * 4096
pml4_table         = 1
pml4_table_bytes   = 1 * 4096

Output: total_overhead = page_tables_bytes + page_dirs_bytes +  
                         pdp_tables_bytes + pml4_table_bytes

Remember that tables are allocated as a whole. Even if a table is partially filled, it still occupies the full 4 KB. This is why the calculation needs to round up to the nearest multiple of 512 (expressed as the CEILING function from Excel).

Using this method, we calculate that a 512 GB allocation of virtual memory requires slightly over 1 GB of page tables!

VM Alloc GB             512
Bytes Alloc             549755813888
Pages Alloc             134217728
PTEs                    134217728
Page Tables             2621445
Page Tables Bytes       1073741824
Page Dirs               512
Page Dirs Bytes         2097152
PDP Tables              1
PDP Tables Bytes        4096
PML4 Tables             1
PML4 Tables Bytes       4096

Total Overhead Bytes    1075847168
Total Overhead GB       1.001960754

Here’s a spreadsheet so you can play with the numbers yourself: https://docs.google.com/spreadsheets/d/10h8sSvocjI-Fu6NW7B3jv1C3NGoflgaGTp1HGpkV1e4/edit?usp=sharing

My machine only has 1 GB of memory, so this should be more than enough to exhaust memory and ideally, trigger the OOM killer.

Time for another experiment! I changed the code above to allocate 512 GB instead of 100 GB, and reran it.

Here’s the output:

...
0x7eca126ba000 1022
0x7ec9f26ba000 1023
0x7ec9d26ba000 1024
Allocated Virtual Mem (GB): 512
Base Addr: 0x7ec9d26ba000
Press enter to start reading.

Reading each page...
fish: './demo2' terminated by signal SIGKILL (Forced quit)

It worked! After a few minutes, memory was exhausted, and the OOM killer killed the process.

Here’s a screenshot right before the process is killed. Points of interest:

  • System memory meter close to maxed out: 893 MB of 981 MB used.
  • Resident memory remains extremely low despite the OOM score being extremely high. This shows that page table overhead does not affect resident memory.
  • Minor faults are orders of magnitude greater than anything else on the system.
  • The bottom section shows the VmPTE field of /proc/*/status, confirming that close to 800MB of memory is used by page tables (close to physical memory limit).

The pictures are nice, but it’s much more exciting to see it happen live.

Note that the code performs all the mmaps first, then touches the memory. This is intentional. An alternate approach where it touches the mapping immediately after mmap’ing won’t work because mmap will eventually fail. This will prevent us from allocating more memory and pushing the system to its limit.

When we mmap everything up front and then touch the pages, we take advantage of overcommit, which will allow mmaps to succeed long after physical memory limits would have been bypassed. We then exploit the fact that a valid memory access has no way to “fail” — page faults must be handled.

An old friend

With a sufficiently quiet system (and some luck), it’s possible to get this stack trace in the OOM killer’s logging:

This is the kernel stack trace of the allocation that triggered the OOM kill. In the middle of the trace, we find familiar functions: handle_mm_fault, and do_anonymous_page. This is the page fault handler for anonymous pages, which we saw earlier.

This trace shows that the final allocation that triggered the OOM killer was our own attempt at mapping another zero page — an operation that’s “free” in theory, but can get you OOM killed in practice.

Conclusion

If you look closely enough at demand paging you’ll find nuance. Writes obviously trigger allocations, but reads can be efficiently serviced by the shared zero page for “free”.

This enables applications to allocate vast amounts of virtual address space and access all of it!.. as long as they only want to read (zeros). This can be applied practically with sparse data structures, whose elements are mostly zero.

That said, the eager virtual-memory-savvy programmer should be warned — while invisible to userspace, “free” zero page mappings are not free in practice, and can incur substantial memory overhead from the virtual memory infrastructure itself.

Page tables are just one of several sources of system memory overhead (per-thread kernel stacks being another example). Usually application developers can safely ignore these, but I hope this article offers a glimpse into the world of systems developers whose responsibility it is to provide that safety.


Learn something new? Let me know!

Did you learn something from this post? I’d love to hear what it was — tweet me @offlinemark!

I also have a mailing list if you want to know when I write new posts:


Acknowledgements

Thanks to Jann Horn for help understanding the resident memory spike. Thanks to Jake Miller, Roderic Morris, James Larisch, and William Woodruff for reviewing earlier drafts of this post.

Summary: if you need to measure resident memory usage with high accuracy, do not use htop (or /proc/*/{statm,status}). Use /proc/*/smaps or /proc/*/smaps_rollup instead.

In these experiments, a mysterious 1.5 MB spike in resident memory appears while triggering reads of all the pages. This should not be the case — as described above, faulting zero pages only allocates page tables, which don’t count toward resident memory. The resident memory reported immediately before performing the reads should stay constant for the remainder of execution.

I was mystified, and convinced that I found a kernel bug. I confirmed that there is nothing happening in userspace that would cause this — the page faulting loop has a single memory access instruction in its body. Single stepping confirms that this instruction alone triggers the spike. I also confirmed that the source of htop’s data, /proc/*/statm, shows this spike also, so it’s not a htop bug.

To my disappointment, it turns out that this isn’t a bug. (Thanks to Jann Horn for showing me this.)

Different files in /proc offer varying levels of accuracy. This is not documented in the man page, but is mentioned in the kernel’s internal documentation and this stack overflow post. htop receives memory statistics information from /proc/*/statm which intentionally offers a lower accuracy measurement, in exchange for better multithreaded performance. To avoid lock contention, the kernel records memory usage in per-thread caches, and then synchronizes those with the process-wide cache after every 64 page faults.

So, the “spike” I was observing is not actually a spike — it’s the synchronization threshold being hit, which updates the process-wide information in /proc/*/statm. The resident memory is actually always at 3 MB, it’s just that the reporting layer takes some time to get updated because my program happens to stop on a getchar() in the middle of the 64 fault cycle.

If you want high accuracy memory information, use /proc/*/smaps and /proc/*/smaps_rollup. At the expense of performance, these files offer higher accuracy because their implementations walk internal data structures, rather than using internal caches.

A lingering mystery remains. Can you help?

A small mystery remains: if the statistics sync every 64 page faults, how can 1.5 MB of error accumulate? Syncing every 64 faults suggests that the maximum error would be 4 KB * 63 faults = 252 KB of error.

Health

Wisconsin health officials worry about contact tracing plans

Avatar

Published

on

By

APPLETON, Wis. — A key element of Wisconsin’s plan to help contain the COVID-19 pandemic is proving to be difficult.

With an average of 3,400 daily cases reported in the last week in Wisconsin, and an additional 3,626 positive tests confirmed in Sunday’s update, contact tracers are now so overrun that some have begun to wonder whether they can keep up with the task at hand. The total number of cases since the pandemic began is approaching 200,000

“That’s a very valid question. I’ve been in many meetings where it’s been asked,” said Kim Goffard, communicable disease nurse supervisor in Winnebago County, among the state’s hardest-hit places. “At what point is enough enough?”

Some Wisconsin counties are so strapped that they’re now asking infected residents to reach out to their contacts themselves. That’s why Dane County, the second-most populous in the state, said in a news release it has switched to a “crisis model” of contact tracing.

Officials at the state Department of Health Services have declined to provide specific metrics about the success of Wisconsin’s tracing efforts, such as the percentage of people it has reached and how quickly, but recent press briefings have shown the difficulty of the job, the Post Crescent reported.

The department’s chief medical officer, Dr. Ryan Westergaard, told reporters last week that Wisconsin’s entire public health infrastructure “cannot keep up,” including state, local and tribal health offices.

“Public health is so strained that you can’t count on us to tell you (where the virus is), which is really, really not where we want to be,” he said.

The state said it hasn’t given up. It plans to hire more tracers who can chip in where spread is most rampant. Local health departments are adjusting their priorities to protect the most at-risk populations, such as those in long-term care facilities or homeless shelters.

Even so, public health officials say the only way to get contact tracing back on track is for residents to increase their efforts to control the virus. That means keeping circles small, wearing face masks, physically distancing and washing hands.

“We control our destiny,” said Kurt Eggebrecht, Appleton’s city health officer.

Continue Reading

Latest

Long-term care residents in S.D. suffering health crisis due to COVID-19 isolation

Avatar

Published

on

By

As nursing homes and assisted-living facilities have restricted the movements of residents and eliminated most in-person visits by friends and families amid the COVID-19 pandemic, an unexpected elderly health crisis has emerged in South Dakota and across the country.

The mental and physical health of many residents of long-term care facilities are declining and in some cases leading to death from the extended isolation residents are undergoing due to restrictions on visitation and interaction aimed at preventing the spread of the coronavirus among a highly vulnerable population.

Recent efforts in South Dakota to use federal relief funds to increase testing for the coronavirus among facility visitors or to implement other measures to reduce isolation have failed, leaving many worried that more elderly residents will suffer the slow decline caused by loneliness, isolation and sensory deprivation.

Anecdotal reports of the decline of residents of long-term care facilities have become commonplace in South Dakota and beyond. Advocates for the elderly and family members of facility residents recently testified before a South Dakota legislative committee that patients with Alzheimer’s disease have declined rapidly, that residents are losing the will to live, and that some have died unexpectedly from causes not directly related to the virus.

Terryl Cadwell told lawmakers in September that her father, Jim Rumbolz, 88, was living an active lifestyle at the Avera Prince of Peace Retirement Community in Sioux Falls before the pandemic. When coronavirus restrictions were implemented, including limits on social activities and family visits, Rumbolz quickly deteriorated and died in mid-June.

“There was never a slip we noticed before COVID in any of his mental capacities at all, so this was really devastating,” Cadwell said. “I feel like it was the isolation that ended this life shorter than it should have been.”

Terryl Cadwell of Sioux Falls visited with her father, Jim Rumbolz, at a Sioux Falls long-term care facility in late 2019. Rumbolz suffered a rapid decline in health during the COVID-19 pandemic and died in June; Cadwell blames his death in part on isolation he endured. Photo: Courtesy Terryl Cadwell

As the pandemic drags on, medical researchers are starting to drill in on the potential consequences of extreme isolation among elderly people.

In a paper published this summer in the Journal of the American Medical Directors Association, researchers from the U.S. and Australia noted that severe loneliness was present among residents of long-term care facilities at double the rate compared with people living in a personal residence, and that the pandemic has exacerbated that difference.

“A feeling of loneliness has many deleterious consequences … including increased risk of depression, alcoholism, suicidal thoughts, aggressive behaviors, anxiety and impulsivity,” the article said.

Other studies have shown that loneliness is a risk factor for stroke, obesity, elevated blood pressure, worsening Alzheimer’s symptoms and death. Some studies have shown that extreme isolation and loneliness among the elderly can rival the mortality risks of smoking, obesity and high blood pressure.

Data compiled by the federal Centers for Disease Control and Prevention indicate that deaths among dementia patients in the U.S. have risen sharply during the COVID-19 pandemic, not only due to the virus itself.

In the second week of April, U.S. deaths due to dementia rose by 42% compared with the average deaths in the same week from 2015 to 2019.

From March to September 2020, dementia deaths in the country increased by as many as 2,500 per week compared with the average during that time period in 2015-2019. In all, about 15,000 more people with dementia have died since March due to “unexpected” causes, which could be from the virus but also for unknown reasons, according to the CDC.

Advocates for the elderly joined with family members of long-term care residents to make the case to state lawmakers in September that some of the $1.5 billion South Dakota received from the federal CARES Act pandemic relief fund should be spent on efforts to help facilities open safely to visitors.

Their pleas for rapid testing of potential visitors and other efforts drew sympathetic responses from lawmakers, but in the end fell short. The Legislature held a special session in October and did raise the aid level for long-term care facilities from the $100 million originally proposed by Gov. Kristi Noem to $115 million.

But rules set by Congress allow CARES Act money to be used only to offset net financial losses by businesses, including long-term facilities. None of the money, which must be spent by Dec. 31, can be used on capital projects or to create new programs or install new infrastructure. The additional $15 million was added to help long-term care facilities offset financial losses during September instead of only March through August, which was the allowable time frame for losses to be offset by most of the CARES Act money. Many facilities were already facing financial headwinds before the pandemic hit.

That leaves long-term care facilities with no new money to buy or use rapid COVID-19 tests on visitors who want to safely enter the facilities to visit relatives. Furthermore, the Legislature allocated no state funds for other unique efforts beyond testing that could improve visitor access to long-term care facilities, such as creating new entrances, allowing relatives to be designated as essential caregivers or developing structural ways to allow physical visits at reduced risk.

Efforts to find money and methods to help reduce isolation are continuing in South Dakota as more attention gets paid to the plight of long-term care residents, said Joe Schartz, public policy director for the Alzheimer’s Association of South Dakota.

“It’s been a groundswell of interest from people who haven’t seen mom or dad or grandma or grandpa for months except through a window or by phone, and they’re seeing them wither away,” Schartz said. “It’s not unique to South Dakota; it’s a national crisis and is happening all over the world.”

“People who haven’t seen mom or dad or grandma or grandpa for months except through a window or by phone [are] seeing them wither away. It’s not unique to South Dakota; it’s a national crisis and is happening all over the world.”

— Joe Schartz, Alzheimer’s Association of South Dakota

Virus protections needed; health declines result

State Sen. Deb Soholt, R-Sioux Falls, has a keen interest in helping facility residents and their families, both as the chair of the Senate Health and Human Services Committee and as a daughter whose 91-year-old mother is in an assisted-living facility.

Her mother suffers from cognitive decline but is mostly self-sufficient in the facility, Soholt said.
And yet, since the pandemic hit in mid-March, Soholt has seen her mother’s mental, spiritual and physical condition decline.

“They don’t eat together, they don’t socialize together, they’re completely isolated,” Soholt said. “Think how we would be if, since the second week of March, we had been in only three rooms 100% of that time.”

Soholt, a registered nurse, said she supports the facility’s measures to protect her mother, other residents, staff and visitors from the virus that is especially deadly to the elderly. But Soholt said that as COVID-19 protections have restricted the movement of residents within the facility, reduced opportunities for socialization in the home and limited opportunities for family and friends to meet with and physically touch her mother, she has watched her mother suffer emotionally and her physical condition worsen.

“She’s a trooper,” Soholt said. “My mom has a great attitude and a sweet personality, but you can just see the exponential decline.”

Soholt supported efforts to use federal CARES Act funding to mitigate isolation of residents, and was disappointed that the money was unavailable. She said she would not give up on efforts to find money to pay for programs to test visitors or find other ways to bring residents and families together while still keeping them safe from the virus.

“We’ve gone so far from protecting them from the virus, and I understand why, but at the expense of any kind of quality of life,” she said. “It’s really about dignity and respect for our elderly.”

Soholt said she remains hopeful that there may be CARES Act funds allocated to South Dakota that do not get spent by the Dec. 31 deadline, and that Congress may allow that money to be used for reasons other than to cover net losses. For instance, the rules could be changed to allow unspent money in South Dakota to be distributed to facilities to implement rapid testing or other programs to safely allow visitors to enter the homes.

If that fails, Soholt said she expects that elderly advocates will appear before the Legislature during the 2021 session starting in January to request state money for programs aimed at mitigating isolation of long-term residents.

Soholt, who will leave the Legislature before the upcoming session due to term limits, said the isolation issue has been put on the radar of the governor and others in government and will likely be seen as a priority in 2021.

“We want to provide funded support for the facilities to be able to develop their plan,” Soholt said. “I’m very encouraged that the legislative branch is interested in trying to do something to mitigate the isolation for our elders.”

Visits to long-term care facilities in South Dakota have been curtailed as protections have been put in place to protect vulnerable residents from the potentially deadly coronavirus. In October, an employee at the Edgewood Rapid City assisted-living facility put gloves on the hands of a resident before a visit with relatives through a plastic barrier.
Photo: Grace Pritchett, Rapid City Journal

Dementia patients at higher risk

Schartz, who testified twice before legislative committees this fall, said research on Alzheimer’s disease and dementia shows that patients fare best when they have regular routines and are surrounded by friends and family in their own homes.

In addition to losing the stability of living at home, dementia patients who reside in long-term facilities are now losing that critical human touch and sense of familiarity due to restrictions on visitation and interaction.

“When you pull someone with Alzheimer’s or dementia out of their routines and you disrupt that and their loved ones are turned away, that isolation can cause the disease to progress and results in worsened outcomes or death,” Schartz said. “The isolation is leading to premature deaths for these people.”

Those cognitive conditions are common among people in long-term facilities across the country, Schartz said. In the U.S., about 48% of nursing-home patients and 42% of assisted-living residents have Alzheimer’s or dementia, he said.

Meanwhile, Schartz said his association and other elderly advocates expect to file a bill, as they did without success last year, for South Dakota to create a new dementia coordinator position within state government.
The coordinator could apply for federal grants, improve training for front-line workers and establish programs to aid Alzheimer’s and dementia patients and caregivers.

About 18,000 people over 65 have Alzheimer’s disease in South Dakota, and the disease if the fifth-leading cause of death in the state. South Dakota has roughly 140,000 people in that age group overall.

The number of patients is expected to climb as the population ages, and the number of dementia patients is undoubtedly rising as well, Schartz said. The state is home to about 40,000 people who are caregivers to patients with Alzheimer’s, and they too have suffered great emotional and physical stress during the pandemic, Schartz said.

“I wish that you could hear their phone calls to our office; I wish you could hear the panic and the strain and the hurt in their voices,” Schartz told lawmakers in September. “The pandemic is pushing these families to the breaking point.”

Experts say the negative effects of isolation on long-term care residents, staff and family members will worsen during the winter months in South Dakota because the few outdoor visits taking place now will become less frequent and the facilities will become more locked down from the cold.

“At the end of the day, nothing is going to replace in-person visiting and human contact, which as humans we all rely on for part of our health and part of our humanity,” he said. “This has gone on for many months, and it’s only going to get much worse as we approach winter and outdoor visitation will become much tougher.”

Tim Mercy hugs his mother, 88-year-old Patsy Mercy, through a “hugging wall” built with plastic barriers to allow a safe physical connection between visitors and residents of the Edgewood Rapid City assisted-living facility in Rapid City.
Photo: Grace Pritchett, Rapid City Journal

Facilities find ways to reduce isolation

Many facilities have allowed residents to have brief visits with patients outdoors on patios or in courtyards, though without physical touching. Other novel methods have been used to reduce isolation, including visits through windows, via internet connections or even through messages left for staff to deliver to residents.

When the pandemic hit in March, assisted-living facilities such as the Edgewood Rapid City senior living complex underwent immediate and significant changes to keep residents and staff safe.

Edgewood Director Erin Andersen said restrictions on visitation and movement within the facility, which has about 95 residents mostly in small apartments, were needed to fight the spread of the coronavirus but took a noticeable emotional and physical toll on residents and staff.

“Prior to COVID, we were like an apartment building where people could come and go and visit as they pleased,” Andersen said. “It was a beehive of activity.”

Under the threat of the virus invading the complex, shopping trips by residents were banned, visitors were not allowed inside, communal meals were replaced by food delivered to individual rooms in disposable containers, and popular activities such as bingo were halted.

Restrictions and testing of staff were heightened when a resident tested positive for COVID-19 early in the pandemic, even though no one ever got sick, Andersen said.

“We didn’t want to be cruel, but we also knew that if it got into our building, it could be really, really deadly, so we wanted to protect everybody even if it wasn’t always the most popular decision,” Andersen said.

A general pall fell over the facility and some situations in particular were especially hard to endure, she said.
“We have one resident whose husband was in a skilled-care community and he passed away from COVID,” Andersen said. “She was not able to see him or be with him in his final days, and it was devastating.”

As the pandemic dragged on, a staff member at Edgewood heard about an effort by an Idaho facility run by the Edgewood group that allowed for a safe way for residents and visitors to physically connect.

After some research, and with donations of materials from several Rapid City businesses, the staff at Edgewood built a so-called “hugging wall,” an 8-foot-tall, 6-foot-wide wooden structure that resembles a window frame in which visitors and guest can embrace. The two users are separated by a vinyl sheet that is cleaned after each use and are able to safely hug by wearing long plastic sleeves that typically would be used to artificially inseminate cattle. The plastic barriers prevent the spread of the coronavirus while allowing two people to fully embrace, Andersen said.

“It does get used a lot,” she said. “And the reactions from people the first time they touched a loved one, it was tears of joy, and actually staff cried too, because you could see the relief. You could see a lot of that loneliness wash away with just a 30-second hug.”

Jim Rumbolz was an active, engaged resident of the Avera Prince of Peace Retirement Community in Sioux Falls when this picture was taken in December 2019. Just six months later, Rumbolz had suffered rapid mental and physical decline that led to his death in June 2020. He did not become infected with the coronavirus, and his daughter believes isolation and sensory deprivation contributed significantly to his passing. Photo: Courtesy Terryl Cadwell

Family members in pain as loved ones suffer

During phone testimony before the legislative Joint Health & Human Services Committee in September, Cadwell of Sioux Falls shared the story of her father’s mental and physical decline and eventual death that she blames largely on isolation he endured during the pandemic.

Cadwell said her father, Jim Rumbolz, had a multi-faceted career that included stints as a state trooper, a criminal-justice educator at the University of South Dakota and finally as a hospital administrator in Custer, S.D. After retiring to Sioux Falls, Rumbolz most recently lived at the Avera Prince of Peace community, where he was well known as active, gregarious and full of life.

Rumbolz, who buzzed around the facility on a scooter, eschewed games like bingo and instead sought out more stimulating and meaningful activities to keep his mind active and sharp, Cadwell said.

He was part of a book club, took art lessons and formed a coffee club to discuss current events. He exercised regularly, read a newspaper every day and relished time spent with friends or relatives either in person or by phone.

“My dad would kiss my hand every day before I left, and I was able to hug him,” Cadwell said, fighting back tears.

But then the COVID-19 pandemic hit, and Cadwell said her father’s overall condition quickly deteriorated as opportunities for activities and visits were restricted and touching was not allowed.

Rumbolz became fearful of caregivers who wore masks, gloves and protective gowns. He felt isolated because he could no longer hear or read the lips of nurses and others as they spoke through masks. He longed for the touch of his daughter and other family members, but was unable to have physical contact with anyone.

In a period of just a few months, Cadwell’s father began to falter, and he died on June 16. In a follow-up interview with News Watch, Cadwell said her father’s official cause of death was a swallowing problem that required his being fed through a tube. He was never diagnosed with COVID-19, she said.

Cadwell insists that the loss of meaning, purpose and stimulation suffered by her father hastened his mental and physical decline and played a large role in his death.

“I believe we are of mind, body and spirit,” Cadwell said. “You shut that off on a guy like that, with no joy from eating, no activities that were his lifeblood, and he can’t see me or hear the nurses, and the joy of life was gone.”

Cadwell’s story is not uncommon among families of residents of long-term care facilities.

Doctors and caregivers have reported that patients with Alzeheimer’s or dementia have become withdrawn or show slowed mental cognition. Some patients have suffered falls, become more susceptible to infections and shown sudden frailty.

Soholt said she noticed her mother is walking with more of a shuffle and may not be as physically strong as she was before the pandemic.

“I’ve seen a change in her gait, and there’s no way to stay strong,” Soholt said. “The fact they’re not out walking in hallways, there’s a strength decline, and they’re shuffling their feet because the space they’re in is so small.”

Tom and Lee Raines of Sioux Falls have worked as a loving couple to make the best of Tom’s diagnosis of Alzheimber’s Disease. But Lee now worries that Tom’s condition has declined rapidly since the COVID-19 pandemic led to restrictions at his long-term care facility that at first prevented Lee from visiting Tom, and later blocked her from touching him physically during outdoor visits. Photo: Courtesy Lee Raines

Lee Raines of Sioux Falls is suffering through what she said is the devastating decline of a man who became the love of her life after a 50-year separation.

Raines, 82, and her husband, Tom, knew each other in the 1950s growing up in eastern South Dakota. They led separate lives, and then were reintroduced in 2006 when both were free of prior relationships; they married in 2007.

After several happy years, Raines began to notice problems. On one occasion a few years ago, Tom was driving them on the interstate near Brookings and had to ask which exit to take to a city where he had worked as a banker for a quarter century.

Tom was eventually diagnosed with Alzheimer’s disease in 2015, Raines said, and he later moved to an assisted-living facility for safety. The disease took a toll on his mind and body, but since the pandemic struck in March, and restrictions on visitation were implemented, his decline has become more rapid.

“Tom was such a bright person, intelligent and fun and always telling a joke, and to see that decline so quickly …” Raines said. “I’ve seen the slow decline since 2015, but he’s definitely declining faster in the past several months.”

His physical condition has also deteriorated amid the pandemic, Raines said, noting that Tom, 82, has lost more than 20 pounds.

Now, they are only able to visit briefly together outside, six feet apart, with no touching allowed. Tom struggles with wearing a mask that sometimes entangles with his hearing aid, which then falls out. Raines tries to stimulate Tom’s mind and memory by asking him about his childhood or writing things on cards that may spur recollections of the past and generate a greater connection to the present.

The inability to fully communicate, and especially the restriction on touching, are especially trying for her husband and other people with Alzheimer’s, many of whom rely on close relationships to stay healthy and aware, Raines said.

Though the visits are a mix of love, hope, appreciation and some sadness, the trips alone back to the home she once shared with Tom can be crushing, Raines said.

“Sometimes I cry all the way home,” she said. “It just breaks your heart, it really does.”

Raines is pushing state government to implement a rapid COVID-19 testing program for visitors to long-term facilities that she said could provide results in less than 20 minutes and allow for safe in-person visits by family and friends. So far, no widespread testing programs for visitors at long-term facilities have been implemented in South Dakota.

Continue Reading

Latest

The Hockey Niñas, four Twin Cities girls, take to the ice to promote Minnesota’s traditional sport

Emily walpole

Published

on

At a time when hockey officials at all levels are working to expand the sport, particularly among minorities, four young girls in the Bloomington area have made it their mission, as well.

Introducing the Hockey Niñas, two pairs of sisters of color who are enjoying the sport while also encouraging others to join them. The four girls are part of the Bloomington Girls Hockey Club, which governs the sport in the hockey-rich suburb.

After a recent practice at Bloomington Ice Garden, the quartet — 11-year-old Adelyn Janzig de la Luz and her sister Elisa, 8, plus Aubrey Lang, 11, and her sister Mia, 9 — engaged in the typical, good-natured chirping about their games. Nearby, their mothers, Laura Janzig de la Luz of Bloomington and Meredith Lang of Richfield, exuded pride in their trailblazing daughters.

“That’s why we started the Hockey Niñas, because we did want to put a spotlight on girls of color that are out here playing hockey — a nontraditional sport for us,” Lang said. “They’re enjoying it and building friendships all the time. Other kids [of color] can see it. ‘Look, we have these players that are represented in hockey. Maybe it’s something we can try, too.’ ”

Added Janzig de la Luz: “The girls love it. They look so forward to spending time on the ice.”

Janzig de la Luz and Lang were introduced to hockey in different ways. A Mexico City native, Janzig de la Luz and her family moved to the United States when she was 5, and she later attended the Academy of Holy Angels.

“That’s when I learned about hockey, but I never really paid attention,” she said. “For Latinos, it’s soccer. You watch soccer, you play soccer.”

Embracing Minnesota

Lang attended Richfield High School, where girls’ hockey became a varsity sport when she was a freshman. After college, she lived in North Carolina before returning to Minnesota.

“We were moving to Minnesota, and Aubrey was like, ‘OK, I want to play hockey.’ I was like, ‘What?’ ” Lang said. “So, I just researched it, and that was the first thing we did when we got here. ‘If we’re in Minnesota, do like the Romans do. We’re playing hockey.’ ”

The Lang and Janzig de la Luz families want to see more people of color playing hockey. So does Glen Andresen, executive director of Minnesota Hockey, the governing body of youth and amateur hockey in the state. The organization falls under the USA Hockey banner.

“If you were to rank our focuses, No. 1 would be growing the game,” Andresen said. “We’ve been excited in the success in that over the last five to 10 years. However, if we were to be honest with ourselves and look at where we have more potential for growth, it’s with families of color or families with different ethnicities. We have prioritized that.”

Andresen said USA Hockey is in the early stages of tracking race and ethnicity in its membership, so complete statistics aren’t available nationally or in Minnesota. He said making people of color comfortable in hockey is important.

“As we’ve been meeting with families and kids who have played the game, one of the things we’ve come to realize is we can’t expect to grow the game much if we don’t change the culture of the experience for those players,” he said.

To make the game more inclusive, Minnesota Hockey started the Little Wild Learn to Play program; the Gear Up Minnesota! program, which supplies equipment to associations to provide to kids; and the Family Mentor program, in which established hockey families welcome new families to the sport.

“We have some more that are more diversity-focused programs that we’ll be launching this year,” Andresen said.

Welcoming new players

Lang said feeling welcome is important to people of color in hockey.

“There’s always minorities playing hockey, but is it an inclusive environment for minorities feeling welcome?” she said. “ … With all the things that are happening with racial injustice and civil unrest, it’s sparking a lot of conversation.”

She hopes that as opportunities for youngsters of color in hockey expand, discrimination on the ice will disappear to the point where they won’t have to warn their children about it. She has participated in focus groups with Minnesota Hockey about issues for people of color.

“We came with our own experiences, but it was really cool to see what everybody else experienced,” she said. “The hard thing to listen to are things on the boys’ side. They experience racial slurs when they’re so young. It’s to the point where some of the kids have signals to their parents in the stands to let them know this is happening.”

Minnesota Hockey on Sept. 30 announced a rule change that will assess match penalties for language, gestures or conduct that is offensive, hateful or discriminatory in nature.

Fun and inclusion

For the Janzig de la Luz and Lang girls, the fun aspect of hockey is front and center. They enjoy the sport and the friendships that have developed from it.

“In second grade, Adelyn came in and I gave her this piece of paper to join hockey,” Aubrey Lang said. “ … I liked that I had a friend playing hockey.”

Aubrey’s and Adelyn’s younger sisters, Mia and Elisa, respectively, relish being the “Littles” in the Hockey Ninas.

“We are strong, independent, cute, fashionable, expensive Littles,” the duo chanted in singsong fashion.

All the while, the quartet shows the way for others like them to become involved in the sport they love.

“It’s nice to see that joy where they’re saying, ‘I’m one of the first but not the last. We’re going to bring more girls of color to play hockey,’ ” Laura Janzig de la Luz said.

Continue Reading

US Election

US Election Remaining

Advertisement

Trending