如何将迭代器(或类似的东西)存储到硬盘上

How to store iterators (or something simmilar) onto harddisk?

本文关键字:存储 硬盘 迭代器      更新时间:2023-10-16

我想将巨大的std::list<my_big_struct>(但在此示例中很少)存放到硬质上。

此外,我也想将许多std::list<std::list<my_big_struct>::const_iterator>也存储在硬盘上。

我如何在没有深复制的情况下存档(会用完太空)?

在一个简短的示例下,用my_big_struct作为简单的int

#include <list>
#include <fstream>
#include <boost/serialization/serialization.hpp>
#include <boost/serialization/list.hpp>
#include <boost/archive/binary_oarchive.hpp>
typedef int my_big_struct;
int main()
{
    std::list<my_big_struct> a = {3,2,7,4,9};
    std::list<std::list<my_big_struct>::const_iterator> aa;
    std::list<std::list<my_big_struct>::const_iterator> ab;
    std::list<my_big_struct>::const_iterator find = a.begin();
    aa.emplace(aa.begin(), find++);
    ab.emplace(ab.begin(), find);
    {
        std::ofstream f("test_a.bin", std::ios::binary);
        boost::archive::binary_oarchive o(f);
        o << a; // OK
    }
    // how to store aa and ab onto harddisk without hardcopy ???
}

我不确定设计(听起来像是保留挥发性记忆关系的设计气味)。

如果您确信自己的设计是合理的,我建议使用managed_shared_memory(这是Boost Cranscess)。

然后,您可以依靠std::list。的迭代稳定性。

demo

这说明了它的工作方式:

活在coliru

#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/list.hpp>
#include <iostream>
namespace bip = boost::interprocess;
typedef int my_big_struct;
namespace shared {
    using segment = bip::managed_mapped_file;
    using manager = segment::segment_manager;
    template <typename T> using alloc = bip::allocator<T, manager>;
    template <typename T> using list  = boost::interprocess::list<T, alloc<T> >;
}
int main()
{
    using List   = shared::list<my_big_struct>;
    using ItList = shared::list<List::const_iterator>;
    std::remove("test.dat");
    {
        shared::segment segment(bip::open_or_create, "test.dat", 10u<<20); // 10 megabyte
        auto& a  = *segment.find_or_construct<List>("a")(segment.get_segment_manager());
        auto& aa = *segment.find_or_construct<ItList>("aa")(segment.get_segment_manager());
        auto& ab = *segment.find_or_construct<ItList>("ab")(segment.get_segment_manager());
        a.assign({3,2,7,4,9});
        List::const_iterator find = a.begin();
        aa.emplace(aa.begin(), find++);
        ab.emplace(ab.begin(), find);
        ab.emplace(ab.begin(), std::next(find,3));
    }
    // the file is persisted
    {
        shared::segment segment(bip::open_only, "test.dat"); // read it back
        auto& a  = *segment.find_or_construct<List>("a")(segment.get_segment_manager());
        auto& ab = *segment.find_or_construct<ItList>("ab")(segment.get_segment_manager());
        for (auto iterator : ab) {
            std::cout << "Iterator in ab points to " << *iterator << " in list an";
            my_big_struct const* pointer = &*iterator;
            for (auto& element : a) {
                if (&element == pointer) {
                    std::cout << "Matching element is found in a (by physical address)n";
                }
            }
        }
    }
}

打印

Iterator in ab points to 9 in list a
Matching element is found in a (by physical address)
Iterator in ab points to 2 in list a
Matching element is found in a (by physical address)

注释

为某些碎片开销做好准备,因此请谨慎构建数据架构(如果可能的话,前面管理分配)。

您可以在Boost Intersocess

上查看我的一些较旧的答案,以获取灵感
  • boost :: Interpocess :: Managed_shared_memory
  • boost :: interpocess :: managed_mapped_file
  • scoped_allocator_adaptor以提供更多方便

κ通过使用offset_ptr在内部内部管理抽象(您不必知道此实现细节)

更多样品

如果您也想在my_big_struct中包含复杂成员,则需要在此处使用细分分配器:

活在coliru

打印

Iterator in ab points to my_big_struct { name:nine, value: 9 } in list a
Matching element is found in a (by physical address)
Iterator in ab points to my_big_struct { name:two, value: 2 } in list a
Matching element is found in a (by physical address)