当负载/清除大量数据时，STD ::向量越慢
std::vector get slower and slower when load/clear huge amount of data
问题
我有一个相当复杂的图像处理应用程序，其中一个子模块需要将巨大的二进制位图加载到内存中。实际上最多高达96 GB（意思是888 888 x 888 888像素图像）。磁盘为2xSSD RAID0，读/写入约1 GB/s。它将图像加载到使用字节向量（每个元素代表8个像素）的矢量的向量（每个元素代表位图中的一行）中。这里的奇怪问题是，在重复加载和清除向量之后（我看到内存实际上是填充和清除的，而没有内存泄漏），每次迭代似乎需要越来越长的时间。特别清除内存需要很长时间。
测试
我进行了一些简单的测试应用，以测试此孤立的和不同的角度。用原始指针代替智能球员也产生了同样的奇怪行为。然后，我尝试使用本机阵列而不是向量，这是解决问题的。在使用向量时，在100次迭代/清除24 GB时间的迭代后，阵列实现稳定。以下是测试应用程序用24 GB的垃圾填充内存，而不是加载实际图像，结果相同。使用128 GB RAM在Windows 10 Pro上进行的测试，并使用Visual Studio 2013 Update进行5。
此功能使用向量进行负载/清除：
void SimpleLoadAndClear_Vector(int width, int height) {
time_t start_time, end_time;
// Load memory
time(&start_time);
cout << "Loading image into memory...";
auto width_bytes = width / 8;
auto image = new vector<vector<unsigned char>*>(height);
for (auto y = 0; y < height; y++) {
(*image)[y] = new vector<unsigned char>(width_bytes);
auto row_ptr = (*image)[y];
for (auto b = 0; b < width_bytes; b++) {
(*row_ptr)[b] = 0xFF;
}
}
cout << "DONE: ";
time(&end_time);
auto mem_load = (int)difftime(end_time, start_time);
cout << to_string(mem_load) << " sec" << endl;
// Clear memory
time(&start_time);
cout << "Clearing memory...";
for (auto y = 0; y < height; y++) {
delete (*image)[y];
}
delete image;
cout << "DONE: ";
time(&end_time);
auto mem_clear = (int)difftime(end_time, start_time);
cout << to_string(mem_clear) + " sec" << endl;
}
此功能使用阵列进行负载清除：
void SimpleLoadAndClear_Array(int width, int height) {
time_t start_time, end_time;
// Load memory
time(&start_time);
cout << "Loading image into memory...";
auto width_bytes = width / 8;
auto image = new unsigned char*[height];
for (auto y = 0; y < height; y++) {
image[y] = new unsigned char[width_bytes];
auto row_ptr = image[y];
for (auto b = 0; b < width_bytes; b++) {
row_ptr[b] = 0xFF;
}
}
cout << "DONE: ";
time(&end_time);
auto mem_load = (int)difftime(end_time, start_time);
cout << to_string(mem_load) << " sec" << endl;
// Clear memory
time(&start_time);
cout << "Clearing memory...";
for (auto y = 0; y < height; y++) {
delete[] image[y];
}
delete[] image;
cout << "DONE: ";
time(&end_time);
auto mem_clear = (int)difftime(end_time, start_time);
cout << to_string(mem_clear) + " sec" << endl;
}
这是调用上述负载/清除功能的主要功能：
void main()
{
auto width = 455960;
auto height = 453994;
auto i_max = 50;
for (auto i = 0; i < i_max; i++){
SimpleLoadAndClear_Vector(width, height);
}
}
矢量版本的测试输出在50个迭代后如下如下（显然，负载/清晰的时间增加了越来越多）：
Loading image into memory...DONE: 19 sec
Clearing memory...DONE: 24 sec
Loading image into memory...DONE: 40 sec
Clearing memory...DONE: 20 sec
Loading image into memory...DONE: 27 sec
Clearing memory...DONE: 39 sec
Loading image into memory...DONE: 35 sec
Clearing memory...DONE: 24 sec
Loading image into memory...DONE: 27 sec
Clearing memory...DONE: 34 sec
Loading image into memory...DONE: 33 sec
Clearing memory...DONE: 29 sec
Loading image into memory...DONE: 27 sec
Clearing memory...DONE: 35 sec
Loading image into memory...DONE: 32 sec
Clearing memory...DONE: 33 sec
Loading image into memory...DONE: 28 sec
Clearing memory...DONE: 37 sec
Loading image into memory...DONE: 31 sec
Clearing memory...DONE: 35 sec
Loading image into memory...DONE: 30 sec
Clearing memory...DONE: 38 sec
Loading image into memory...DONE: 31 sec
Clearing memory...DONE: 38 sec
Loading image into memory...DONE: 31 sec
Clearing memory...DONE: 41 sec
Loading image into memory...DONE: 32 sec
Clearing memory...DONE: 40 sec
Loading image into memory...DONE: 33 sec
Clearing memory...DONE: 42 sec
Loading image into memory...DONE: 35 sec
Clearing memory...DONE: 43 sec
Loading image into memory...DONE: 34 sec
Clearing memory...DONE: 46 sec
Loading image into memory...DONE: 36 sec
Clearing memory...DONE: 47 sec
Loading image into memory...DONE: 35 sec
Clearing memory...DONE: 49 sec
Loading image into memory...DONE: 37 sec
Clearing memory...DONE: 50 sec
Loading image into memory...DONE: 37 sec
Clearing memory...DONE: 51 sec
Loading image into memory...DONE: 39 sec
Clearing memory...DONE: 51 sec
Loading image into memory...DONE: 39 sec
Clearing memory...DONE: 53 sec
Loading image into memory...DONE: 40 sec
Clearing memory...DONE: 52 sec
Loading image into memory...DONE: 40 sec
Clearing memory...DONE: 55 sec
Loading image into memory...DONE: 41 sec
Clearing memory...DONE: 56 sec
Loading image into memory...DONE: 41 sec
Clearing memory...DONE: 59 sec
Loading image into memory...DONE: 42 sec
Clearing memory...DONE: 59 sec
Loading image into memory...DONE: 42 sec
Clearing memory...DONE: 60 sec
Loading image into memory...DONE: 44 sec
Clearing memory...DONE: 60 sec
Loading image into memory...DONE: 44 sec
Clearing memory...DONE: 63 sec
Loading image into memory...DONE: 44 sec
Clearing memory...DONE: 63 sec
Loading image into memory...DONE: 45 sec
Clearing memory...DONE: 64 sec
Loading image into memory...DONE: 46 sec
Clearing memory...DONE: 65 sec
Loading image into memory...DONE: 45 sec
Clearing memory...DONE: 67 sec
Loading image into memory...DONE: 47 sec
Clearing memory...DONE: 69 sec
Loading image into memory...DONE: 47 sec
Clearing memory...DONE: 70 sec
Loading image into memory...DONE: 48 sec
Clearing memory...DONE: 72 sec
Loading image into memory...DONE: 48 sec
Clearing memory...DONE: 74 sec
Loading image into memory...DONE: 49 sec
Clearing memory...DONE: 74 sec
Loading image into memory...DONE: 50 sec
Clearing memory...DONE: 74 sec
Loading image into memory...DONE: 50 sec
Clearing memory...DONE: 76 sec
Loading image into memory...DONE: 51 sec
Clearing memory...DONE: 78 sec
Loading image into memory...DONE: 53 sec
Clearing memory...DONE: 78 sec
Loading image into memory...DONE: 53 sec
Clearing memory...DONE: 80 sec
Loading image into memory...DONE: 54 sec
Clearing memory...DONE: 80 sec
Loading image into memory...DONE: 54 sec
Clearing memory...DONE: 82 sec
Loading image into memory...DONE: 55 sec
Clearing memory...DONE: 91 sec
Loading image into memory...DONE: 56 sec
Clearing memory...DONE: 84 sec
Loading image into memory...DONE: 56 sec
Clearing memory...DONE: 88 sec
来自数组版本的测试输出在50次迭代后如下如下（显然，负载/清晰的时间稳定且不会增加更多）：
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 17 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 17 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 17 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 25 sec
Loading image into memory...DONE: 27 sec
Clearing memory...DONE: 17 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 17 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 17 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 17 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 25 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 25 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 25 sec
Clearing memory...DONE: 19 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 25 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 25 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 25 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 25 sec
Loading image into memory...DONE: 25 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 25 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 25 sec
Clearing memory...DONE: 17 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 26 sec
Loading image into memory...DONE: 25 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 25 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 25 sec
Loading image into memory...DONE: 25 sec
Clearing memory...DONE: 19 sec
Loading image into memory...DONE: 18 sec
Clearing memory...DONE: 25 sec
Loading image into memory...DONE: 26 sec
Clearing memory...DONE: 18 sec
问题
- 是这个窗口，当时以不好的方式处理内存操作处理巨大的std ::矢量？
- 是std :: vector，只是表现cr脚吗大量数据，设计？
- 我是否完全错过了某些东西？
- 我应该使用其他明显的STD容器（我需要从不同线程中X和Y中的索引访问图像数据）？
- 还有其他好的解释和建议的解决方案吗？
我做错了什么是我称其为图像中每个行的矢量分配器（数千次）。当首先将整个事物分配为一个向量时，然后将不同的行映射到大量向量中的正确位置时，问题已解决。
感谢@paulmckenzie的答案，将我指向正确的方向。
相关文章：
- std::向量与传递值的动态数组
- 如何在旧c++中初始化const-std向量
- 如何检查两个 std::向量在小于 O(n） 的时间复杂度内是否相等
- 当键值是 std 向量时，为什么使用 at in C++ 访问映射值如此缓慢?
- 如何初始化 std::向量的映射?
- 迭代时将指向differents类型的指针保存在std::向量中
- std::向量迭代器和调整大小/保留的奇怪/有趣行为
- yaml-cpp到std::向量迭代的怪异行为
- 故意泄漏std::向量的内存
- 调整STD ::向量的大小是否可以降低其能力
- std ::向量距离函数如何给出比.size(）更高的值
- C++ std::向量插入两个元素替代算法失败
- 错误:数字常数之前的预期无限制ID:std :: array和std ::向量大小分配
- 在我提供的此示例中，如何将2维std ::向量的逻辑更改为具有向量[row] [col] [col] [col] [co
- GUI滑块的动态数量，该数量更新具有回调中值的std ::向量
- 是一个std ::向量线程，我的目的是安全的
- 将具有std ::向量的C 函数称为Julia的输入和输出参数
- 将std ::向量作为指针参考
- STD ::向量如何调整其内部缓冲区大小
- 在C 中，是否有可能在不兼容类型的std ::向量对象之间传输不同类型的缓冲区
