如何更改谷歌原型中的缓冲区限制?

How to change the buffer limit in Google's protobuf?

本文关键字:缓冲区 何更改 谷歌 原型      更新时间:2023-10-16

当我试图解析一条大消息时,我收到了这个警告和一个错误。我知道超过64MB,这是默认的限制。我正在使用消息。ParseFromIstream现在。有人知道要访问CodedInputStream对象来调用SetTotalBytesLimit函数吗?或者其他解决这个问题的方法?

正在读取危险的大型协议消息。如果消息变成如果大于67108864字节,则出于安全考虑将停止解析原因。要增加限制(或禁用这些警告),请参阅中的CodedInputStream::SetTotalBytesLimit()google/protobuf/io/coded_stream.h

正确的解决方案:您应该尝试限制protobuf消息的大小。请参阅:https://developers.google.com/protocol-buffers/docs/techniques#streaming

快速而肮脏(不建议阅读)的方法:在protobuf库源的文件coded_stream.h中,更改kDefaultTotalBytesLimitkDefaultTotalBytesWarningThreshold的值,重新编译并重新安装。

只要阅读错误已经告诉你的函数的文档,就会回答这个问题:

提示:如果您正在阅读这篇文章,因为您的程序正在打印警告危险的大型协议消息,您可能会感到困惑关于下一步该做什么。最好的选择是改变你的设计不需要过大的消息。例如,尝试将文件格式设计为由许多小消息组成,而不是一个大的。如果这不可行,则需要增加限度不过,您的代码可能从未构造可以设置限制的CodedInputStream。你可能会分析消息,方法是调用Message::ParseFromString()。在这个在这种情况下,您将需要更改代码来构造某种排序的ZeroCopyInputStream(例如ArrayInputStream),构造CodedInputStream,然后调用消息::改为ParseFromCodedStream()。然后你可以调整限度是的,这是更多的工作,但你正在做一些不同寻常的事情。

此外,遵循建议的第一部分并重新设计应用程序可能是一个非常好的主意。

这是代码(google/protobuf/io/coded_stream.h)中的一条注释,它为那些想知道他们谈论的安全原因是什么的人设置了消息限制。在我的情况下,我无法修改我的应用程序的工作方式,所以我必须更改此限制。

这个线程已经很古老了,但最近深度学习引起了人们的注意,Caffe图书馆使用了Protobuf,所以也许更多的人会偶然发现这一点。我必须用Caffe做神经网络的事情,即使是最小的批量,整个网络也占用了很多内存。

// Total Bytes Limit -----------------------------------------------
// To prevent malicious users from sending excessively large messages
// and causing integer overflows or memory exhaustion, CodedInputStream
// imposes a hard limit on the total number of bytes it will read.
// Sets the maximum number of bytes that this CodedInputStream will read
// before refusing to continue.  To prevent integer overflows in the
// protocol buffers implementation, as well as to prevent servers from
// allocating enormous amounts of memory to hold parsed messages, the
// maximum message length should be limited to the shortest length that
// will not harm usability.  The theoretical shortest message that could
// cause integer overflows is 512MB.  The default limit is 64MB.  Apps
// should set shorter limits if possible.  If warning_threshold is not -1,
// a warning will be printed to stderr after warning_threshold bytes are
// read.  For backwards compatibility all negative values get squashed to -1,
// as other negative values might have special internal meanings.
// An error will always be printed to stderr if the limit is reached.
//
// This is unrelated to PushLimit()/PopLimit().
//
// Hint:  If you are reading this because your program is printing a
//   warning about dangerously large protocol messages, you may be
//   confused about what to do next.  The best option is to change your
//   design such that excessively large messages are not necessary.
//   For example, try to design file formats to consist of many small
//   messages rather than a single large one.  If this is infeasible,
//   you will need to increase the limit.  Chances are, though, that
//   your code never constructs a CodedInputStream on which the limit
//   can be set.  You probably parse messages by calling things like
//   Message::ParseFromString().  In this case, you will need to change
//   your code to instead construct some sort of ZeroCopyInputStream
//   (e.g. an ArrayInputStream), construct a CodedInputStream around
//   that, then call Message::ParseFromCodedStream() instead.  Then
//   you can adjust the limit.  Yes, it's more work, but you're doing
//   something unusual.