将句子标记为单词，考虑特殊字符

Tokenize sentence into words, considering special characters

本文关键字：特殊字符 单词 句子 记为      更新时间：2023-10-16

我有一个函数，它接收一个句子，并根据空格"标记成单词" "。现在，我想改进功能以消除一些特殊字符，例如：

I am a boy.   => {I, am, a, boy}, no period after "boy"
I said :"are you ok?"  => {I, said, are, you, ok}, no question and quotation mark

原来的功能在这里，我该如何改进它？

void Tokenize(const string& str, vector<string>& tokens, const string& delimiters = " ")
{
    string::size_type lastPos = str.find_first_not_of(delimiters, 0);
    string::size_type pos = str.find_first_of(delimiters, lastPos);
    while (string::npos != pos || string::npos != lastPos)
    {
        tokens.push_back(str.substr(lastPos, pos - lastPos));
        lastPos = str.find_first_not_of(delimiters, pos);
        pos = str.find_first_of(delimiters, lastPos);
    }
}
您可以使用

std::regex .在那里，您可以搜索任何您想要的内容，然后将结果放入向量中。这很简单。

看：

#include <iostream>
#include <string>
#include <algorithm>
#include <vector>
#include <regex>
// Our test data (raw string). So, containing also " and so on
std::string testData(R"#(I said :"are you ok?")#");
std::regex re(R"#((bw+b,?))#");
int main(void)
{
    // Define the variable id as vector of string and use the range constructor to read the test data and tokenize it
    std::vector<std::string> id{ std::sregex_token_iterator(testData.begin(), testData.end(), re, 1), std::sregex_token_iterator() };
    // For debug output. Print complete vector to std::cout
    std::copy(id.begin(), id.end(), std::ostream_iterator<std::string>(std::cout, " "));
    return 0;
}
相关文章：