KiCad PCB EDA Suite
UTF8 Class Reference

Class UTF8 is an 8 bit string that is assuredly encoded in UTF8, and supplies special conversion support to and from wxString, to and from std::string, and has non-mutating iteration over unicode characters. More...

#include <utf8.h>

Classes

class  uni_iter
 class uni_iter is a non-mutating iterator that walks through unicode code points in the UTF8 encoded string. More...
 

Public Member Functions

 UTF8 (const wxString &o)
 
 UTF8 (const char *txt)
 This is a constructor for which you could end up with non-UTF8 encoding, but that would be your fault. More...
 
 UTF8 (const wchar_t *txt)
 For use with _() function on wx 2.8. More...
 
 UTF8 (const std::string &o)
 
 UTF8 ()
 
 ~UTF8 ()
 
const char * c_str () const
 
bool empty () const
 
std::string::size_type find (char c) const
 
std::string::size_type find (char c, size_t &s) const
 
void clear ()
 
std::string::size_type length () const
 
std::string::size_type size () const
 
int compare (const std::string &s) const
 
bool operator== (const UTF8 &rhs) const
 
bool operator== (const std::string &rhs) const
 
bool operator== (const char *s) const
 
std::string::size_type find_first_of (const std::string &str, std::string::size_type pos=0) const
 
UTF8operator+= (const UTF8 &str)
 
UTF8operator+= (char ch)
 
UTF8operator+= (const char *s)
 
UTF8operator+= (unsigned w_ch)
 Append a wide (unicode) char to the UTF8 string. More...
 
UTF8operator= (const wxString &o)
 
UTF8operator= (const std::string &o)
 
UTF8operator= (const char *s)
 
UTF8operator= (char c)
 
std::string substr (size_t pos=0, size_t len=npos) const
 
 operator const std::string & () const
 
wxString wx_str () const
 
 operator wxString () const
 
std::string::const_iterator begin () const
 
std::string::const_iterator end () const
 
uni_iter ubegin () const
 Function ubegin returns a uni_iter initialized to the start of "this" UTF8 byte sequence. More...
 
uni_iter uend () const
 Function uend returns a uni_iter initialized to the end of "this" UTF8 byte sequence. More...
 

Static Public Member Functions

static int uni_forward (const unsigned char *aSequence, unsigned *aResult=NULL)
 Function uni_forward advances over a single UTF8 encoded multibyte character, capturing the unicode character as it goes, and returning the number of bytes consumed. More...
 

Static Public Attributes

static constexpr std::string::size_type npos = -1
 

Protected Attributes

std::string m_s
 

Detailed Description

Class UTF8 is an 8 bit string that is assuredly encoded in UTF8, and supplies special conversion support to and from wxString, to and from std::string, and has non-mutating iteration over unicode characters.

I've been careful to supply only conversion facilities and not try and duplicate wxString() with many member functions. There are multiple ways to create text into a std::string without the need of too many member functions:

Because this class used no virtuals, it should be possible to cast any std::string into a UTF8 using this kind of cast: (UTF8 &) without construction or copying being the effect of the cast. Be sure the source std::string holds UTF8 encoded text before you do that.

Author
Dick Hollenbeck

Definition at line 73 of file utf8.h.

Constructor & Destructor Documentation

◆ UTF8() [1/5]

UTF8::UTF8 ( const wxString &  o)

Definition at line 45 of file utf8.cpp.

45  :
46  m_s( (const char*) o.utf8_str() )
47 {
48 }
std::string m_s
Definition: utf8.h:312

◆ UTF8() [2/5]

UTF8::UTF8 ( const char *  txt)
inline

This is a constructor for which you could end up with non-UTF8 encoding, but that would be your fault.

Definition at line 81 of file utf8.h.

81  :
82  m_s( txt )
83  {
85  }
const char * c_str() const
Definition: utf8.h:107
std::string m_s
Definition: utf8.h:312
#define MAYBE_VERIFY_UTF8(x)
Definition: utf8.h:47

References c_str(), and MAYBE_VERIFY_UTF8.

◆ UTF8() [3/5]

UTF8::UTF8 ( const wchar_t *  txt)

For use with _() function on wx 2.8.

BTW _() on wx >= 2.9 returns wxString, not wchar_t* like on 2.8.

Definition at line 203 of file utf8.cpp.

204 {
205  try
206  {
207  std::vector< char > temp( wcslen( txt ) * 4 + 1 );
208  wxConvUTF8.WC2MB( temp.data(), txt, temp.size() );
209  m_s.assign( temp.data() );
210  }
211  catch(...)
212  {
213  auto string = wxSafeConvertWX2MB( txt );
214  m_s.assign( string );
215  }
216 
217  m_s.shrink_to_fit();
218 }
std::string m_s
Definition: utf8.h:312

References m_s.

◆ UTF8() [4/5]

UTF8::UTF8 ( const std::string &  o)
inline

Definition at line 91 of file utf8.h.

91  :
92  m_s( o )
93  {
95  }
const char * c_str() const
Definition: utf8.h:107
std::string m_s
Definition: utf8.h:312
#define MAYBE_VERIFY_UTF8(x)
Definition: utf8.h:47

References c_str(), and MAYBE_VERIFY_UTF8.

◆ UTF8() [5/5]

UTF8::UTF8 ( )
inline

Definition at line 97 of file utf8.h.

98  {
99  }

◆ ~UTF8()

UTF8::~UTF8 ( )
inline

Definition at line 101 of file utf8.h.

102  {
103  }

Member Function Documentation

◆ begin()

std::string::const_iterator UTF8::begin ( ) const
inline

Definition at line 196 of file utf8.h.

196 { return m_s.begin(); }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by LIB_TABLE::FormatOptions(), and KIGFX::STROKE_FONT::linesCount().

◆ c_str()

◆ clear()

void UTF8::clear ( )
inline

Definition at line 113 of file utf8.h.

113 { m_s.clear(); }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by LIB_ID::clear(), SCH_EDIT_FRAME::CreateNewText(), and RESCUE_CASE_CANDIDATE::FindRescues().

◆ compare()

int UTF8::compare ( const std::string &  s) const
inline

Definition at line 116 of file utf8.h.

116 { return m_s.compare( s ); }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by LIB_ID::compare(), SCH_REFERENCE::CompareLibName(), and SCH_REFERENCE::CompareRef().

◆ empty()

◆ end()

std::string::const_iterator UTF8::end ( ) const
inline

Definition at line 197 of file utf8.h.

197 { return m_s.end(); }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by LIB_TABLE::FormatOptions(), and KIGFX::STROKE_FONT::linesCount().

◆ find() [1/2]

std::string::size_type UTF8::find ( char  c) const
inline

Definition at line 110 of file utf8.h.

110 { return m_s.find( c ); }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by KIGFX::STROKE_FONT::Draw(), and LIB_ID::Parse().

◆ find() [2/2]

std::string::size_type UTF8::find ( char  c,
size_t &  s 
) const
inline

Definition at line 111 of file utf8.h.

111 { return m_s.find( c, s ); }
std::string m_s
Definition: utf8.h:312

References m_s.

◆ find_first_of()

std::string::size_type UTF8::find_first_of ( const std::string &  str,
std::string::size_type  pos = 0 
) const
inline

Definition at line 122 of file utf8.h.

123  {
124  return m_s.find_first_of( str, pos );
125  }
const string & str
Definition: json11.cpp:648
std::string m_s
Definition: utf8.h:312

References m_s, and str.

Referenced by okLogical(), and LIB_ID::SetLibItemName().

◆ length()

std::string::size_type UTF8::length ( ) const
inline

Definition at line 114 of file utf8.h.

114 { return m_s.length(); }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by PLACE_FILE_EXPORTER::GenPositionData(), and LIB_ID::Parse().

◆ operator const std::string &()

UTF8::operator const std::string & ( ) const
inline

Definition at line 187 of file utf8.h.

187 { return m_s; }
std::string m_s
Definition: utf8.h:312

References m_s.

◆ operator wxString()

UTF8::operator wxString ( ) const

Definition at line 57 of file utf8.cpp.

58 {
59  return wxString( c_str(), wxConvUTF8 );
60 }
const char * c_str() const
Definition: utf8.h:107

◆ operator+=() [1/4]

UTF8& UTF8::operator+= ( const UTF8 str)
inline

Definition at line 127 of file utf8.h.

128  {
129  m_s += str.m_s;
131  return (UTF8&) *this;
132  }
Class UTF8 is an 8 bit string that is assuredly encoded in UTF8, and supplies special conversion supp...
Definition: utf8.h:73
const string & str
Definition: json11.cpp:648
const char * c_str() const
Definition: utf8.h:107
std::string m_s
Definition: utf8.h:312
#define MAYBE_VERIFY_UTF8(x)
Definition: utf8.h:47

References c_str(), m_s, MAYBE_VERIFY_UTF8, and str.

◆ operator+=() [2/4]

UTF8& UTF8::operator+= ( char  ch)
inline

Definition at line 134 of file utf8.h.

135  {
136  m_s.operator+=( ch );
138  return (UTF8&) *this;
139  }
Class UTF8 is an 8 bit string that is assuredly encoded in UTF8, and supplies special conversion supp...
Definition: utf8.h:73
const char * c_str() const
Definition: utf8.h:107
std::string m_s
Definition: utf8.h:312
#define MAYBE_VERIFY_UTF8(x)
Definition: utf8.h:47

References c_str(), m_s, and MAYBE_VERIFY_UTF8.

◆ operator+=() [3/4]

UTF8& UTF8::operator+= ( const char *  s)
inline

Definition at line 141 of file utf8.h.

142  {
143  m_s.operator+=( s );
145  return (UTF8&) *this;
146  }
Class UTF8 is an 8 bit string that is assuredly encoded in UTF8, and supplies special conversion supp...
Definition: utf8.h:73
const char * c_str() const
Definition: utf8.h:107
std::string m_s
Definition: utf8.h:312
#define MAYBE_VERIFY_UTF8(x)
Definition: utf8.h:47

References c_str(), m_s, and MAYBE_VERIFY_UTF8.

◆ operator+=() [4/4]

UTF8 & UTF8::operator+= ( unsigned  w_ch)

Append a wide (unicode) char to the UTF8 string.

if this wide char is not a ASCII7 char, it will be added as a UTF8 multibyte seqence

Parameters
w_chis a UTF-16 value (can be a UTF-32 on Linux)

Definition at line 221 of file utf8.cpp.

222 {
223  if( w_ch <= 0x7F )
224  m_s.operator+=( char( w_ch ) );
225  else
226  {
227  //TODO: Remove wchar use. Replace with std::byte*
228  wchar_t wide_chr[2]; // buffer to store wide chars (UTF16) read from aText
229  wide_chr[1] = 0;
230  wide_chr[0] = w_ch;
231  UTF8 substr( wide_chr );
232  m_s += substr.m_s;
233  }
234 
235  return (UTF8&) *this;
236 }
Class UTF8 is an 8 bit string that is assuredly encoded in UTF8, and supplies special conversion supp...
Definition: utf8.h:73
std::string m_s
Definition: utf8.h:312
std::string substr(size_t pos=0, size_t len=npos) const
Definition: utf8.h:182

References m_s, and substr().

◆ operator=() [1/4]

UTF8 & UTF8::operator= ( const wxString &  o)

Definition at line 63 of file utf8.cpp.

64 {
65  m_s = (const char*) o.utf8_str();
66  return *this;
67 }
std::string m_s
Definition: utf8.h:312

References m_s.

◆ operator=() [2/4]

UTF8& UTF8::operator= ( const std::string &  o)
inline

Definition at line 159 of file utf8.h.

160  {
161  m_s = o;
163  return *this;
164  }
const char * c_str() const
Definition: utf8.h:107
std::string m_s
Definition: utf8.h:312
#define MAYBE_VERIFY_UTF8(x)
Definition: utf8.h:47

References c_str(), m_s, and MAYBE_VERIFY_UTF8.

◆ operator=() [3/4]

UTF8& UTF8::operator= ( const char *  s)
inline

Definition at line 166 of file utf8.h.

167  {
168  m_s = s;
170  return *this;
171  }
const char * c_str() const
Definition: utf8.h:107
std::string m_s
Definition: utf8.h:312
#define MAYBE_VERIFY_UTF8(x)
Definition: utf8.h:47

References c_str(), m_s, and MAYBE_VERIFY_UTF8.

◆ operator=() [4/4]

UTF8& UTF8::operator= ( char  c)
inline

Definition at line 173 of file utf8.h.

174  {
175  m_s = c;
177  return *this;
178  }
const char * c_str() const
Definition: utf8.h:107
std::string m_s
Definition: utf8.h:312
#define MAYBE_VERIFY_UTF8(x)
Definition: utf8.h:47

References c_str(), m_s, and MAYBE_VERIFY_UTF8.

◆ operator==() [1/3]

bool UTF8::operator== ( const UTF8 rhs) const
inline

Definition at line 118 of file utf8.h.

118 { return m_s == rhs.m_s; }
std::string m_s
Definition: utf8.h:312

References m_s.

◆ operator==() [2/3]

bool UTF8::operator== ( const std::string &  rhs) const
inline

Definition at line 119 of file utf8.h.

119 { return m_s == rhs; }
std::string m_s
Definition: utf8.h:312

References m_s.

◆ operator==() [3/3]

bool UTF8::operator== ( const char *  s) const
inline

Definition at line 120 of file utf8.h.

120 { return m_s == s; }
std::string m_s
Definition: utf8.h:312

References m_s.

◆ size()

std::string::size_type UTF8::size ( ) const
inline

◆ substr()

std::string UTF8::substr ( size_t  pos = 0,
size_t  len = npos 
) const
inline

Definition at line 182 of file utf8.h.

183  {
184  return m_s.substr( pos, len );
185  }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by KIGFX::STROKE_FONT::Draw(), operator+=(), LIB_ID::Parse(), and LIB_ID::SetLibItemName().

◆ ubegin()

uni_iter UTF8::ubegin ( ) const
inline

Function ubegin returns a uni_iter initialized to the start of "this" UTF8 byte sequence.

Definition at line 285 of file utf8.h.

286  {
287  return uni_iter( m_s.data() );
288  }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by KIGFX::OPENGL_GAL::BitmapText(), KIGFX::OPENGL_GAL::computeBitmapTextSize(), KIGFX::STROKE_FONT::ComputeStringBoundaryLimits(), KIGFX::STROKE_FONT::drawSingleLineText(), and LIB_ID::FixIllegalChars().

◆ uend()

uni_iter UTF8::uend ( ) const
inline

Function uend returns a uni_iter initialized to the end of "this" UTF8 byte sequence.

Definition at line 294 of file utf8.h.

295  {
296  return uni_iter( m_s.data() + m_s.size() );
297  }
std::string m_s
Definition: utf8.h:312

References m_s.

Referenced by KIGFX::OPENGL_GAL::BitmapText(), KIGFX::OPENGL_GAL::computeBitmapTextSize(), KIGFX::STROKE_FONT::ComputeStringBoundaryLimits(), KIGFX::STROKE_FONT::drawSingleLineText(), and LIB_ID::FixIllegalChars().

◆ uni_forward()

int UTF8::uni_forward ( const unsigned char *  aSequence,
unsigned *  aResult = NULL 
)
static

Function uni_forward advances over a single UTF8 encoded multibyte character, capturing the unicode character as it goes, and returning the number of bytes consumed.

Parameters
aSequenceis the UTF8 byte sequence, must be aligned on start of character.
aResultis where to put the unicode character, and may be NULL if no interest.
Returns
int - the count of bytes consumed.

Definition at line 73 of file utf8.cpp.

74 {
75  unsigned ch = *aSequence;
76 
77  if( ch < 0x80 )
78  {
79  if( aResult )
80  *aResult = ch;
81  return 1;
82  }
83 
84  const unsigned char* s = aSequence;
85 
86  static const unsigned char utf8_len[] = {
87  // Map encoded prefix byte to sequence length. Zero means
88  // illegal prefix. See RFC 3629 for details
89  /*
90  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 00-0F
91  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
92  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
93  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
94  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
95  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
96  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
97  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 70-7F
98  */
99  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 80-8F
100  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
101  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
102  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // B0-BF
103  0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, // C0-C1 + C2-CF
104  2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, // D0-DF
105  3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, // E0-EF
106  4, 4, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 // F0-F4 + F5-FF
107  };
108 
109  int len = utf8_len[ *s - 0x80 /* top half of table is missing */ ];
110 
111  switch( len )
112  {
113  default:
114  case 0:
115  THROW_IO_ERROR( "invalid start byte" );
116  break;
117 
118  case 2:
119  if( ( s[1] & 0xc0 ) != 0x80 )
120  {
121  THROW_IO_ERROR( "invalid continuation byte" );
122  }
123 
124  ch = ((s[0] & 0x1f) << 6) +
125  ((s[1] & 0x3f) << 0);
126 
127  // assert( ch > 0x007F && ch <= 0x07FF );
128  break;
129 
130  case 3:
131  if( (s[1] & 0xc0) != 0x80 ||
132  (s[2] & 0xc0) != 0x80 ||
133  (s[0] == 0xE0 && s[1] < 0xA0)
134  // || (s[0] == 0xED && s[1] > 0x9F)
135  )
136  {
137  THROW_IO_ERROR( "invalid continuation byte" );
138  }
139 
140  ch = ((s[0] & 0x0f) << 12) +
141  ((s[1] & 0x3f) << 6 ) +
142  ((s[2] & 0x3f) << 0 );
143 
144  // assert( ch > 0x07FF && ch <= 0xFFFF );
145  break;
146 
147  case 4:
148  if( (s[1] & 0xc0) != 0x80 ||
149  (s[2] & 0xc0) != 0x80 ||
150  (s[3] & 0xc0) != 0x80 ||
151  (s[0] == 0xF0 && s[1] < 0x90) ||
152  (s[0] == 0xF4 && s[1] > 0x8F) )
153  {
154  THROW_IO_ERROR( "invalid continuation byte" );
155  }
156 
157  ch = ((s[0] & 0x7) << 18) +
158  ((s[1] & 0x3f) << 12) +
159  ((s[2] & 0x3f) << 6 ) +
160  ((s[3] & 0x3f) << 0 );
161 
162  // assert( ch > 0xFFFF && ch <= 0x10ffff );
163  break;
164  }
165 
166  if( aResult )
167  *aResult = ch;
168 
169  return len;
170 }
#define THROW_IO_ERROR(msg)

References THROW_IO_ERROR.

Referenced by IsUTF8(), UTF8::uni_iter::operator *(), UTF8::uni_iter::operator++(), and UTF8::uni_iter::operator->().

◆ wx_str()

Member Data Documentation

◆ m_s

◆ npos

constexpr std::string::size_type UTF8::npos = -1
static

Definition at line 155 of file utf8.h.

Referenced by KIGFX::STROKE_FONT::Draw(), and LIB_ID::Parse().


The documentation for this class was generated from the following files: