KiCad PCB EDA Suite
GITHUB_GETLIBLIST Class Reference

Class GITHUB_GETLIBLIST implements a portion of pcbnew's PLUGIN interface to provide read only access to a github repo to extract pretty footprints library list, in json format, or extract 3D shapes libraries list (.3dshapes folders) and download the 3d shapes files. More...

#include <github_getliblist.h>

Public Member Functions

bool GetFootprintLibraryList (wxArrayString &aList)
 Fills aList by the name of footprint libraries found on the github repo. More...
 
bool Get3DshapesLibsList (wxArrayString *aList, bool(*aFilter)(const wxString &aData))
 Fills aList by the URL of libraries found on the github repo. More...
 
std::string & GetBuffer ()
 
void ClearBuffer ()
 Clear the buffer which stores all the downloaded raw data. More...
 
void SetLibraryExt (const wxString &aExt)
 The library names are expecting ending by .pretty SetLibraryExt set the extension to aExt. More...
 
 GITHUB_GETLIBLIST (const wxString &aRepoURL)
 
 ~GITHUB_GETLIBLIST ()
 

Protected Member Functions

bool repoURL2listURL (const wxString &aRepoURL, std::string *aFullURLCommand, int aItemCountMax, int aPage=1)
 Function repoURL2listURL translates a repo URL to the URL name which gives the state of repos URL as commonly seen on github.com. More...
 
bool remoteGetJSON (const std::string &aFullURLCommand, wxString *aMsgError)
 Function remoteGetJSON Download a json text from a github repo. More...
 

Protected Attributes

wxString m_github_path
 Something like https://api.github.com/orgs/KiCad. More...
 
std::string m_image
 image of the downloaded data in its entirety. More...
 
wxString m_repoURL
 the URL of the Github repo More...
 
wxString m_libs_ext
 the extension of the name of libraries (default = .pretty) More...
 
char m_option_string [64]
 option for transfert type, like "application/json" More...
 

Detailed Description

Class GITHUB_GETLIBLIST implements a portion of pcbnew's PLUGIN interface to provide read only access to a github repo to extract pretty footprints library list, in json format, or extract 3D shapes libraries list (.3dshapes folders) and download the 3d shapes files.

To extract pretty footprints library list, this plugin simply reads in a zip file of the repo and unzips it from RAM as needed. Therefore this "Github" plugin is read only for accessing remote at https:////api.github.com//orgs//KiCad//repos

To extract 3D shapes libraries list (.3dshapes folders) we cannot use api.github.com to read this list, becuse it is in a subdirectory of https://github.com/KiCad. The corresponding html page of the server is read, and URLs of all .3dshapes folders are extracted. files are then read from https://raw.githubusercontent.com/<lib path>="">, but not zipped because they are not accessible in zipped form.

Definition at line 47 of file github_getliblist.h.

Constructor & Destructor Documentation

GITHUB_GETLIBLIST::GITHUB_GETLIBLIST ( const wxString &  aRepoURL)

Definition at line 53 of file github_getliblist.cpp.

References m_libs_ext, m_option_string, and m_repoURL.

54 {
55  m_repoURL = aRepoURL;
56  m_libs_ext = wxT( ".pretty" );
57  strcpy( m_option_string, "application/json" );
58 }
char m_option_string[64]
option for transfert type, like "application/json"
wxString m_repoURL
the URL of the Github repo
wxString m_libs_ext
the extension of the name of libraries (default = .pretty)
GITHUB_GETLIBLIST::~GITHUB_GETLIBLIST ( )
inline

Definition at line 86 of file github_getliblist.h.

86 {}

Member Function Documentation

void GITHUB_GETLIBLIST::ClearBuffer ( )
inline

Clear the buffer which stores all the downloaded raw data.

Definition at line 75 of file github_getliblist.h.

References m_image.

Referenced by GetFootprintLibraryList().

75 { m_image.clear(); }
std::string m_image
image of the downloaded data in its entirety.
bool GITHUB_GETLIBLIST::Get3DshapesLibsList ( wxArrayString *  aList,
bool(*)(const wxString &aData)  aFilter 
)

Fills aList by the URL of libraries found on the github repo.

Parameters
aList= a reference to a wxArrayString to fill with names
aFilter(const wxString& aData ) = a callback funtion to to filter URLs to put in aList. If NULL, no URL will be stored in aList

Definition at line 61 of file github_getliblist.cpp.

References GetBuffer(), m_option_string, m_repoURL, HTML_LINK_PARSER::ParseLinks(), and remoteGetJSON().

Referenced by WIZARD_3DSHAPE_LIBS_DOWNLOADER::downloadOneLib(), and WIZARD_3DSHAPE_LIBS_DOWNLOADER::getLibsListGithub().

63 {
64  std::string fullURLCommand;
65 
66  strcpy( m_option_string, "text/html" );
67 
68  wxString repoURL = m_repoURL;
69 
70  wxString errorMsg;
71 
72  fullURLCommand = repoURL.utf8_str();
73  bool success = remoteGetJSON( fullURLCommand, &errorMsg );
74 
75  if( !success )
76  {
77  wxMessageBox( errorMsg );
78  return false;
79  }
80 
81  if( aFilter && aList )
82  {
83  //Convert m_image (std::string) to a wxString for HTML_LINK_PARSER
84  wxString buffer( GetBuffer() );
85 
86  HTML_LINK_PARSER html_parser( buffer, *aList );
87  html_parser.ParseLinks( aFilter );
88  }
89 
90  return true;
91 }
char m_option_string[64]
option for transfert type, like "application/json"
bool remoteGetJSON(const std::string &aFullURLCommand, wxString *aMsgError)
Function remoteGetJSON Download a json text from a github repo.
wxString m_repoURL
the URL of the Github repo
std::string & GetBuffer()
std::string& GITHUB_GETLIBLIST::GetBuffer ( )
inline
Returns
the buffer which stores all the downloaded raw data

Definition at line 70 of file github_getliblist.h.

References m_image.

Referenced by WIZARD_3DSHAPE_LIBS_DOWNLOADER::downloadOneLib(), Get3DshapesLibsList(), and GetFootprintLibraryList().

70 { return m_image; }
std::string m_image
image of the downloaded data in its entirety.
bool GITHUB_GETLIBLIST::GetFootprintLibraryList ( wxArrayString &  aList)

Fills aList by the name of footprint libraries found on the github repo.

Definition at line 94 of file github_getliblist.cpp.

References ClearBuffer(), Format(), GetBuffer(), GetChars(), m_libs_ext, m_option_string, m_repoURL, QUOTE, remoteGetJSON(), and repoURL2listURL().

95 {
96  std::string fullURLCommand;
97  int page = 1;
98  int itemCountMax = 99; // Do not use a valu > 100, it does not work
99 
100  strcpy( m_option_string, "application/json" );
101 
102  // Github max items returned is 100 per page
103 
104  if( !repoURL2listURL( m_repoURL, &fullURLCommand, itemCountMax, page ) )
105  {
106  wxString msg = wxString::Format( _( "malformed URL:\n'%s'" ), GetChars( m_repoURL ) );
107  wxMessageBox( msg );
108  return false;
109  }
110 
111  // The URL lib names are relative to the server name.
112  // so add the server name to them.
113  wxURI repo( m_repoURL );
114  wxString urlPrefix = repo.GetScheme() + wxT( "://" ) + repo.GetServer() + wxT( "/" );
115 
116  wxString errorMsg;
117  const char sep = ','; // Separator fields, in json returned file
118  wxString tmp;
119  int items_count_per_page = 0;
120  std::string& json_image = GetBuffer();
121 
122  while( 1 )
123  {
124  bool success = remoteGetJSON( fullURLCommand, &errorMsg );
125 
126  if( !success )
127  {
128  wxMessageBox( errorMsg );
129  return false;
130  }
131 
132 
133  for( unsigned ii = 0; ii < json_image.size(); ii++ )
134  {
135  if( json_image[ii] == sep || ii == json_image.size() - 1 )
136  {
137  if( tmp.StartsWith( wxT( "\"full_name\"" ) ) )
138  {
139  #define QUOTE '\"'
140  // Remove useless quotes:
141  if( tmp[tmp.Length() - 1] == QUOTE )
142  tmp.RemoveLast();
143 
144  if( tmp.EndsWith( m_libs_ext ) )
145  {
146  aList.Add( tmp.AfterLast( ':' ) );
147  int idx = aList.GetCount() - 1;
148 
149  if( aList[idx][0] == QUOTE )
150  aList[idx].Remove( 0, 1 );
151 
152  aList[idx].Prepend( urlPrefix );
153  }
154 
155  items_count_per_page++;
156  }
157 
158  tmp.Clear();
159  }
160  else
161  tmp << json_image[ii];
162  }
163 
164  if( items_count_per_page >= itemCountMax )
165  {
166  page++;
167  repoURL2listURL( m_repoURL, &fullURLCommand, itemCountMax, page );
168  items_count_per_page = 0;
169  ClearBuffer();
170  }
171  else
172  break;
173  }
174 
175  aList.Sort();
176  return true;
177 }
char m_option_string[64]
option for transfert type, like "application/json"
bool repoURL2listURL(const wxString &aRepoURL, std::string *aFullURLCommand, int aItemCountMax, int aPage=1)
Function repoURL2listURL translates a repo URL to the URL name which gives the state of repos URL as ...
#define QUOTE
static const wxChar * GetChars(const wxString &s)
Function GetChars returns a wxChar* to the actual wxChar* data within a wxString, and is helpful for ...
Definition: macros.h:92
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, CPTREE &aTree)
Function Format outputs a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:205
bool remoteGetJSON(const std::string &aFullURLCommand, wxString *aMsgError)
Function remoteGetJSON Download a json text from a github repo.
wxString m_repoURL
the URL of the Github repo
wxString m_libs_ext
the extension of the name of libraries (default = .pretty)
void ClearBuffer()
Clear the buffer which stores all the downloaded raw data.
std::string & GetBuffer()
bool GITHUB_GETLIBLIST::remoteGetJSON ( const std::string &  aFullURLCommand,
wxString *  aMsgError 
)
protected

Function remoteGetJSON Download a json text from a github repo.

The text image is received into the m_input_stream.

Parameters
aFullURLCommandthe full command, i.e. the url with options like "https://api.github.com/users/KiCad/repos?per_page=100?page=1"
aMsgErrora pointer to a wxString which can store an error message
Returns
true if OK, false if error (which an error message in *aMsgError

Definition at line 214 of file github_getliblist.cpp.

References FROM_UTF8(), KICAD_CURL_EASY::GetBuffer(), m_image, m_option_string, KICAD_CURL_EASY::Perform(), KICAD_CURL_EASY::SetFollowRedirects(), KICAD_CURL_EASY::SetHeader(), KICAD_CURL_EASY::SetURL(), KICAD_CURL_EASY::SetUserAgent(), StrPrintf(), TO_UTF8, and IO_ERROR::What().

Referenced by Get3DshapesLibsList(), and GetFootprintLibraryList().

215 {
216  KICAD_CURL_EASY kcurl;
217 
218  wxLogDebug( wxT( "Attempting to download: " ) + aFullURLCommand );
219 
220  kcurl.SetURL( aFullURLCommand );
221  kcurl.SetUserAgent( "http://kicad-pcb.org" );
222  kcurl.SetHeader( "Accept", m_option_string );
223  kcurl.SetFollowRedirects( true );
224 
225  try
226  {
227  kcurl.Perform();
228  m_image = kcurl.GetBuffer();
229  return true;
230  }
231  catch( const IO_ERROR& ioe )
232  {
233  if( aMsgError )
234  {
235  UTF8 fmt( _( "Error fetching JSON data from URL '%s'.\nReason: '%s'" ) );
236 
237  std::string msg = StrPrintf( fmt.c_str(),
238  aFullURLCommand.c_str(),
239  TO_UTF8( ioe.What() ) );
240 
241  *aMsgError = FROM_UTF8( msg.c_str() );
242  }
243  return false;
244  }
245 }
Class UTF8 is an 8 bit std::string that is assuredly encoded in UTF8, and supplies special conversion...
Definition: utf8.h:53
bool SetUserAgent(const std::string &aAgent)
Function SetUserAgent sets the request user agent.
void Perform()
Function perform equivalent to curl_easy_perform.
bool SetFollowRedirects(bool aFollow)
Function SetFollowRedirects enables the following of HTTP(s) and other redirects, by default curl doe...
static wxString FROM_UTF8(const char *cstring)
function FROM_UTF8 converts a UTF8 encoded C string to a wxString for all wxWidgets build modes...
Definition: macros.h:53
Class KICAD_CURL_EASY wrapper interface around the curl_easy API.
char m_option_string[64]
option for transfert type, like "application/json"
int StrPrintf(std::string *result, const char *format,...)
Function StrPrintf is like sprintf() but the output is appended to a std::string instead of to a char...
Definition: richio.cpp:75
#define TO_UTF8(wxstring)
Macro TO_UTF8 converts a wxString to a UTF8 encoded C string for all wxWidgets build modes...
Definition: macros.h:47
const std::string & GetBuffer()
Function GetBuffer returns a const reference to the recevied data buffer.
std::string m_image
image of the downloaded data in its entirety.
bool SetURL(const std::string &aURL)
Function SetURL sets the request URL.
virtual const wxString What() const
A composite of Problem() and Where()
Definition: exceptions.cpp:33
void SetHeader(const std::string &aName, const std::string &aValue)
Function SetHeader sets an arbitrary header for the HTTP(s) request.
Struct IO_ERROR is a class used to hold an error message and may be used when throwing exceptions con...
Definition: ki_exception.h:47
bool GITHUB_GETLIBLIST::repoURL2listURL ( const wxString &  aRepoURL,
std::string *  aFullURLCommand,
int  aItemCountMax,
int  aPage = 1 
)
protected

Function repoURL2listURL translates a repo URL to the URL name which gives the state of repos URL as commonly seen on github.com.

Parameters
aRepoURLpoints to the base of the repo.
aFullURLCommandis URL the full URL command (URL+options).
aItemCountMaxis the max item count in a page, and is 100 for github repo.
aPageis the page number, if there are more than one page in repo.
Returns
bool - true if aRepoULR was parseable, else false

Definition at line 180 of file github_getliblist.cpp.

References Format().

Referenced by GetFootprintLibraryList().

183 {
184  // aListURL is e.g. "https://api.github.com/orgs/KiCad/repos"
185  // or "https://api.github.com/users/KiCad/repos"
186  // aRepoURL is e.g. "https://github.com/KiCad"
187  // Github has a default pagination set to 30 items.
188  // but allows up to 100 items max if we add the "?per_page=100" option
189 
190  wxURI repo( aRepoURL );
191 
192  if( repo.HasServer() && repo.HasPath() )
193  {
194  // goal: "https://api.github.com/users/KiCad"
195  // or "https://api.github.com/orgs/KiCad"
196  // "https://api.github.com/users/..." works both for orgs and users
197  // if we just retrieve the .pretty list
198  wxString target_url( wxT( "https://api.github.com/users" ) );
199  target_url += repo.GetPath();
200  target_url += wxT( "/repos" );
201 
202  // Github has a default pagination set to 30 items.
203  // but allows up to 100 items max. Use this limit
204  target_url += wxString::Format( "?per_page=%d&page=%d", aItemCountMax, aPage );
205 
206  *aFullURLCommand = target_url.utf8_str();
207  return true;
208  }
209 
210  return false;
211 }
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, CPTREE &aTree)
Function Format outputs a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:205
void GITHUB_GETLIBLIST::SetLibraryExt ( const wxString &  aExt)
inline

The library names are expecting ending by .pretty SetLibraryExt set the extension to aExt.

Definition at line 81 of file github_getliblist.h.

References m_libs_ext.

81 { m_libs_ext = aExt; }
wxString m_libs_ext
the extension of the name of libraries (default = .pretty)

Member Data Documentation

wxString GITHUB_GETLIBLIST::m_github_path
protected

Something like https://api.github.com/orgs/KiCad.

Definition at line 116 of file github_getliblist.h.

std::string GITHUB_GETLIBLIST::m_image
protected

image of the downloaded data in its entirety.

Definition at line 117 of file github_getliblist.h.

Referenced by ClearBuffer(), GetBuffer(), and remoteGetJSON().

wxString GITHUB_GETLIBLIST::m_libs_ext
protected

the extension of the name of libraries (default = .pretty)

Definition at line 119 of file github_getliblist.h.

Referenced by GetFootprintLibraryList(), GITHUB_GETLIBLIST(), and SetLibraryExt().

char GITHUB_GETLIBLIST::m_option_string[64]
protected

option for transfert type, like "application/json"

Definition at line 120 of file github_getliblist.h.

Referenced by Get3DshapesLibsList(), GetFootprintLibraryList(), GITHUB_GETLIBLIST(), and remoteGetJSON().

wxString GITHUB_GETLIBLIST::m_repoURL
protected

the URL of the Github repo

Definition at line 118 of file github_getliblist.h.

Referenced by Get3DshapesLibsList(), GetFootprintLibraryList(), and GITHUB_GETLIBLIST().


The documentation for this class was generated from the following files: