KiCad PCB EDA Suite
GITHUB_GETLIBLIST Class Reference

Class GITHUB_GETLIBLIST implements a portion of pcbnew's PLUGIN interface to provide read only access to a github repo to extract pretty footprints library list, in json format, or extract 3D shapes libraries list (.3dshapes folders) and download the 3d shapes files. More...

#include <github_getliblist.h>

Public Member Functions

bool GetFootprintLibraryList (wxArrayString &aList)
 Fills aList by the name of footprint libraries found on the github repo. More...
 
bool Get3DshapesLibsList (wxArrayString *aList, bool(*aFilter)(const wxString &aData))
 Fills aList by the URL of libraries found on the github repo. More...
 
std::string & GetBuffer ()
 
void ClearBuffer ()
 Clear the buffer which stores all the downloaded raw data. More...
 
void SetLibraryExt (const wxString &aExt)
 The library names are expecting ending by .pretty SetLibraryExt set the extension to aExt. More...
 
 GITHUB_GETLIBLIST (const wxString &aRepoURL)
 
 ~GITHUB_GETLIBLIST ()
 

Protected Member Functions

bool repoURL2listURL (const wxString &aRepoURL, std::string *aFullURLCommand, int aItemCountMax, int aPage=1)
 Function repoURL2listURL translates a repo URL to the URL name which gives the state of repos URL as commonly seen on github.com. More...
 
bool remoteGetJSON (const std::string &aFullURLCommand, wxString *aMsgError)
 Function remoteGetJSON Download a json text from a github repo. More...
 

Protected Attributes

wxString m_github_path
 Something like https://api.github.com/orgs/KiCad. More...
 
std::string m_image
 image of the downloaded data in its entirety. More...
 
wxString m_repoURL
 the URL of the Github repo More...
 
wxString m_libs_ext
 the extension of the name of libraries (default = .pretty) More...
 
char m_option_string [64]
 option for transfert type, like "application/json" More...
 

Detailed Description

Class GITHUB_GETLIBLIST implements a portion of pcbnew's PLUGIN interface to provide read only access to a github repo to extract pretty footprints library list, in json format, or extract 3D shapes libraries list (.3dshapes folders) and download the 3d shapes files.

To extract pretty footprints library list, this plugin simply reads in a zip file of the repo and unzips it from RAM as needed. Therefore this "Github" plugin is read only for accessing remote at "https://api.github.com/orgs/KiCad/repos"

To extract 3D shapes libraries list (.3dshapes folders) we cannot use api.github.com to read this list, because it is in a subdirectory of "https://github.com/KiCad". The corresponding html page of the server is read, and URLs of all .3dshapes folders are extracted. files are then read from "https://raw.githubusercontent.com/<lib path>", but not zipped because they are not accessible in zipped form.

Definition at line 47 of file github_getliblist.h.

Constructor & Destructor Documentation

GITHUB_GETLIBLIST::GITHUB_GETLIBLIST ( const wxString &  aRepoURL)

Definition at line 53 of file github_getliblist.cpp.

References m_libs_ext, m_option_string, and m_repoURL.

54 {
55  m_repoURL = aRepoURL;
56  m_libs_ext = wxT( ".pretty" );
57  strcpy( m_option_string, "application/json" );
58 }
char m_option_string[64]
option for transfert type, like "application/json"
wxString m_repoURL
the URL of the Github repo
wxString m_libs_ext
the extension of the name of libraries (default = .pretty)
GITHUB_GETLIBLIST::~GITHUB_GETLIBLIST ( )
inline

Definition at line 86 of file github_getliblist.h.

86 {}

Member Function Documentation

void GITHUB_GETLIBLIST::ClearBuffer ( )
inline

Clear the buffer which stores all the downloaded raw data.

Definition at line 75 of file github_getliblist.h.

References m_image.

Referenced by GetFootprintLibraryList().

75 { m_image.clear(); }
std::string m_image
image of the downloaded data in its entirety.
bool GITHUB_GETLIBLIST::Get3DshapesLibsList ( wxArrayString *  aList,
bool(*)(const wxString &aData)  aFilter 
)

Fills aList by the URL of libraries found on the github repo.

Parameters
aList= a reference to a wxArrayString to fill with names
aFilter(const wxString& aData ) = a callback funtion to to filter URLs to put in aList. If NULL, no URL will be stored in aList

Definition at line 61 of file github_getliblist.cpp.

References GetBuffer(), m_option_string, m_repoURL, HTML_LINK_PARSER::ParseLinks(), and remoteGetJSON().

Referenced by WIZARD_3DSHAPE_LIBS_DOWNLOADER::downloadOneLib(), and WIZARD_3DSHAPE_LIBS_DOWNLOADER::getLibsListGithub().

63 {
64  std::string fullURLCommand;
65 
66  strcpy( m_option_string, "text/html" );
67 
68  wxString errorMsg;
69 
70  wxString repoURL = m_repoURL;
71  fullURLCommand = repoURL.utf8_str();
72  bool success = remoteGetJSON( fullURLCommand, &errorMsg );
73 
74  if( !success )
75  {
76  wxMessageBox( errorMsg );
77  return false;
78  }
79 
80  if( aFilter && aList )
81  {
82  //Convert m_image (std::string) to a wxString for HTML_LINK_PARSER
83  wxString buffer( GetBuffer() );
84 
85  HTML_LINK_PARSER html_parser( buffer, *aList );
86  html_parser.ParseLinks( aFilter );
87  }
88 
89  return true;
90 }
char m_option_string[64]
option for transfert type, like "application/json"
bool remoteGetJSON(const std::string &aFullURLCommand, wxString *aMsgError)
Function remoteGetJSON Download a json text from a github repo.
wxString m_repoURL
the URL of the Github repo
std::string & GetBuffer()
std::string& GITHUB_GETLIBLIST::GetBuffer ( )
inline
Returns
the buffer which stores all the downloaded raw data

Definition at line 70 of file github_getliblist.h.

References m_image.

Referenced by WIZARD_3DSHAPE_LIBS_DOWNLOADER::downloadOneLib(), Get3DshapesLibsList(), and GetFootprintLibraryList().

70 { return m_image; }
std::string m_image
image of the downloaded data in its entirety.
bool GITHUB_GETLIBLIST::GetFootprintLibraryList ( wxArrayString &  aList)

Fills aList by the name of footprint libraries found on the github repo.

Definition at line 93 of file github_getliblist.cpp.

References ClearBuffer(), Format(), GetBuffer(), GetChars(), m_libs_ext, m_option_string, m_repoURL, QUOTE, remoteGetJSON(), and repoURL2listURL().

94 {
95  std::string fullURLCommand;
96  int page = 1;
97  int itemCountMax = 99; // Do not use a value >= 100, it does not work
98 
99  strcpy( m_option_string, "application/json" );
100 
101  // Github max items returned is 100 per page
102 
103  if( !repoURL2listURL( m_repoURL, &fullURLCommand, itemCountMax, page ) )
104  {
105  wxString msg = wxString::Format( _( "malformed URL:\n'%s'" ), GetChars( m_repoURL ) );
106  wxMessageBox( msg );
107  return false;
108  }
109 
110  // The URL lib names are relative to the server name.
111  // so add the server name to them.
112  wxURI repo( m_repoURL );
113  wxString urlPrefix = repo.GetScheme() + wxT( "://" ) + repo.GetServer() + wxT( "/" );
114 
115  wxString errorMsg;
116  const char sep = ','; // Separator fields, in json returned file
117  wxString tmp;
118  int items_count_per_page = 0;
119  std::string& json_image = GetBuffer();
120 
121  while( 1 )
122  {
123  bool success = remoteGetJSON( fullURLCommand, &errorMsg );
124 
125  if( !success )
126  {
127  wxMessageBox( errorMsg );
128  return false;
129  }
130 
131 
132  for( unsigned ii = 0; ii < json_image.size(); ii++ )
133  {
134  if( json_image[ii] == sep || ii == json_image.size() - 1 )
135  {
136  if( tmp.StartsWith( wxT( "\"full_name\"" ) ) )
137  {
138  #define QUOTE '\"'
139  // Remove useless quotes:
140  if( tmp[tmp.Length() - 1] == QUOTE )
141  tmp.RemoveLast();
142 
143  if( tmp.EndsWith( m_libs_ext ) )
144  {
145  aList.Add( tmp.AfterLast( ':' ) );
146  int idx = aList.GetCount() - 1;
147 
148  if( aList[idx][0] == QUOTE )
149  aList[idx].Remove( 0, 1 );
150 
151  aList[idx].Prepend( urlPrefix );
152  }
153 
154  items_count_per_page++;
155  }
156 
157  tmp.Clear();
158  }
159  else
160  tmp << json_image[ii];
161  }
162 
163  if( items_count_per_page >= itemCountMax )
164  {
165  page++;
166  repoURL2listURL( m_repoURL, &fullURLCommand, itemCountMax, page );
167  items_count_per_page = 0;
168  ClearBuffer();
169  }
170  else
171  break;
172  }
173 
174  aList.Sort();
175  return true;
176 }
char m_option_string[64]
option for transfert type, like "application/json"
bool repoURL2listURL(const wxString &aRepoURL, std::string *aFullURLCommand, int aItemCountMax, int aPage=1)
Function repoURL2listURL translates a repo URL to the URL name which gives the state of repos URL as ...
#define QUOTE
static const wxChar * GetChars(const wxString &s)
Function GetChars returns a wxChar* to the actual wxChar* data within a wxString, and is helpful for ...
Definition: macros.h:92
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, CPTREE &aTree)
Function Format outputs a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:205
bool remoteGetJSON(const std::string &aFullURLCommand, wxString *aMsgError)
Function remoteGetJSON Download a json text from a github repo.
wxString m_repoURL
the URL of the Github repo
wxString m_libs_ext
the extension of the name of libraries (default = .pretty)
void ClearBuffer()
Clear the buffer which stores all the downloaded raw data.
std::string & GetBuffer()
bool GITHUB_GETLIBLIST::remoteGetJSON ( const std::string &  aFullURLCommand,
wxString *  aMsgError 
)
protected

Function remoteGetJSON Download a json text from a github repo.

The text image is received into the m_input_stream.

Parameters
aFullURLCommandthe full command, i.e. the url with options like "https://api.github.com/users/KiCad/repos?per_page=100?page=1"
aMsgErrora pointer to a wxString which can store an error message
Returns
true if OK, false if error (which an error message in *aMsgError

Definition at line 213 of file github_getliblist.cpp.

References UTF8::c_str(), FROM_UTF8(), KICAD_CURL_EASY::GetBuffer(), m_image, m_option_string, KICAD_CURL_EASY::Perform(), KICAD_CURL_EASY::SetFollowRedirects(), KICAD_CURL_EASY::SetHeader(), KICAD_CURL_EASY::SetURL(), KICAD_CURL_EASY::SetUserAgent(), StrPrintf(), TO_UTF8, and IO_ERROR::What().

Referenced by Get3DshapesLibsList(), and GetFootprintLibraryList().

214 {
215  KICAD_CURL_EASY kcurl;
216 
217  wxLogDebug( wxT( "Attempting to download: " ) + aFullURLCommand );
218 
219  kcurl.SetURL( aFullURLCommand );
220  kcurl.SetUserAgent( "http://kicad-pcb.org" );
221  kcurl.SetHeader( "Accept", m_option_string );
222  kcurl.SetFollowRedirects( true );
223 
224  try
225  {
226  kcurl.Perform();
227  m_image = kcurl.GetBuffer();
228  return true;
229  }
230  catch( const IO_ERROR& ioe )
231  {
232  if( aMsgError )
233  {
234  UTF8 fmt( _( "Error fetching JSON data from URL '%s'.\nReason: '%s'" ) );
235 
236  std::string msg = StrPrintf( fmt.c_str(),
237  aFullURLCommand.c_str(),
238  TO_UTF8( ioe.What() ) );
239 
240  *aMsgError = FROM_UTF8( msg.c_str() );
241  }
242  return false;
243  }
244 }
Class UTF8 is an 8 bit string that is assuredly encoded in UTF8, and supplies special conversion supp...
Definition: utf8.h:73
bool SetUserAgent(const std::string &aAgent)
Function SetUserAgent sets the request user agent.
void Perform()
Function perform equivalent to curl_easy_perform.
bool SetFollowRedirects(bool aFollow)
Function SetFollowRedirects enables the following of HTTP(s) and other redirects, by default curl doe...
static wxString FROM_UTF8(const char *cstring)
function FROM_UTF8 converts a UTF8 encoded C string to a wxString for all wxWidgets build modes...
Definition: macros.h:53
Class KICAD_CURL_EASY wrapper interface around the curl_easy API.
char m_option_string[64]
option for transfert type, like "application/json"
int StrPrintf(std::string *result, const char *format,...)
Function StrPrintf is like sprintf() but the output is appended to a std::string instead of to a char...
Definition: richio.cpp:75
#define TO_UTF8(wxstring)
Macro TO_UTF8 converts a wxString to a UTF8 encoded C string for all wxWidgets build modes...
Definition: macros.h:47
const std::string & GetBuffer()
Function GetBuffer returns a const reference to the recevied data buffer.
std::string m_image
image of the downloaded data in its entirety.
bool SetURL(const std::string &aURL)
Function SetURL sets the request URL.
virtual const wxString What() const
A composite of Problem() and Where()
Definition: exceptions.cpp:33
void SetHeader(const std::string &aName, const std::string &aValue)
Function SetHeader sets an arbitrary header for the HTTP(s) request.
Struct IO_ERROR is a class used to hold an error message and may be used when throwing exceptions con...
Definition: ki_exception.h:47
bool GITHUB_GETLIBLIST::repoURL2listURL ( const wxString &  aRepoURL,
std::string *  aFullURLCommand,
int  aItemCountMax,
int  aPage = 1 
)
protected

Function repoURL2listURL translates a repo URL to the URL name which gives the state of repos URL as commonly seen on github.com.

Parameters
aRepoURLpoints to the base of the repo.
aFullURLCommandis URL the full URL command (URL+options).
aItemCountMaxis the max item count in a page, and is 100 for github repo.
aPageis the page number, if there are more than one page in repo.
Returns
bool - true if aRepoULR was parseable, else false

Definition at line 179 of file github_getliblist.cpp.

References Format().

Referenced by GetFootprintLibraryList().

182 {
183  // aListURL is e.g. "https://api.github.com/orgs/KiCad/repos"
184  // or "https://api.github.com/users/KiCad/repos"
185  // aRepoURL is e.g. "https://github.com/KiCad"
186  // Github has a default pagination set to 30 items.
187  // but allows up to 100 items max if we add the "?per_page=100" option
188 
189  wxURI repo( aRepoURL );
190 
191  if( repo.HasServer() && repo.HasPath() )
192  {
193  // goal: "https://api.github.com/users/KiCad"
194  // or "https://api.github.com/orgs/KiCad"
195  // "https://api.github.com/users/..." works both for orgs and users
196  // if we just retrieve the .pretty list
197  wxString target_url( wxT( "https://api.github.com/users" ) );
198  target_url += repo.GetPath();
199  target_url += wxT( "/repos" );
200 
201  // Github has a default pagination set to 30 items.
202  // but allows up to 100 items max. Use this limit
203  target_url += wxString::Format( "?per_page=%d&page=%d", aItemCountMax, aPage );
204 
205  *aFullURLCommand = target_url.utf8_str();
206  return true;
207  }
208 
209  return false;
210 }
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, CPTREE &aTree)
Function Format outputs a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:205
void GITHUB_GETLIBLIST::SetLibraryExt ( const wxString &  aExt)
inline

The library names are expecting ending by .pretty SetLibraryExt set the extension to aExt.

Definition at line 81 of file github_getliblist.h.

References m_libs_ext.

81 { m_libs_ext = aExt; }
wxString m_libs_ext
the extension of the name of libraries (default = .pretty)

Member Data Documentation

wxString GITHUB_GETLIBLIST::m_github_path
protected

Something like https://api.github.com/orgs/KiCad.

Definition at line 116 of file github_getliblist.h.

std::string GITHUB_GETLIBLIST::m_image
protected

image of the downloaded data in its entirety.

Definition at line 117 of file github_getliblist.h.

Referenced by ClearBuffer(), GetBuffer(), and remoteGetJSON().

wxString GITHUB_GETLIBLIST::m_libs_ext
protected

the extension of the name of libraries (default = .pretty)

Definition at line 119 of file github_getliblist.h.

Referenced by GetFootprintLibraryList(), GITHUB_GETLIBLIST(), and SetLibraryExt().

char GITHUB_GETLIBLIST::m_option_string[64]
protected

option for transfert type, like "application/json"

Definition at line 120 of file github_getliblist.h.

Referenced by Get3DshapesLibsList(), GetFootprintLibraryList(), GITHUB_GETLIBLIST(), and remoteGetJSON().

wxString GITHUB_GETLIBLIST::m_repoURL
protected

the URL of the Github repo

Definition at line 118 of file github_getliblist.h.

Referenced by Get3DshapesLibsList(), GetFootprintLibraryList(), and GITHUB_GETLIBLIST().


The documentation for this class was generated from the following files: