★ wanayoo — archive 1999 https://github.com/gousiosg/github-mirrorNouvelle recherche | Portail wanayoo
This repository
This repository
All repositories

Scripts to mirror Github in a cloudy fashion

Ruby
branch: master
Switch branches/tags
Nothing to show
Show File Finder
README.md

ghtorrent: Mirror and process data from the Github API

A library and a collection of scripts used to retrieve data from the Github API and extract metadata in an SQL database, in a modular and scalable manner. The scripts are distributed as a Gem (ghtorrent), but they can also be run by checking out this repository.

GHTorrent can be used for a variety of purposes, such as:

  • Mirror the Github API event stream and follow links from events to actual data to gradually build a Github index
  • Create a queriable metadata index for a specific repository
  • Query the Github API using intelligent caching to avoid duplicate queries

GHTorrent is comprised from the following components (which can be used individually):

  • APIClient: Knows how to query the Github API (both single entities and pages) and respect the API request limit. Can be configured to override the default IP address, in case of multihomed hosts. Uses configurable on disk caching to avoid retrieving data that do not change.
  • Retriever: Knows how to retrieve specific Github entities (users, repositories, watchers) by name. Uses an optional persister to avoid retrieving data that have not changed.
  • Persister: A key/value store, which can be backed by a real key/value store, to store Github JSON replies and query them on request. The backing key/value store must support arbitrary queries to the stored JSON objects.
  • GHTorrent: Knows how to extract information from the data retrieved by the retriever in order to update an SQL database (see schema) with metadata.

The Persister and GHTorrent components have configurable back ends:

  • Persister: Either uses MongoDB > 2.0 (mongo driver) or no persister (noop driver)
  • GHTorrent: GHTorrent is tested mainly with MySQL, but can theoretically be used with any SQL database compatible with Sequel. Your milaege may vary.

The distributed mirroring scripts also require RabbitMQ >= 2.8 or other

Installing

GHTorrent is written in Ruby (tested with 1.9). To install it as a Gem do:

sudo gem install ghtorrent

Depending on which SQL database you want to use, install the appropriate dependency gem.

sudo gem install mysql2 #or sqlite3-ruby #or postgres

Configuring

Copy the contents of the config.yaml.tmpl file to a file in your home directory. All provided scripts accept the -c option, which you can use to pass the location of the configuration file as a parameter.

You can find more information of how you can setup a mirroring cluster of machines to retrieve data in parallel on the Wiki.

Running

To mirror the event stream and capture all data:

  • ght-mirror-events.rb periodically polls Github's event queue (https://api.github.com/events), stores all new events in the configured pestister and posts them to the github exchange in RabbitMQ.

  • ght-data_retrieval.rb creates queues that route posted events to processor functions, which in turn use the appropriate Github API call to retrieve the linked contents, extract metadata to store in the SQL database and store the retrieved data in the appropriate collection in the persister, to avoid duplicate API calls. Data in the SQL database contain pointers (the ext_ref_id field) to the "raw" data in the persister.

To retrieve data for a repository or user:

  • ght-retrieve-repo retrieves all data for a specific repository
  • ght-retrieve-user retrieves all data for a specific user

To perform maintenance:

  • ght-load loads selected events from the persister to the queue in order for the ght-data-retrieval script to reprocess them
  • ght-get-more-commits retrieves all commits for a specific repository

Data

You can find torrents for retrieving data on the Available Torrents page. You can find two sets of data:

  • Raw events: Github's event stream. These are the roots for mirroring operations. The ght-data-retrieval crawler starts from an event and goes deep into the rabbit hole.
  • SQL dumps+Linked data: Data dumps from the SQL database and the corresponding MongoDB entities.

Reporting bugs

Please use the Issue Tracker for reporting bugs and feature requests.

Patches, bug fixes etc are welcome. Please fork the repository and create a pull request when done fixing/implementing the new feature.

Citation information

If you find GHTorrent and the accompanying datasets useful in your research, please consider citing the following paper:

Georgios Gousios and Diomidis Spinellis, "GHTorrent: GitHub’s data from a firehose," in MSR '12: Proceedings of the 9th Working Conference on Mining Software Repositories, June 2-–3, 2012. Zurich, Switzerland.

See also the following presentation:

Authors

Georgios Gousios gousiosg@gmail.com

Diomidis Spinellis dds@aueb.gr

License

2-clause BSD

Something went wrong with that request. Please try again.