Sketches by Component Repository

The DataSketches Library is organized into the following repository groups:

Java

datasketches-java

This repository has the core-java sketching classes, which are leveraged by some of the other repositories.
This repository has no external dependencies outside of the DataSketches/memory repository, Java and TestNG for unit tests. This code is versioned and the latest release can be obtained from Downloads.

High-level Repositories Structure

datasketches-memory

This code is versioned and the latest release can be obtained from Downloads.

datasketches-hive

This repository contains Hive UDFs and UDAFs for use within Hadoop grid enviornments. This code has dependencies on sketches-core as well as Hadoop and Hive. Users of this code are advised to use Maven to bring in all the required dependencies. This code is versioned and the latest release can be obtained from Downloads.

datasketches-pig

This repository contains Pig User Defined Functions (UDF) for use within Hadoop grid environments. This code has dependencies on sketches-core as well as Hadoop and Pig. Users of this code are advised to use Maven to bring in all the required dependencies. This code is versioned and the latest release can be obtained from Downloads.

datasketches-characterization

This relatively new repository is for Java and C++ code that we use to characterize the accuracy and speed performance of the sketches in the library and is constantly being updated. Examples of the job command files used for various tests can be found in the src/main/resources directory. Some of these tests can run for hours depending on its configuration. This component is not formally released and code must be obtained from the GitHub site.

datasketches-server

This is a new repository for our experimental docker/container server that enables easy access to the core sketches in the library via HTTP. This component is not formally released and code must be obtained from the GitHub site.

C++ Characterizations

• CPC
• Frequent Items
• HLL
• KLL
• Theta

datasketches-vector

This component implements the Frequent Directions Algorithm [GLP16]. It is still experimental in that the theoretical work has not yet supplied a suitable measure of error for production work. It can be used as is, but it will not go through a formal Apache Release until we can find a way to provide better error properties. It has a dependence on the Memory component. This component is not formally released and code must be obtained from the GitHub site.

C++ and Python

datasketches-cpp

This is the evolving C++ implementations of the same sketches that are available in Java. These implementations are binary compatible with their counterparts in Java. In other words, a sketch created and stored in C++ can be opened and read in Java and visa-versa. This code is versioned and the latest release can be obtained from Downloads.

This site also has our Python adaptors that basically wrap the C++ implementations, making the high performance C++ implementations available from Python.

datasketches-postgresql

This site provides the postgres-specific adaptors that wrap the C++ implementations making them available to the PostgreSQL database users. PostgreSQL users should download the PostgreSQL extension from pgxn.org. For examples refer to the README on the component site. This code is versioned and the latest release can be obtained from Downloads.