Keep statistics about a feed and its offline processor. More...

#include <offline_feed_statistics.hpp>

Classes
class	config
	Configure an offline_feed_statistics object. More...

Public Member Functions
	offline_feed_statistics (config const &cfg)
	Constructor. More...

template<typename event_timestamp_t , typename duration_t >
void	sample (event_timestamp_t ts, duration_t processing_latency)
	Record a sample, that is process a message received at the given timestamp. More...

void	print_csv (std::string const &name, std::ostream &os) const
	Print all the measurements in CSV format. More...

void	log_final_progress () const
	Final progress report at the end of the input. More...

Static Public Member Functions
static void	print_csv_header (std::ostream &os)
	Print a CSV header. More...

Private Types
typedef event_rate_histogram< std::chrono::nanoseconds, std::int64_t >	rate_histogram

typedef histogram< integer_range_binning< std::int64_t > >	interarrival_histogram_t

typedef histogram< integer_range_binning< std::uint64_t > >	processing_latency_histogram_t

Private Member Functions
void	log_progress (std::chrono::nanoseconds ts) const
	Report progress up to a certain point in the input. More...

void	record_sample (std::chrono::nanoseconds ts, std::chrono::nanoseconds processing_latency)
	Refactor non-template portions of sample() More...

Private Attributes
rate_histogram	per_sec_rate_

rate_histogram	per_msec_rate_

rate_histogram	per_usec_rate_

interarrival_histogram_t	interarrival_

processing_latency_histogram_t	processing_latency_

std::chrono::seconds	reporting_interval_

std::chrono::nanoseconds	last_ts_

std::chrono::nanoseconds	last_report_ts_

Detailed Description

Keep statistics about a feed and its offline processor.

In JayBeams we will find ourselves processing all kinds of offline feeds, that is, files with the contents of a real-time feed. We want to measure things like the processing time for the data, what is the distribution of message rates like, what is the distribution of mesage inter-arrival times like.

This class encapsulates a lot of the logic for those computations. The class can be thoroughly configured at run-time, for ease of experimentation, but the default values should be reasonable.

The source of timestamps can be an actual clock, the recorded timestamps is a feed file, or some virtual clock used in simulations. The only requirement is for the timestamps to be compatible with std::chrono::duration<>. It might seem odd to use std::chrono::duration<> instead of std::chrono::time_point<>, this is because most feeds are timestamps to a well known time point: midnight at the beginning of the day, or 00:00:00am.

Likewise, the processing latency measurements can be generated from an actual clock or some virtual measurement (say counting interrupts, or CPU cycles vs. elapsed time). The only requirement is for the measurements to be compatible with std::chono::duration<>.

Definition at line 40 of file offline_feed_statistics.hpp.

Member Typedef Documentation

◆ interarrival_histogram_t

typedef histogram<integer_range_binning<std::int64_t> > jb::offline_feed_statistics::interarrival_histogram_t

private

Definition at line 145 of file offline_feed_statistics.hpp.

◆ processing_latency_histogram_t

typedef histogram<integer_range_binning<std::uint64_t> > jb::offline_feed_statistics::processing_latency_histogram_t

private

Definition at line 149 of file offline_feed_statistics.hpp.

◆ rate_histogram

typedef event_rate_histogram<std::chrono::nanoseconds, std::int64_t> jb::offline_feed_statistics::rate_histogram

private

Definition at line 140 of file offline_feed_statistics.hpp.

Constructor & Destructor Documentation

◆ offline_feed_statistics()

jb::offline_feed_statistics::offline_feed_statistics ( config const & cfg )

explicit

Constructor.

Definition at line 166 of file offline_feed_statistics.cpp.

Member Function Documentation

◆ log_final_progress()

void jb::offline_feed_statistics::log_final_progress ( ) const

Final progress report at the end of the input.

Definition at line 205 of file offline_feed_statistics.cpp.

References log_progress().

Referenced by BOOST_AUTO_TEST_CASE(), and sample().

◆ log_progress()

void jb::offline_feed_statistics::log_progress ( std::chrono::nanoseconds ts ) const

private

Report progress up to a certain point in the input.

Parameters

ts	the (logical) timestamp of the current event in the input

Definition at line 210 of file offline_feed_statistics.cpp.

References interarrival_, jb::histogram< binning_strategy_t, counter_type_t >::nsamples(), per_msec_rate_, per_sec_rate_, per_usec_rate_, and processing_latency_.

Referenced by log_final_progress(), record_sample(), and sample().

◆ print_csv()

void jb::offline_feed_statistics::print_csv	(	std::string const &	name,
		std::ostream &	os
	)		const

Print all the measurements in CSV format.

Definition at line 254 of file offline_feed_statistics.cpp.

References interarrival_, jb::histogram< binning_strategy_t, counter_type_t >::nsamples(), per_msec_rate_, per_sec_rate_, per_usec_rate_, and processing_latency_.

Referenced by BOOST_AUTO_TEST_CASE(), and sample().

◆ print_csv_header()

void jb::offline_feed_statistics::print_csv_header ( std::ostream & os )

static

Print a CSV header.

Assuming there are many offline_feed_statistics<> objects (say one for each symbol, or one for each minute interval) this function can be used to print the CSV header for all of them.

The fields include:

name: the name of the offline_feed_statistics<> object.
nsamples: the number of samples received.
minRatePerSec: the minimum messages/second rate
p25RatePerSec: the 25th percentile for the messages/second rate
p50RatePerSec: the 50th percentile for the messages/second rate
p75RatePerSec: the 75th percentile for the messages/second rate
p90RatePerSec: the 90th percentile for the messages/second rate
p99RatePerSec: the 99th percentile for the messages/second rate
p999RatePerSec: the 99.9th percentile for the messages/second rate
p9999RatePerSec: the 99.99th percentile for the messages/second rate
maxRatePerSec: the maximum for the messages/second rate
minRatePerMSec, p25RatePerMSec, ..., maxRatePerMSec: the statistics for the messages/millisecond rate.
minRatePerUSec, p25RatePerUSec, ..., maxRatePerUSec: the statistics for the messages/microsecond rate.
minProcessingLatency, ..., maxProcessingLatency: the statistics for processing latency of the events, in nanoseconds.
minArrival: the minimum of the timestamp difference between two consecutive messages, in nanoseconds.
p0001Arrival: the 0.01th percentile of the timestamp difference between two consecutive messages, in nanoseconds.
p001Arrival: the 0.1th percentile of the timestamp difference between two consecutive messages, in nanoseconds.
p01Arrival: the 1st percentile of the timestamp difference between two consecutive messages, in nanoseconds.
p10Arrival: the 10th percentile of the timestamp difference between two consecutive messages, in nanoseconds.
p25Arrival, p50Arrival, p75Arrival, maxArrival: more statistics about the arrival time.