Probabilistic data types

Learn how to use approximate calculations with Redis.

Redis supports several probabilistic data types that let you calculate values approximately rather than exactly. The types fall into two basic categories:

Set operations: These types let you calculate (approximately) the number of items in a set of distinct values, and whether or not a given value is a member of a set.
Statistics: These types give you an approximation of statistics such as the quantiles, ranks, and frequencies of numeric data points in a list.

To see why these approximate calculations would be useful, consider the task of counting the number of distinct IP addresses that access a website in one day.

Assuming that you already have code that supplies you with each IP address as a string, you could record the addresses in Redis using a set:

jedis.sadd("ip_tracker", new_ip_address)

The set can only contain each key once, so if the same address appears again during the day, the new instance will not change the set. At the end of the day, you could get the exact number of distinct addresses using the scard() function:

long num_distinct_ips = jedis.scard("ip_tracker")

This approach is simple, effective, and precise but if your website is very busy, the ip_tracker set could become very large and consume a lot of memory.

You would probably round the count of distinct IP addresses to the nearest thousand or more to deliver the usage statistics, so getting it exactly right is not important. It would be useful if you could trade off some accuracy in exchange for lower memory consumption. The probabilistic data types provide exactly this kind of trade-off. Specifically, you can count the approximate number of items in a set using the HyperLogLog data type, as described below.

In general, the probabilistic data types let you perform approximations with a bounded degree of error that have much lower memory consumption or execution time than the equivalent precise calculations.

Set operations

Redis supports the following approximate set operations:

Membership: The Bloom filter and Cuckoo filter data types let you track whether or not a given item is a member of a set.
Cardinality: The HyperLogLog data type gives you an approximate value for the number of items in a set, also known as the cardinality of the set.

The sections below describe these operations in more detail.

Set membership

Bloom filter and Cuckoo filter objects provide a set membership operation that lets you track whether or not a particular item has been added to a set. These two types provide different trade-offs for memory usage and speed, so you can select the best one for your use case. Note that for both types, there is an asymmetry between presence and absence of items in the set. If an item is reported as absent, then it is definitely absent, but if it is reported as present, then there is a small chance it may really be absent.

Instead of storing strings directly, like a set, a Bloom filter records the presence or absence of the hash value of a string. This gives a very compact representation of the set's membership with a fixed memory size, regardless of how many items you add. The following example adds some names to a Bloom filter representing a list of users and checks for the presence or absence of users in the list.

Java-Sync

package io.redis.examples;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class HomeProbDtsExample {

    public void run() {
        UnifiedJedis jedis = new UnifiedJedis("redis://localhost:6379");


        List<Boolean> res1 = jedis.bfMAdd(
            "recorded_users",
            "andy", "cameron", "david", "michelle"
        );
        System.out.println(res1);  // >>> [true, true, true, true]

        boolean res2 = jedis.bfExists("recorded_users", "cameron");
        System.out.println(res2);  // >>> true

        boolean res3 = jedis.bfExists("recorded_users", "kaitlyn");
        System.out.println(res3);  // >>> false

        boolean res4 = jedis.cfAdd("other_users", "paolo");
        System.out.println(res4);  // >>> true

        boolean res5 = jedis.cfAdd("other_users", "kaitlyn");
        System.out.println(res5);  // >>> true

        boolean res6 = jedis.cfAdd("other_users", "rachel");
        System.out.println(res6);  // >>> true

        List<Boolean> res7 = jedis.cfMExists(
            "other_users",
            "paolo", "rachel", "andy"
        );
        System.out.println(res7);  // >>> [true, true, false]

        boolean res8 = jedis.cfDel("other_users", "paolo");
        System.out.println(res8);  // >>> true

        boolean res9 = jedis.cfExists("other_users", "paolo");
        System.out.println(res9);  // >>> false

        long res10 = jedis.pfadd("group:1", "andy", "cameron", "david");
        System.out.println(res10);  // >>> 1

        long res11 = jedis.pfcount("group:1");
        System.out.println(res11);  // >>> 3

        long res12 = jedis.pfadd(
            "group:2",
            "kaitlyn", "michelle", "paolo", "rachel"
        );
        System.out.println(res12);  // >>> 1

        long res13 = jedis.pfcount("group:2");
        System.out.println(res13);  // >>> 4

        String res14 = jedis.pfmerge("both_groups", "group:1", "group:2");
        System.out.println(res14);  // >>> OK

        long res15 = jedis.pfcount("both_groups");
        System.out.println(res15);  // >>> 7

        // Specify that you want to keep the counts within 0.01
        // (0.1%) of the true value with a 0.005 (0.05%) chance
        // of going outside this limit.
        String res16 = jedis.cmsInitByProb("items_sold", 0.01, 0.005);
        System.out.println(res16);  // >>> OK

        Map<String, Long> firstItemIncrements = new HashMap<>();
        firstItemIncrements.put("bread", 300L);
        firstItemIncrements.put("tea", 200L);
        firstItemIncrements.put("coffee", 200L);
        firstItemIncrements.put("beer", 100L);

        List<Long> res17 = jedis.cmsIncrBy("items_sold",
            firstItemIncrements
        );
        res17.sort(null);
        System.out.println();  // >>> [100, 200, 200, 300]

        Map<String, Long> secondItemIncrements = new HashMap<>();
        secondItemIncrements.put("bread", 100L);
        secondItemIncrements.put("coffee", 150L);

        List<Long> res18 = jedis.cmsIncrBy("items_sold",
            secondItemIncrements
        );
        res18.sort(null);
        System.out.println(res18);  // >>> [350, 400]

        List<Long> res19 = jedis.cmsQuery(
            "items_sold",
            "bread", "tea", "coffee", "beer"
        );
        res19.sort(null);
        System.out.println(res19);  // >>> [100, 200, 350, 400]

        String res20 = jedis.tdigestCreate("male_heights");
        System.out.println(res20);  // >>> OK

        String res21 = jedis.tdigestAdd("male_heights", 
            175.5, 181, 160.8, 152, 177, 196, 164);
        System.out.println(res21);  // >>> OK

        double res22 = jedis.tdigestMin("male_heights");
        System.out.println(res22);  // >>> 152.0

        double res23 = jedis.tdigestMax("male_heights");
        System.out.println(res23);  // >>> 196.0

        List<Double> res24 = jedis.tdigestQuantile("male_heights", 0.75);
        System.out.println(res24);  // >>> [181.0]

        // Note that the CDF value for 181 is not exactly 0.75.
        // Both values are estimates.
        List<Double> res25 = jedis.tdigestCDF("male_heights", 181);
        System.out.println(res25);  // >>> [0.7857142857142857]

        String res26 = jedis.tdigestCreate("female_heights");
        System.out.println(res26);  // >>> OK

        String res27 = jedis.tdigestAdd("female_heights",
            155.5, 161, 168.5, 170, 157.5, 163, 171);
        System.out.println(res27);  // >>> OK

        List<Double> res28 = jedis.tdigestQuantile("female_heights", 0.75);
        System.out.println(res28);  // >>> [170.0]

        String res29 = jedis.tdigestMerge(
            "all_heights",
            "male_heights", "female_heights"
        );
        System.out.println(res29);  // >>> OK
        List<Double> res30 = jedis.tdigestQuantile("all_heights", 0.75);
        System.out.println(res30);  // >>> [175.5]

        String res31 = jedis.topkReserve("top_3_songs", 3L, 2000L, 7L, 0.925D);
        System.out.println(res31);  // >>> OK

        Map<String, Long> songIncrements = new HashMap<>();
        songIncrements.put("Starfish Trooper", 3000L);
        songIncrements.put("Only one more time", 1850L);
        songIncrements.put("Rock me, Handel", 1325L);
        songIncrements.put("How will anyone know?", 3890L);
        songIncrements.put("Average lover", 4098L);
        songIncrements.put("Road to everywhere", 770L);

        List<String> res32 = jedis.topkIncrBy("top_3_songs",
            songIncrements
        );
        System.out.println(res32);
        // >>> [null, null, null, null, null, Rock me, Handel]

        List<String> res33 = jedis.topkList("top_3_songs");
        System.out.println(res33);
        // >>> [Average lover, How will anyone know?, Starfish Trooper]

        List<Boolean> res34 = jedis.topkQuery("top_3_songs",
            "Starfish Trooper", "Road to everywhere"
        );
        System.out.println(res34);
        // >>> [true, false]
    }
}

A Cuckoo filter has similar features to a Bloom filter, but also supports a deletion operation to remove hashes from a set, as shown in the example below.

Java-Sync

package io.redis.examples;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class HomeProbDtsExample {

    public void run() {
        UnifiedJedis jedis = new UnifiedJedis("redis://localhost:6379");


        List<Boolean> res1 = jedis.bfMAdd(
            "recorded_users",
            "andy", "cameron", "david", "michelle"
        );
        System.out.println(res1);  // >>> [true, true, true, true]

        boolean res2 = jedis.bfExists("recorded_users", "cameron");
        System.out.println(res2);  // >>> true

        boolean res3 = jedis.bfExists("recorded_users", "kaitlyn");
        System.out.println(res3);  // >>> false

        boolean res4 = jedis.cfAdd("other_users", "paolo");
        System.out.println(res4);  // >>> true

        boolean res5 = jedis.cfAdd("other_users", "kaitlyn");
        System.out.println(res5);  // >>> true

        boolean res6 = jedis.cfAdd("other_users", "rachel");
        System.out.println(res6);  // >>> true

        List<Boolean> res7 = jedis.cfMExists(
            "other_users",
            "paolo", "rachel", "andy"
        );
        System.out.println(res7);  // >>> [true, true, false]

        boolean res8 = jedis.cfDel("other_users", "paolo");
        System.out.println(res8);  // >>> true

        boolean res9 = jedis.cfExists("other_users", "paolo");
        System.out.println(res9);  // >>> false

        long res10 = jedis.pfadd("group:1", "andy", "cameron", "david");
        System.out.println(res10);  // >>> 1

        long res11 = jedis.pfcount("group:1");
        System.out.println(res11);  // >>> 3

        long res12 = jedis.pfadd(
            "group:2",
            "kaitlyn", "michelle", "paolo", "rachel"
        );
        System.out.println(res12);  // >>> 1

        long res13 = jedis.pfcount("group:2");
        System.out.println(res13);  // >>> 4

        String res14 = jedis.pfmerge("both_groups", "group:1", "group:2");
        System.out.println(res14);  // >>> OK

        long res15 = jedis.pfcount("both_groups");
        System.out.println(res15);  // >>> 7

        // Specify that you want to keep the counts within 0.01
        // (0.1%) of the true value with a 0.005 (0.05%) chance
        // of going outside this limit.
        String res16 = jedis.cmsInitByProb("items_sold", 0.01, 0.005);
        System.out.println(res16);  // >>> OK

        Map<String, Long> firstItemIncrements = new HashMap<>();
        firstItemIncrements.put("bread", 300L);
        firstItemIncrements.put("tea", 200L);
        firstItemIncrements.put("coffee", 200L);
        firstItemIncrements.put("beer", 100L);

        List<Long> res17 = jedis.cmsIncrBy("items_sold",
            firstItemIncrements
        );
        res17.sort(null);
        System.out.println();  // >>> [100, 200, 200, 300]

        Map<String, Long> secondItemIncrements = new HashMap<>();
        secondItemIncrements.put("bread", 100L);
        secondItemIncrements.put("coffee", 150L);

        List<Long> res18 = jedis.cmsIncrBy("items_sold",
            secondItemIncrements
        );
        res18.sort(null);
        System.out.println(res18);  // >>> [350, 400]

        List<Long> res19 = jedis.cmsQuery(
            "items_sold",
            "bread", "tea", "coffee", "beer"
        );
        res19.sort(null);
        System.out.println(res19);  // >>> [100, 200, 350, 400]

        String res20 = jedis.tdigestCreate("male_heights");
        System.out.println(res20);  // >>> OK

        String res21 = jedis.tdigestAdd("male_heights", 
            175.5, 181, 160.8, 152, 177, 196, 164);
        System.out.println(res21);  // >>> OK

        double res22 = jedis.tdigestMin("male_heights");
        System.out.println(res22);  // >>> 152.0

        double res23 = jedis.tdigestMax("male_heights");
        System.out.println(res23);  // >>> 196.0

        List<Double> res24 = jedis.tdigestQuantile("male_heights", 0.75);
        System.out.println(res24);  // >>> [181.0]

        // Note that the CDF value for 181 is not exactly 0.75.
        // Both values are estimates.
        List<Double> res25 = jedis.tdigestCDF("male_heights", 181);
        System.out.println(res25);  // >>> [0.7857142857142857]

        String res26 = jedis.tdigestCreate("female_heights");
        System.out.println(res26);  // >>> OK

        String res27 = jedis.tdigestAdd("female_heights",
            155.5, 161, 168.5, 170, 157.5, 163, 171);
        System.out.println(res27);  // >>> OK

        List<Double> res28 = jedis.tdigestQuantile("female_heights", 0.75);
        System.out.println(res28);  // >>> [170.0]

        String res29 = jedis.tdigestMerge(
            "all_heights",
            "male_heights", "female_heights"
        );
        System.out.println(res29);  // >>> OK
        List<Double> res30 = jedis.tdigestQuantile("all_heights", 0.75);
        System.out.println(res30);  // >>> [175.5]

        String res31 = jedis.topkReserve("top_3_songs", 3L, 2000L, 7L, 0.925D);
        System.out.println(res31);  // >>> OK

        Map<String, Long> songIncrements = new HashMap<>();
        songIncrements.put("Starfish Trooper", 3000L);
        songIncrements.put("Only one more time", 1850L);
        songIncrements.put("Rock me, Handel", 1325L);
        songIncrements.put("How will anyone know?", 3890L);
        songIncrements.put("Average lover", 4098L);
        songIncrements.put("Road to everywhere", 770L);

        List<String> res32 = jedis.topkIncrBy("top_3_songs",
            songIncrements
        );
        System.out.println(res32);
        // >>> [null, null, null, null, null, Rock me, Handel]

        List<String> res33 = jedis.topkList("top_3_songs");
        System.out.println(res33);
        // >>> [Average lover, How will anyone know?, Starfish Trooper]

        List<Boolean> res34 = jedis.topkQuery("top_3_songs",
            "Starfish Trooper", "Road to everywhere"
        );
        System.out.println(res34);
        // >>> [true, false]
    }
}

Which of these two data types you choose depends on your use case. Bloom filters are generally faster than Cuckoo filters when adding new items, and also have better memory usage. Cuckoo filters are generally faster at checking membership and also support the delete operation. See the Bloom filter and Cuckoo filter reference pages for more information and comparison between the two types.

Set cardinality

A HyperLogLog object calculates the cardinality of a set. As you add items, the HyperLogLog tracks the number of distinct set members but doesn't let you retrieve them or query which items have been added. You can also merge two or more HyperLogLogs to find the cardinality of the union of the sets they represent.

Java-Sync

package io.redis.examples;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class HomeProbDtsExample {

    public void run() {
        UnifiedJedis jedis = new UnifiedJedis("redis://localhost:6379");


        List<Boolean> res1 = jedis.bfMAdd(
            "recorded_users",
            "andy", "cameron", "david", "michelle"
        );
        System.out.println(res1);  // >>> [true, true, true, true]

        boolean res2 = jedis.bfExists("recorded_users", "cameron");
        System.out.println(res2);  // >>> true

        boolean res3 = jedis.bfExists("recorded_users", "kaitlyn");
        System.out.println(res3);  // >>> false

        boolean res4 = jedis.cfAdd("other_users", "paolo");
        System.out.println(res4);  // >>> true

        boolean res5 = jedis.cfAdd("other_users", "kaitlyn");
        System.out.println(res5);  // >>> true

        boolean res6 = jedis.cfAdd("other_users", "rachel");
        System.out.println(res6);  // >>> true

        List<Boolean> res7 = jedis.cfMExists(
            "other_users",
            "paolo", "rachel", "andy"
        );
        System.out.println(res7);  // >>> [true, true, false]

        boolean res8 = jedis.cfDel("other_users", "paolo");
        System.out.println(res8);  // >>> true

        boolean res9 = jedis.cfExists("other_users", "paolo");
        System.out.println(res9);  // >>> false

        long res10 = jedis.pfadd("group:1", "andy", "cameron", "david");
        System.out.println(res10);  // >>> 1

        long res11 = jedis.pfcount("group:1");
        System.out.println(res11);  // >>> 3

        long res12 = jedis.pfadd(
            "group:2",
            "kaitlyn", "michelle", "paolo", "rachel"
        );
        System.out.println(res12);  // >>> 1

        long res13 = jedis.pfcount("group:2");
        System.out.println(res13);  // >>> 4

        String res14 = jedis.pfmerge("both_groups", "group:1", "group:2");
        System.out.println(res14);  // >>> OK

        long res15 = jedis.pfcount("both_groups");
        System.out.println(res15);  // >>> 7

        // Specify that you want to keep the counts within 0.01
        // (0.1%) of the true value with a 0.005 (0.05%) chance
        // of going outside this limit.
        String res16 = jedis.cmsInitByProb("items_sold", 0.01, 0.005);
        System.out.println(res16);  // >>> OK

        Map<String, Long> firstItemIncrements = new HashMap<>();
        firstItemIncrements.put("bread", 300L);
        firstItemIncrements.put("tea", 200L);
        firstItemIncrements.put("coffee", 200L);
        firstItemIncrements.put("beer", 100L);

        List<Long> res17 = jedis.cmsIncrBy("items_sold",
            firstItemIncrements
        );
        res17.sort(null);
        System.out.println();  // >>> [100, 200, 200, 300]

        Map<String, Long> secondItemIncrements = new HashMap<>();
        secondItemIncrements.put("bread", 100L);
        secondItemIncrements.put("coffee", 150L);

        List<Long> res18 = jedis.cmsIncrBy("items_sold",
            secondItemIncrements
        );
        res18.sort(null);
        System.out.println(res18);  // >>> [350, 400]

        List<Long> res19 = jedis.cmsQuery(
            "items_sold",
            "bread", "tea", "coffee", "beer"
        );
        res19.sort(null);
        System.out.println(res19);  // >>> [100, 200, 350, 400]

        String res20 = jedis.tdigestCreate("male_heights");
        System.out.println(res20);  // >>> OK

        String res21 = jedis.tdigestAdd("male_heights", 
            175.5, 181, 160.8, 152, 177, 196, 164);
        System.out.println(res21);  // >>> OK

        double res22 = jedis.tdigestMin("male_heights");
        System.out.println(res22);  // >>> 152.0

        double res23 = jedis.tdigestMax("male_heights");
        System.out.println(res23);  // >>> 196.0

        List<Double> res24 = jedis.tdigestQuantile("male_heights", 0.75);
        System.out.println(res24);  // >>> [181.0]

        // Note that the CDF value for 181 is not exactly 0.75.
        // Both values are estimates.
        List<Double> res25 = jedis.tdigestCDF("male_heights", 181);
        System.out.println(res25);  // >>> [0.7857142857142857]

        String res26 = jedis.tdigestCreate("female_heights");
        System.out.println(res26);  // >>> OK

        String res27 = jedis.tdigestAdd("female_heights",
            155.5, 161, 168.5, 170, 157.5, 163, 171);
        System.out.println(res27);  // >>> OK

        List<Double> res28 = jedis.tdigestQuantile("female_heights", 0.75);
        System.out.println(res28);  // >>> [170.0]

        String res29 = jedis.tdigestMerge(
            "all_heights",
            "male_heights", "female_heights"
        );
        System.out.println(res29);  // >>> OK
        List<Double> res30 = jedis.tdigestQuantile("all_heights", 0.75);
        System.out.println(res30);  // >>> [175.5]

        String res31 = jedis.topkReserve("top_3_songs", 3L, 2000L, 7L, 0.925D);
        System.out.println(res31);  // >>> OK

        Map<String, Long> songIncrements = new HashMap<>();
        songIncrements.put("Starfish Trooper", 3000L);
        songIncrements.put("Only one more time", 1850L);
        songIncrements.put("Rock me, Handel", 1325L);
        songIncrements.put("How will anyone know?", 3890L);
        songIncrements.put("Average lover", 4098L);
        songIncrements.put("Road to everywhere", 770L);

        List<String> res32 = jedis.topkIncrBy("top_3_songs",
            songIncrements
        );
        System.out.println(res32);
        // >>> [null, null, null, null, null, Rock me, Handel]

        List<String> res33 = jedis.topkList("top_3_songs");
        System.out.println(res33);
        // >>> [Average lover, How will anyone know?, Starfish Trooper]

        List<Boolean> res34 = jedis.topkQuery("top_3_songs",
            "Starfish Trooper", "Road to everywhere"
        );
        System.out.println(res34);
        // >>> [true, false]
    }
}

The main benefit that HyperLogLogs offer is their very low memory usage. They can count up to 2^64 items with less than 1% standard error using a maximum 12KB of memory. This makes them very useful for counting things like the total of distinct IP addresses that access a website or the total of distinct bank card numbers that make purchases within a day.

Statistics

Redis supports several approximate statistical calculations on numeric data sets:

Frequency: The Count-min sketch data type lets you find the approximate frequency of a labeled item in a data stream.
Quantiles: The t-digest data type estimates the quantile of a query value in a data stream.
Ranking: The Top-K data type estimates the ranking of labeled items by frequency in a data stream.

The sections below describe these operations in more detail.

Frequency

A Count-min sketch (CMS) object keeps count of a set of related items represented by string labels. The count is approximate, but you can specify how close you want to keep the count to the true value (as a fraction) and the acceptable probability of failing to keep it in this desired range. For example, you can request that the count should stay within 0.1% of the true value and have a 0.05% probability of going outside this limit. The example below shows how to create a Count-min sketch object, add data to it, and then query it.

Java-Sync

package io.redis.examples;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class HomeProbDtsExample {

    public void run() {
        UnifiedJedis jedis = new UnifiedJedis("redis://localhost:6379");


        List<Boolean> res1 = jedis.bfMAdd(
            "recorded_users",
            "andy", "cameron", "david", "michelle"
        );
        System.out.println(res1);  // >>> [true, true, true, true]

        boolean res2 = jedis.bfExists("recorded_users", "cameron");
        System.out.println(res2);  // >>> true

        boolean res3 = jedis.bfExists("recorded_users", "kaitlyn");
        System.out.println(res3);  // >>> false

        boolean res4 = jedis.cfAdd("other_users", "paolo");
        System.out.println(res4);  // >>> true

        boolean res5 = jedis.cfAdd("other_users", "kaitlyn");
        System.out.println(res5);  // >>> true

        boolean res6 = jedis.cfAdd("other_users", "rachel");
        System.out.println(res6);  // >>> true

        List<Boolean> res7 = jedis.cfMExists(
            "other_users",
            "paolo", "rachel", "andy"
        );
        System.out.println(res7);  // >>> [true, true, false]

        boolean res8 = jedis.cfDel("other_users", "paolo");
        System.out.println(res8);  // >>> true

        boolean res9 = jedis.cfExists("other_users", "paolo");
        System.out.println(res9);  // >>> false

        long res10 = jedis.pfadd("group:1", "andy", "cameron", "david");
        System.out.println(res10);  // >>> 1

        long res11 = jedis.pfcount("group:1");
        System.out.println(res11);  // >>> 3

        long res12 = jedis.pfadd(
            "group:2",
            "kaitlyn", "michelle", "paolo", "rachel"
        );
        System.out.println(res12);  // >>> 1

        long res13 = jedis.pfcount("group:2");
        System.out.println(res13);  // >>> 4

        String res14 = jedis.pfmerge("both_groups", "group:1", "group:2");
        System.out.println(res14);  // >>> OK

        long res15 = jedis.pfcount("both_groups");
        System.out.println(res15);  // >>> 7

        // Specify that you want to keep the counts within 0.01
        // (0.1%) of the true value with a 0.005 (0.05%) chance
        // of going outside this limit.
        String res16 = jedis.cmsInitByProb("items_sold", 0.01, 0.005);
        System.out.println(res16);  // >>> OK

        Map<String, Long> firstItemIncrements = new HashMap<>();
        firstItemIncrements.put("bread", 300L);
        firstItemIncrements.put("tea", 200L);
        firstItemIncrements.put("coffee", 200L);
        firstItemIncrements.put("beer", 100L);

        List<Long> res17 = jedis.cmsIncrBy("items_sold",
            firstItemIncrements
        );
        res17.sort(null);
        System.out.println();  // >>> [100, 200, 200, 300]

        Map<String, Long> secondItemIncrements = new HashMap<>();
        secondItemIncrements.put("bread", 100L);
        secondItemIncrements.put("coffee", 150L);

        List<Long> res18 = jedis.cmsIncrBy("items_sold",
            secondItemIncrements
        );
        res18.sort(null);
        System.out.println(res18);  // >>> [350, 400]

        List<Long> res19 = jedis.cmsQuery(
            "items_sold",
            "bread", "tea", "coffee", "beer"
        );
        res19.sort(null);
        System.out.println(res19);  // >>> [100, 200, 350, 400]

        String res20 = jedis.tdigestCreate("male_heights");
        System.out.println(res20);  // >>> OK

        String res21 = jedis.tdigestAdd("male_heights", 
            175.5, 181, 160.8, 152, 177, 196, 164);
        System.out.println(res21);  // >>> OK

        double res22 = jedis.tdigestMin("male_heights");
        System.out.println(res22);  // >>> 152.0

        double res23 = jedis.tdigestMax("male_heights");
        System.out.println(res23);  // >>> 196.0

        List<Double> res24 = jedis.tdigestQuantile("male_heights", 0.75);
        System.out.println(res24);  // >>> [181.0]

        // Note that the CDF value for 181 is not exactly 0.75.
        // Both values are estimates.
        List<Double> res25 = jedis.tdigestCDF("male_heights", 181);
        System.out.println(res25);  // >>> [0.7857142857142857]

        String res26 = jedis.tdigestCreate("female_heights");
        System.out.println(res26);  // >>> OK

        String res27 = jedis.tdigestAdd("female_heights",
            155.5, 161, 168.5, 170, 157.5, 163, 171);
        System.out.println(res27);  // >>> OK

        List<Double> res28 = jedis.tdigestQuantile("female_heights", 0.75);
        System.out.println(res28);  // >>> [170.0]

        String res29 = jedis.tdigestMerge(
            "all_heights",
            "male_heights", "female_heights"
        );
        System.out.println(res29);  // >>> OK
        List<Double> res30 = jedis.tdigestQuantile("all_heights", 0.75);
        System.out.println(res30);  // >>> [175.5]

        String res31 = jedis.topkReserve("top_3_songs", 3L, 2000L, 7L, 0.925D);
        System.out.println(res31);  // >>> OK

        Map<String, Long> songIncrements = new HashMap<>();
        songIncrements.put("Starfish Trooper", 3000L);
        songIncrements.put("Only one more time", 1850L);
        songIncrements.put("Rock me, Handel", 1325L);
        songIncrements.put("How will anyone know?", 3890L);
        songIncrements.put("Average lover", 4098L);
        songIncrements.put("Road to everywhere", 770L);

        List<String> res32 = jedis.topkIncrBy("top_3_songs",
            songIncrements
        );
        System.out.println(res32);
        // >>> [null, null, null, null, null, Rock me, Handel]

        List<String> res33 = jedis.topkList("top_3_songs");
        System.out.println(res33);
        // >>> [Average lover, How will anyone know?, Starfish Trooper]

        List<Boolean> res34 = jedis.topkQuery("top_3_songs",
            "Starfish Trooper", "Road to everywhere"
        );
        System.out.println(res34);
        // >>> [true, false]
    }
}

The advantage of using a CMS over keeping an exact count with a sorted set is that that a CMS has very low and fixed memory usage, even for large numbers of items. Use CMS objects to keep daily counts of items sold, accesses to individual web pages on your site, and other similar statistics.

Quantiles

A quantile is the value below which a certain fraction of samples lie. For example, with a set of measurements of people's heights, the quantile of 0.75 is the value of height below which 75% of all people's heights lie. Percentiles are equivalent to quantiles, except that the fraction is expressed as a percentage.

A t-digest object can estimate quantiles from a set of values added to it without having to store each value in the set explicitly. This can save a lot of memory when you have a large number of samples.

The example below shows how to add data samples to a t-digest object and obtain some basic statistics, such as the minimum and maximum values, the quantile of 0.75, and the cumulative distribution function (CDF), which is effectively the inverse of the quantile function. It also shows how to merge two or more t-digest objects to query the combined data set.

Java-Sync

package io.redis.examples;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class HomeProbDtsExample {

    public void run() {
        UnifiedJedis jedis = new UnifiedJedis("redis://localhost:6379");


        List<Boolean> res1 = jedis.bfMAdd(
            "recorded_users",
            "andy", "cameron", "david", "michelle"
        );
        System.out.println(res1);  // >>> [true, true, true, true]

        boolean res2 = jedis.bfExists("recorded_users", "cameron");
        System.out.println(res2);  // >>> true

        boolean res3 = jedis.bfExists("recorded_users", "kaitlyn");
        System.out.println(res3);  // >>> false

        boolean res4 = jedis.cfAdd("other_users", "paolo");
        System.out.println(res4);  // >>> true

        boolean res5 = jedis.cfAdd("other_users", "kaitlyn");
        System.out.println(res5);  // >>> true

        boolean res6 = jedis.cfAdd("other_users", "rachel");
        System.out.println(res6);  // >>> true

        List<Boolean> res7 = jedis.cfMExists(
            "other_users",
            "paolo", "rachel", "andy"
        );
        System.out.println(res7);  // >>> [true, true, false]

        boolean res8 = jedis.cfDel("other_users", "paolo");
        System.out.println(res8);  // >>> true

        boolean res9 = jedis.cfExists("other_users", "paolo");
        System.out.println(res9);  // >>> false

        long res10 = jedis.pfadd("group:1", "andy", "cameron", "david");
        System.out.println(res10);  // >>> 1

        long res11 = jedis.pfcount("group:1");
        System.out.println(res11);  // >>> 3

        long res12 = jedis.pfadd(
            "group:2",
            "kaitlyn", "michelle", "paolo", "rachel"
        );
        System.out.println(res12);  // >>> 1

        long res13 = jedis.pfcount("group:2");
        System.out.println(res13);  // >>> 4

        String res14 = jedis.pfmerge("both_groups", "group:1", "group:2");
        System.out.println(res14);  // >>> OK

        long res15 = jedis.pfcount("both_groups");
        System.out.println(res15);  // >>> 7

        // Specify that you want to keep the counts within 0.01
        // (0.1%) of the true value with a 0.005 (0.05%) chance
        // of going outside this limit.
        String res16 = jedis.cmsInitByProb("items_sold", 0.01, 0.005);
        System.out.println(res16);  // >>> OK

        Map<String, Long> firstItemIncrements = new HashMap<>();
        firstItemIncrements.put("bread", 300L);
        firstItemIncrements.put("tea", 200L);
        firstItemIncrements.put("coffee", 200L);
        firstItemIncrements.put("beer", 100L);

        List<Long> res17 = jedis.cmsIncrBy("items_sold",
            firstItemIncrements
        );
        res17.sort(null);
        System.out.println();  // >>> [100, 200, 200, 300]

        Map<String, Long> secondItemIncrements = new HashMap<>();
        secondItemIncrements.put("bread", 100L);
        secondItemIncrements.put("coffee", 150L);

        List<Long> res18 = jedis.cmsIncrBy("items_sold",
            secondItemIncrements
        );
        res18.sort(null);
        System.out.println(res18);  // >>> [350, 400]

        List<Long> res19 = jedis.cmsQuery(
            "items_sold",
            "bread", "tea", "coffee", "beer"
        );
        res19.sort(null);
        System.out.println(res19);  // >>> [100, 200, 350, 400]

        String res20 = jedis.tdigestCreate("male_heights");
        System.out.println(res20);  // >>> OK

        String res21 = jedis.tdigestAdd("male_heights", 
            175.5, 181, 160.8, 152, 177, 196, 164);
        System.out.println(res21);  // >>> OK

        double res22 = jedis.tdigestMin("male_heights");
        System.out.println(res22);  // >>> 152.0

        double res23 = jedis.tdigestMax("male_heights");
        System.out.println(res23);  // >>> 196.0

        List<Double> res24 = jedis.tdigestQuantile("male_heights", 0.75);
        System.out.println(res24);  // >>> [181.0]

        // Note that the CDF value for 181 is not exactly 0.75.
        // Both values are estimates.
        List<Double> res25 = jedis.tdigestCDF("male_heights", 181);
        System.out.println(res25);  // >>> [0.7857142857142857]

        String res26 = jedis.tdigestCreate("female_heights");
        System.out.println(res26);  // >>> OK

        String res27 = jedis.tdigestAdd("female_heights",
            155.5, 161, 168.5, 170, 157.5, 163, 171);
        System.out.println(res27);  // >>> OK

        List<Double> res28 = jedis.tdigestQuantile("female_heights", 0.75);
        System.out.println(res28);  // >>> [170.0]

        String res29 = jedis.tdigestMerge(
            "all_heights",
            "male_heights", "female_heights"
        );
        System.out.println(res29);  // >>> OK
        List<Double> res30 = jedis.tdigestQuantile("all_heights", 0.75);
        System.out.println(res30);  // >>> [175.5]

        String res31 = jedis.topkReserve("top_3_songs", 3L, 2000L, 7L, 0.925D);
        System.out.println(res31);  // >>> OK

        Map<String, Long> songIncrements = new HashMap<>();
        songIncrements.put("Starfish Trooper", 3000L);
        songIncrements.put("Only one more time", 1850L);
        songIncrements.put("Rock me, Handel", 1325L);
        songIncrements.put("How will anyone know?", 3890L);
        songIncrements.put("Average lover", 4098L);
        songIncrements.put("Road to everywhere", 770L);

        List<String> res32 = jedis.topkIncrBy("top_3_songs",
            songIncrements
        );
        System.out.println(res32);
        // >>> [null, null, null, null, null, Rock me, Handel]

        List<String> res33 = jedis.topkList("top_3_songs");
        System.out.println(res33);
        // >>> [Average lover, How will anyone know?, Starfish Trooper]

        List<Boolean> res34 = jedis.topkQuery("top_3_songs",
            "Starfish Trooper", "Road to everywhere"
        );
        System.out.println(res34);
        // >>> [true, false]
    }
}

A t-digest object also supports several other related commands, such as querying by rank. See the t-digest reference for more information.

Ranking

A Top-K object estimates the rankings of different labeled items in a data stream according to frequency. For example, you could use this to track the top ten most frequently-accessed pages on a website, or the top five most popular items sold.

The example below adds several different items to a Top-K object that tracks the top three items (this is the second parameter to the topkReserve() method). It also shows how to list the top k items and query whether or not a given item is in the list.

Java-Sync

package io.redis.examples;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class HomeProbDtsExample {

    public void run() {
        UnifiedJedis jedis = new UnifiedJedis("redis://localhost:6379");


        List<Boolean> res1 = jedis.bfMAdd(
            "recorded_users",
            "andy", "cameron", "david", "michelle"
        );
        System.out.println(res1);  // >>> [true, true, true, true]

        boolean res2 = jedis.bfExists("recorded_users", "cameron");
        System.out.println(res2);  // >>> true

        boolean res3 = jedis.bfExists("recorded_users", "kaitlyn");
        System.out.println(res3);  // >>> false

        boolean res4 = jedis.cfAdd("other_users", "paolo");
        System.out.println(res4);  // >>> true

        boolean res5 = jedis.cfAdd("other_users", "kaitlyn");
        System.out.println(res5);  // >>> true

        boolean res6 = jedis.cfAdd("other_users", "rachel");
        System.out.println(res6);  // >>> true

        List<Boolean> res7 = jedis.cfMExists(
            "other_users",
            "paolo", "rachel", "andy"
        );
        System.out.println(res7);  // >>> [true, true, false]

        boolean res8 = jedis.cfDel("other_users", "paolo");
        System.out.println(res8);  // >>> true

        boolean res9 = jedis.cfExists("other_users", "paolo");
        System.out.println(res9);  // >>> false

        long res10 = jedis.pfadd("group:1", "andy", "cameron", "david");
        System.out.println(res10);  // >>> 1

        long res11 = jedis.pfcount("group:1");
        System.out.println(res11);  // >>> 3

        long res12 = jedis.pfadd(
            "group:2",
            "kaitlyn", "michelle", "paolo", "rachel"
        );
        System.out.println(res12);  // >>> 1

        long res13 = jedis.pfcount("group:2");
        System.out.println(res13);  // >>> 4

        String res14 = jedis.pfmerge("both_groups", "group:1", "group:2");
        System.out.println(res14);  // >>> OK

        long res15 = jedis.pfcount("both_groups");
        System.out.println(res15);  // >>> 7

        // Specify that you want to keep the counts within 0.01
        // (0.1%) of the true value with a 0.005 (0.05%) chance
        // of going outside this limit.
        String res16 = jedis.cmsInitByProb("items_sold", 0.01, 0.005);
        System.out.println(res16);  // >>> OK

        Map<String, Long> firstItemIncrements = new HashMap<>();
        firstItemIncrements.put("bread", 300L);
        firstItemIncrements.put("tea", 200L);
        firstItemIncrements.put("coffee", 200L);
        firstItemIncrements.put("beer", 100L);

        List<Long> res17 = jedis.cmsIncrBy("items_sold",
            firstItemIncrements
        );
        res17.sort(null);
        System.out.println();  // >>> [100, 200, 200, 300]

        Map<String, Long> secondItemIncrements = new HashMap<>();
        secondItemIncrements.put("bread", 100L);
        secondItemIncrements.put("coffee", 150L);

        List<Long> res18 = jedis.cmsIncrBy("items_sold",
            secondItemIncrements
        );
        res18.sort(null);
        System.out.println(res18);  // >>> [350, 400]

        List<Long> res19 = jedis.cmsQuery(
            "items_sold",
            "bread", "tea", "coffee", "beer"
        );
        res19.sort(null);
        System.out.println(res19);  // >>> [100, 200, 350, 400]

        String res20 = jedis.tdigestCreate("male_heights");
        System.out.println(res20);  // >>> OK

        String res21 = jedis.tdigestAdd("male_heights", 
            175.5, 181, 160.8, 152, 177, 196, 164);
        System.out.println(res21);  // >>> OK

        double res22 = jedis.tdigestMin("male_heights");
        System.out.println(res22);  // >>> 152.0

        double res23 = jedis.tdigestMax("male_heights");
        System.out.println(res23);  // >>> 196.0

        List<Double> res24 = jedis.tdigestQuantile("male_heights", 0.75);
        System.out.println(res24);  // >>> [181.0]

        // Note that the CDF value for 181 is not exactly 0.75.
        // Both values are estimates.
        List<Double> res25 = jedis.tdigestCDF("male_heights", 181);
        System.out.println(res25);  // >>> [0.7857142857142857]

        String res26 = jedis.tdigestCreate("female_heights");
        System.out.println(res26);  // >>> OK

        String res27 = jedis.tdigestAdd("female_heights",
            155.5, 161, 168.5, 170, 157.5, 163, 171);
        System.out.println(res27);  // >>> OK

        List<Double> res28 = jedis.tdigestQuantile("female_heights", 0.75);
        System.out.println(res28);  // >>> [170.0]

        String res29 = jedis.tdigestMerge(
            "all_heights",
            "male_heights", "female_heights"
        );
        System.out.println(res29);  // >>> OK
        List<Double> res30 = jedis.tdigestQuantile("all_heights", 0.75);
        System.out.println(res30);  // >>> [175.5]

        String res31 = jedis.topkReserve("top_3_songs", 3L, 2000L, 7L, 0.925D);
        System.out.println(res31);  // >>> OK

        Map<String, Long> songIncrements = new HashMap<>();
        songIncrements.put("Starfish Trooper", 3000L);
        songIncrements.put("Only one more time", 1850L);
        songIncrements.put("Rock me, Handel", 1325L);
        songIncrements.put("How will anyone know?", 3890L);
        songIncrements.put("Average lover", 4098L);
        songIncrements.put("Road to everywhere", 770L);

        List<String> res32 = jedis.topkIncrBy("top_3_songs",
            songIncrements
        );
        System.out.println(res32);
        // >>> [null, null, null, null, null, Rock me, Handel]

        List<String> res33 = jedis.topkList("top_3_songs");
        System.out.println(res33);
        // >>> [Average lover, How will anyone know?, Starfish Trooper]

        List<Boolean> res34 = jedis.topkQuery("top_3_songs",
            "Starfish Trooper", "Road to everywhere"
        );
        System.out.println(res34);
        // >>> [true, false]
    }
}

Products

Tools

Get Redis

Connect

Learn

Latest

See how it works

Probabilistic data types

Set operations

Set membership

Set cardinality

Statistics

Frequency

Quantiles

Ranking

On this page